ONLINE SHAMING

Ethical Tools for

Human-Computer Interaction Designers

Erik Campano

Department of Informatics Magister thesis, 15 hp Master’s Program in Human-Computer Interaction and SPM 2020.13

Dedication to Kathleen

Content Warning This paper discusses the topics of racism, transphobia, and gender-based-violence.

Abstract A set of tools – concepts, guidelines, and engineering solutions – are proposed to help human-computer interaction designers build systems that are ethical with regards to . Online shaming’s ethics are unsolved in the literature, and the phenomenon can have devastating consequences, as well as serve social justice. Kantian ethics, as interpreted by Christine Korsgaard, provide our analytical methodology. Her meta-ethics invokes Wittgenstein’s private language argument, which also models relevant concepts in human-computer interaction theory. Empirical studies and other ethicists’ views on online shaming are presented. Korsgaard’s Kantian methodology is used to evaluate the other ethicists’ views’ moral acceptability, and guidelines are drawn from that analysis. These guidelines permit shaming, with strong constraints. Technical engineering solutions to ethical problems in online shaming are discussed. All these results are situated in the public dialogue on online shaming, and future research from other ethical traditions is suggested.

Keywords: online shaming, ethics, human-computer interaction

1. Introduction 1.1 Three Examples of Online Shaming We begin with three examples of online shaming. I will be using these examples throughout this paper. Justine Sacco: Justine Sacco was an American communications executive at a large media and holding company. She had a history of fighting for causes endorsing racial equality. She also had an “ascerbic” sense of humor (Ronson, 2015). On December 20, 2013, on the way to visit family in , just before she boarded a plane in London, she tweeted: “Going to Africa. Hope I don’t get AIDS. Just kidding. I’m white!” [Fig. 1] She says she intended this as a joke, an ironic, self-reflective critique of white privilege. (ibid). The tweet was republished quickly on the blog of technology journalist Sam Biddle. Readers, who did not know Sacco, did not appreciate the comment as an ironic joke, but rather saw it as serious racism. During her 11 hour flight to South Africa – when she was offline and could not explain herself – her tweet went viral. Sacco became the target of online anti-racism vigilantes, who posted tens of thousands of threatening and abusive messages. The number one worldwide trending became #HasJustineLandedYet. “We are about to watch this @JustineSacco bitch get fired. In REAL time. Before she even KNOWS she’s getting fired,” tweeted one critic. Indeed, when she landed, she had been fired (Billingham and Parr, 2019) and branded a racist on social media and search engines. Natalie Wynn: Natalie Wynn is a transwoman, former philosophy graduate student and YouTube vlogger who since 2017 has published movie-length video essays about ethics, politics, race, and gender, on a channel called ContraPoints. In October 2019, in a video,

1 she included a 10 second clip voiced over by pornographic actor Buck Angel, whom some rights activists criticize as a transmedicalist – that is to say, someone who believes that people who do not experience , or undergo medical transition, are not truly transgender. Wynn says she is not a transmedicalist. Nonetheless, transmedicalism critics barraged Wynn with hatred on social media for having used Angel’s voice. “Pee pee, poo poo, I'm ContraPoints and I can't stop being a transmedicalist and sh*tting myself in public bwuh,” wrote one commenter. The critics called for Wynn to be “cancelled” – for her channel to be boycotted. "Deplatform her just like every other racist, sexist, bigoted content creator,” a critic wrote. Wynn disappeared from public view for about a month, and then resurfaced with a widely-praised video called Canceling. In it, she details the process of online shaming, and also movingly describes the negative effects it had on her – and her colleagues. “I'm becoming a burden to anyone who associates with me. … I'm working on this totally alone because I'm aware of how radioactive I am right now. And I don't want to contaminate anyone else,” she says (Wynn, 2020). Canceling has received almost two million views. Brock Turner: Brock Turner was a 20-year old Stanford undergraduate. In January 2015, after a party in which he and a young woman named Chanel Miller drank heavily, he was discovered by two Swedish graduate students outdoors, on top of Miller while she was unconscious from the alcohol. Turner tried to run, but the students caught up and tackled him. Stanford banned Turner from the campus for life. He was convicted of sexual assault with intent to rape an intoxicated person, and two other charges. Judge sentenced Turner to six months in prison, three years’ probation, and lifetime registration as a sex offender. Anti-gender-based-violence advocates, led by Stanford law professor Michele Dauber, felt that this sentence was too lenient, and launched an online campaign to have Persky recalled from the bench. Countless and websites labeled Turner him a rapist; an introductory law textbook even used his picture next to the definition of rape [Figure 2.] (Rennison, 2017). After Dauber’s campaign, county voters recalled Persky from the bench – the first time that a judge in California had been recalled in 80 years. Persky took a new job as a high school tennis instructor, but was fired after a Change.org petition accused that school of “explicitly and ignorantly allowing rape culture to ensue.” (BBC, 2019) Dauber herself was the target of an online counter-campaign, which she says led to offline threats, such as two stalkers, and an envelope sent to her house with white powder in it. (Deruy, 2018) In September 2019, Miller told her story in a bestselling book, Know My Name: A Memoir. Turner still faces online death threats and calls for his execution. On May 14, 2020 one user tweeted, “my homies wanna kill brock turner”. [Fig. 3] There are uncountably many identified cases of online shaming, but for the purposes of this essay we shall focus on these three. Each has a certain extreme quality that makes it particularly useful for our analysis: Justine Sacco’s case was extreme in the relative small scale of the trigger (a single 12-word tweet from a private, largely unknown person) versus the large scale of the reaction (worldwide media coverage). Natalie Wynn’s case is extreme in how successfully the target defended herself, and in her detail of description of the dynamics and personal effects of online shaming.

2

Brock Turner’s case is extreme, as popular commentators have called it one of the most morally justified examples of online shaming. An influential anti-shaming book, Nation: The Global Epidemic of Online Hate, says about Turner: “We raged, we blogged, we vented, but finally, all that was left was to enact vigilante justice.” (Scheff et al., 2017, p. 13) The Turner case is also important for our purposes, because many extant academic arguments for online shaming see it as a tool to combat gender-based-violence.

1.2 A Working Definition of Online Shaming Online shaming, as defined in the literature, has many types. These include: • – publishing private information about an individual, such as his address or workplace, in order to encourage others to harass him • pornography – publishing sexual content without a subject’s permission • – for example, online , , threats, and fake profiles • – for example, children writing hurtful things about another child online • bigotry – publishing hate speech about a group based on race, gender, sexual orientation, religion, ethnic origin, or other identity status • gossiping – sharing false personal information carelessly online (Laidlaw, 2017) Popular and academic literature broadly concede that these types are ethically unacceptable. We will therefore skip them in this paper. Rather, we will explore one other, specific type, which, as will become clear, raises the ethical issues about which we are concerned. For our purposes, shaming is: a large group of people publishing many individual posts on social media, intending to punish a person extrajudicially for doing something they perceive as unjust, for the ostensible purpose of societal modification. This type is sometimes called online vigilantism, although many definitions of that term have been proposed in the academic literature (Billingham and Parr, 2019; Gallardo, 2017; Chandler, 2016; Chang and Poon, 2016; Hou, 2017). In this paper, we shall use online shaming and online vigilantism almost interchangeably. The former emphasizes the act of punishing, whilst the latter, its extrajudicial nature. How large is large, and how many posts are necessary, for punishment to count as shaming? There is no number. As I shall argue, the threshold is crossed when moral accountability becomes distributed among many shamers.

1.3 Knowledge Assumed by this Paper This paper is written for human-computer interaction designers. Therefore, it assumes knowledge of certain human-computer interaction concepts, specifically: distributed cognition, embodied interaction, persuasive technologies, metaphor, blended space, and the spiral of silence. This paper assumes no knowledge of academic philosophy, particularly ideas from Korsgaard’s Kantianism, Wittgenstein’s private language argument, social contract theory, and feminist ethics. These will be explained in due course.

3

2. Research Question and Aim Our research question is then: what tools can we develop to help human-computer interaction designers build systems that are ethical with regards to online shaming? By tools, I mean three different things: • a set of concepts, originating in academic ethics and human-computer interaction theory, that designers can use to evaluate the moral dimension of online shaming • a set of ethical guidelines that designers should follow • practical engineering solutions for preventing unethical online shaming The aim of the paper is, then, to provide these tools to designers of any systems that might enable shaming, which, now at least, are primarily social media platforms and blogs. What justifies this enquiry? There are three main answers to this question. First: online shaming is a sociologically dangerous phenomenon. Its outcomes include suicides (Sommer, 2015), public calls for sexual or physical violence against individuals (Gallardo, 2017), and punishment of innocent parties (Sprecher, 2016). Second: the ethics of online shaming are not a solved problem in the academic literature. Authors from the philosophical ethical traditions of social contract theory, and some feminist ethicists, argue directly against online shaming. Others – primarily feminist ethicists – argue in favor of it, directly or indirectly. Third: no code of ethics has been published for human-computer interaction designers with regards to online shaming. They have no commonly agreed-upon professional best practices to deal with the phenomenon. This paper addresses all three of these answers head-on. It develops a set of conceptual tools allowing the reader to do her own ethical analysis of online shaming. It suggests guidelines which could be primordial bases for a future code of ethics. It then explores technologies to implement these guidelines. My primary goal is not to derive a comprehensive, permanent set of ethical principles about online shaming for designers; that would be impossible to do in 29 pages. The goal is rather to offer original insights on moral issues, surrounding the phenomenon, which will inspire the reader to think through her own opinions about its ethics.

3. Research Methodology* 3.1 Overview of Research Methodology We define our analytical framework as the Kantian ethical test called the Categorical Imperative, as interpreted by prominent modern Kantian scholar Christine Korsgaard (1996). We will intertwine the Categorical Imperative with concepts from human-computer interaction theory listed above. In a moment, I will justify why I made this choice. Before we apply the method, we will need, in the Related Research section below, to establish: • a more complete empirical understanding of the characteristics of online shaming

* In the assignment template for this thesis, Related Research comes before Research Methodology. However, in my paper, I have reversed this order. An understanding of our methodology makes it much easier to understand related research.

4

• the various arguments for and against online vigilantism from the academic philosophical literature, and in a couple of cases, from particularly insightful popular philosophical literature We will then use Korsgaard’s Kantianism’s method to directly analyze the existing arguments for and against online shaming. Each separate analysis should produce one or more ethical guidelines. Once those concepts are compiled, we will suggest engineering solutions to help avoid enabling potentially unwanted online shaming. At a surface level, Kantian methodology is a simple process. Kant’s universal test, the Categorical Imperative, is used to determine whether an action is morally acceptable or not. However, the devil is in the details. Kantianism is particularly useful to human-computer interaction designers, if we describe the Categorical Imperative’s meta-ethical background using concepts that also underlie human-computer interaction theory. That is to say, the semantic, epistemological, and metaphysical theories out of which Korsgaard’s Kantianism can be built can also be used to model distributed cognition, embodied interaction, and so forth. The Categorical Imperative is not just a framework we apply around the phenomenon of online shaming; rather, the Categorical Imperative raises questions, as we shall see, about specific techno-social choices made by online shamers, as those choices are described by human-computer interaction theory. As a human-computer interaction designer, you may be used to reading short method sections in papers, which you might even skip over because many research methods are standardized in the discipline. In philosophy papers, however, method sections are often long, and this one is no exception. This is because much original thought in a philosophy paper occurs because the author has created a new methodology for analyzing perhaps an old problem. In fact, some of your first tools – your set of concepts – for evaluating the ethics of shaming in your interaction design are here, in the methodology section. To keep things clear, key tools borne out of the research methodology are underlined.

3.2 Why This Methodology? Korsgaard’s Kantianism is justified as a research method for three reasons. First: online vigilantism has already received analysis from two other major schools of Western normative academic ethics: social contract theory (as understood by John Locke) and feminist ethics. Furthermore, the most well-founded feminist argument for shaming – Goguen’s – is implicitly utilitarian, as I hope to show when we describe it later. Since there are four major normative Western ethical traditions (Driver, 2007)… 1. deontology, prominently represented by Kantianism and social contract theory 2. consequentialism, prominently represented by utilitarianism 3. feminist ethics, borne in part of critical theory, and 4. virtue ethics, originating in ancient Greek philosophy and arguably finding on modern form in existentialism (Davenport, 2001, p. 265 ff.) …the gap in the literature is in Kantian and virtue ethics. We must pick between the two, and since virtue ethics is the minority opinion among academic philosophers, we default to Kantianism, leaving virtue ethics for later research (Hursthouse & Pettigrove, 2018). Kant

5 is also a good choice because he is often considered the central figure of post-medieval philosophy (Rohlf, 2020). His ethical theory dates from the late 1700s. Second: the Categorial Imperative judges an action on whether or not it is universalizable and whether it treats people as ends and not mere means, that is to say whether it puts value on the dignity of each individual person. In online vigilantism, actions are, more or less, universalized. A single tweet by Sacco was retweeted almost five thousand times and seen by at least tens of thousands of people (Vingiano, 2013). Meanwhile, online vigilantism intends to use the target’s punishment as a means of enacting social change. The target’s own personal improvement is, sometimes, not an end in itself. For example, Wynn’s critics called not for her to be reformed, but rather deplatformed (that is, silenced online) via what Wynn called a “circular firing squad” among her audience, which is generally sympathetic to LGBTQ+ social justice issues. [Fig 4.] Online vigilantism can (often intentionally) ruin its target’s dignity – and that of her associates. Third: Korsgaard’s Kantianism is grounded in a meta-ethics which can also ground much of the relevant human-computer interaction theory. This meta-ethics is Wittgenstein’s account of linguistic obligation and language games which are central to his private language argument. This paper ties ethics into human-computer interaction theory in a both relevant and profound way via Wittegenstein’s meta-ethics. Korsgaard’s Kantianism, like all moral theories, is debatable. For this paper, we will apply what philosophers call the principle of generosity. We will use the best interpretation of Korsgaard’s arguments. We will assume Korsgaard’s Kantianism to be correct until the end of the paper, when we address its weaknesses and suggest avenues for future research.

3.3 Building the Methodology In this section, we describe Korsgaard’s Kantianism, how it relates to human-computer interaction theory, and what methodology evolves from their combination. This section assumes no knowledge of Kant. We will not attempt to explain every aspect of the derivation or meta-ethics of Korsgaard’s Kantianism (which could, and has been, debated endlessly by an untold number of authors), but rather just its parts relevant to human-computer interaction theory. To do this I am indebted to Guyer (2006) and Scruton (1982; 1996). It is easiest to begin with a quick description of the Categorical Imperative. It has three formulations, which Kant believed were equivalent in meaning, but different in wording and emphasis. For the moment, we state only the first formulation. Kant instructs you to follow this rule: I should always act so that I could will my action’s maxim to be a universal law (my translation; Kant 1785). A maxim is the reason you do the action – boiled down to its ultimate aim (Timmermann, 2000). Therefore, our methodology includes testing whether the maxim behind a shamer’s action (retweet, post, injurious message, etc.) could be willed to be a universal law. So, for a non-shaming but hopefully very clear example: you probably avoid driving on the left side of the road not only because Swedish (or US, or any right-side driving country’s) law tells you to; you do it because you do not want to crash and kill yourself; that is your real maxim. Imagine, however, for a moment, that you were in a traffic jam, and so you drove on the left side of the road with a different maxim: “I want to go much faster”. If this

6 maxim were a universal law, then everyone in the traffic jam would drive on the left side of the road for the same reason – to go much faster. In this case, the left side of the road would be crowded, and no one could go much faster. The maxim, “I drive on the left side of the road in a traffic jam because I want to go much faster” therefore, would yield a logical contradiction – this is the key point. Logical contradictions are impossible in reality. No human can possibly believe X and not X to be true; it is a rule of logic that we simply know a priori, just like 1+1=2. So, this maxim could not be willed to be a universal law. Kantians believe that the Categorical Imperative is baked into the structure of cognition itself – and by extension, distributed cognition. Kant’s starting point, which we will not derive because the derivation is irrelevant to our analysis, is that the most valuable thing in the world is a person’s good will, that is, her desire to act in a manner consistent with a free conscience, and consistent with the duties laid out by the Categorical Imperative. (Korsgaard, 2016b, p. 12) Dignity is exercise of the good will. (Kant, 1785). The Categorical Imperative is categorical, because it applies to all persons at all times. It is an imperative, because it commands you how to act. You might notice that it resembles, in form, the Christian Golden Rule: do unto others as you would have them do unto you (Matthew 7:12). The Categorical Imperative differs from the Golden Rule, in that if a person would have others hurt him, Kant does not prescribe him hurting others. Kant believed that the Categorical Imperative is a form of moral rationality understood by all humans a priori – like mathematics, or the laws of logic (Kant, 1785) – or image schemas in distributed cognition, as described by Benyon (2019, p. 558). If human beings have dignity inherent in their value as persons, then to deny them this dignity by constraining their moral choices would be to destroy something of moral value, which is the definition of doing a bad thing. Therefore, we have an obligation always to allow people dignity. This is what Kant calls treating people as ends rather than as means. So, you are obligated to “act in such a way that you always treat people – whether yourself, or someone else – always as an end in themselves, and never merely as a means.” (my translation; Kant 1785) This is the Categorical Imperative’s second formulation. So, when evaluating the ethics of a case of shaming, we have to consider whether the target (or perhaps shamer) is being treated as an end in herself, or as a mere means; that is to say, whether the target’s dignity is being preserved during the shaming. The moral law is “a piece of universal legislation, which binds rational beings equally” (Scruton, 1982). Maximizing morality involves everyone acting in accordance with these formulations of the Categorical Imperative. When everyone is treated as an end, and never as a mere means, we have created a community which is a “Kingdom of Ends”, in Kant’s terms. “Therefore, a reasonable being must act as if through its maxims, it is always a law-making member of a general Kingdom of Ends.” (my translation; Kant 1785) This is the Categorical Imperative’s third formulation. Although Kant probably never envisioned the Internet, I will argue that it is a community which can be a Kingdom of Ends. When evaluating the ethics of a case of shaming, we have to consider whether all shamers and targets are following rules that everyone could follow, furthering a Kingdom of Ends. Now that we understand the three formulations of the Categorical Imperative, we can dive more deeply into the meta-ethics behind them. You may have noticed that Kant has

7 made a conceptual jump that neither he nor I have explained very clearly. The jump is that the Categorial Imperative is not just a law that we must adhere to, but a universal law, that is to say, one which applies to all people at all times. This is where Korsgaard comes in, to explain this jump. Here is how she argues it. Korsgaard (1996) first assumes that the idea that the law is not universal -- that is to say, it does not apply to all people at all times, but rather to a limited group of people with whom one “identifies”, say, all men or all women, or all members of a religion, or all Umeå University affiliates. These identities are all contingent. One could, in theory, stop identifying as a man or woman, or could leave one’s religion, or Umeå University. A law that only applies to a contingent identity would not fulfil Kant’s project, which is to find a moral rule binding on all people at all times. There is one identity, however, that is absolutely not contingent: you are a human being. You cannot stop being a human being, unless you die. Human beings, crucially, are self-reflective. We can turn our perception to our own mental activity. We can think about our thinking. This means that we can question whether any possible action is right.

...I desire and I find myself with a powerful impulse to act. But I back up and bring that impulse into view and then I have a certain distance. Now the impulse doesn’t dominate me and now I have a problem. Shall I act? Is this desire really a reason to act? (Korsgaard, 1996, p. 93).

Because you are human, Korgaard then argues, you need to have some practical conception of your identity, or you would have no reason for action. If you did not – because your consciousness is reflective – then you would then never act at all.

Since you cannot act without reasons and your humanity is the source of all your reasons, you must value your humanity if you are to act at all. It follows from this argument that human beings are valuable. Enlightenment morality is true (p. 83) (emphasis mine).

What, in particular, is valuable about human beings is their good will. Therefore, any case of shaming that causes the target (or shamer) to lose his dignity is morally suspect. Reflectivity is crucial to acting according to conscience. External circumstances can threaten this. For example, the spiral of silence can someone to lose the ability to act according to conscience. “[P]eople’s natural fear of isolation from a social group often leads them to outwardly agree with a dominant public opinion, statement, or norm. This is true even if the same individual rejects the opinion internally.” (Huffman, 2016, p. 9) If a target’s associates are punished along with the target, then any sign of solidarity with the target will induce punishment. The mass media, according to Noelle-Neumann (1993), “more than any court, take it upon themselves” to select which issues matter to the general public (p. 153). “Today, the mass media includes social media services,” suggests Huffman (2016, p. 9). Those caught in the spiral of silence cannot express the rational reflection of their conscience. This is both psychologically distressing (producing cognitive dissonance) and

8 immoral, from a Kantian perspective, as they might be lying, which is always wrong.* Kant sees shaming-induced disgrace as socially forced silence. Sussman (2008) clarifies his position.

When we are disgraced, our freedom is indeed diminished, in that we lose a certain power to act regardless of our own “inner” attitudes or acceptance. Unlike the merely insulted, the disgraced are no longer able to do something that they may be sometimes morally required to do, viz protest against the disrespectful treatment they receive from others.

It follows that any case of shaming that deprives someone of her ability to act according to conscience is morally suspect. A shaming target can also lose the ability to act according to conscience when surrounded by an overwhelming topology and volatility in blended spaces, which “bring together physical, digital, informational, and social spaces and establish new conceptual spaces where people form intentions, make meanings, have sensations and feelings, and take actions.” (Benyon, 2019. p. 214) In a moment, we will have more to say about blended space. For now, we can understand the philosophical underpinnings of blended space, and many other human-computer interaction concepts, in Korsgaard’s defense of the Categorical Imperative against an objection from the skeptic-about-other-minds. This objection – repeated for the last 250 years – holds that Kant is not clear about why valuing your humanity means valuing that of others. That is because scholars have not come to agreement that Kant succeeds in proving that other minds exist at all. One prominent critic here is Aune (1991), who says that Kant’s model of consciousness is “essentially solipsistic.” So, Korsgaard turns to Wittgenstein’s famous private language argument, which, although hotly debated, is one of the premiere arguments for the existence of other minds in contemporary philosophy (Scruton, 1996, p.53). Wittgenstein laid out his private language argument in the mid-20th century. He begins by defining a private language as one which refers to a sensation a person experiences, but which is otherwise incommunicable. Imagine a sensation that a person calls “S”. There is no way she can describe S using a public language, for example as “painful” or “red” or “common”. It is completely unique from all publicly discussed sensations. Wittgenstein’s argument is that it is impossible for her to meaningfully call that sensation “S”. As Korsgaard reads Wittgenstein, language requires two minds: one who defines X to mean Y, and another mind to obey that rule. Otherwise, language has no meaning.

If what you call S is just that sensation which makes you feel like saying “S”, and it cannot be identified in any other way, then you cannot be wrong. … I impress upon myself that “S” will be the name for this sensation. [This] can only mean: this process brings it about that I remember the connection right in the future. But in the present case I have no criterion of correctness. One would like to say:

* For space reasons, I will not include Kant’s argument for the immorality of lying. See Carson (2010).

9

whatever is going to seem right to me is right. And that only means that here we cannot talk about “right” (from Korsgaard, 1996, p. 137).

If private languages are impossible, then language is a public phenomenon. This means a public must exist – that is to say, other minds share the language. If you are having trouble understanding the private language argument immediately, that is normal. Many philosophers agree that it needs to be re-read or re-explained a few times before it clicks (Scruton, 1996, p. 52). This is also my experience. In any case, if the private language argument is correct, then reasons for action are also public. “[T]o say that R is a reason for A is to say that one would do A because of R; and this requires two, a legislator to lay it down, and a citizen to obey” (Korsgaard, 1996, p. 138). A shamer cannot simply say, “I am targeting this person for my own private reason R, impossible to explain to others”. Reasons exist in a web, stretched out over time. Furthermore, when we speak (or write), we “obligate”, in Korsgaard’s words, others to think something. “If I say to you, ‘Picture a yellow spot!’ you will,” she notes. We can also obligate others to consider reasons – maxims – for action.

If I call out your name, I make you stop in your tracks. … [I]f you walk on, you will be ignoring me and slighting me. … By calling out your name, I have obligated you. I have given you a reason to stop. Of course that’s overstated: you don’t have to stop. … But … in ordinary circumstances, you will feel like giving me [a reason] back. “Sorry, I must run, I’m late for an appointment.” (p. 140)

As we trade information, reasons, and responsibility, we engage in what Wittgenstein calls “language games”, which are “social shared linguistic practices ‘consisting of language and the actions into which it is woven.’” (Dourish, 2004, p. 123) Linguistic obligation and language games can be used to model much relevant human-computer interaction theory. In the following paragraphs we will illuminate Korsgaard and Wittgenstein’s contributions to distributed cognition, embodied interaction, metaphors, persuasive technologies, and, finally again, blended spaces. Each of these has an implication for our ethical analysis. During distributed cognition, a web of minds, artefacts, and devices obligate each other to think and reason (a simple definition of cognition from Benyon (2019, p. 546). If a shamer has a reason, then it somehow must be semantically related to a web of reasons communicated among fellow shamers. Since reasons are tied to moral responsibility, online shamers share moral responsibility. Therefore: to analyze the morality of a case of shaming, we must take into consideration how responsibility is distributed amongst the shamers. During embodied interaction, a computing environment is “grounded in and emerging out of everyday, mundane experience” (Dourish, 2004, p.125). The verbal is the embodied. The shamer who says, “feel X”, obligates the target to feel X, or some other kind of reaction – in any case, an emotion, which is felt in the body. Online punishment therefore leads to physical manifestation. As Wynn (2020) described the feeling of being cancelled:

I'm not allowed to get angry. I'm not allowed to show pain. I'm not allowed to get defensive, and I'm not allowed to lash out. All I'm allowed to do is go totally numb on the inside as I try to frantically calculate the ideal public relations

10

response that pays due deference to the valid concerns of these poor marginalized people, all the while ignoring the tsunami of verbal abuse that's crashing over me

A shaming campaign affects the target’s mundane experience. Sacco and Persky, for example, could not keep their jobs. Therefore: the actual emotional and physical experiences caused by a case of shaming matter to its moral value. I am not saying here that a case of shaming can be judged by its consequences; that would be consequentialism, which Kant strenuously argued against. What I am arguing is that shaming is a process, not a single instance of action. It plays out over time and involves many agents, each of whom is affected in an embodied way and responds to that effect. Cases of shaming set precedents for future cases. When evaluating the morality of a case of shaming, you must try to apply the Categorical Imperative to each step in the sequence of events surrounding it, and think about the precedent it sets. Metaphors as they appear in human-computer interaction theory are a language game played between the computer’s designer and the user. Social media branding contains much metaphor; presents itself as an online version of pre-internet printed school facebooks, which contained a picture of all students along with information about them. Twitter’s metaphor is a tweeting bird among a crowd of birds. A predecessor to social media is the bulletin board system, a metaphor for a place to post messages online. Social media platform creators and users are accepting rules to a game. The game is that the platform’s branding linguistically obligates the user to think about the platform as an intimate metaphor, rather than an Internet megaphone. Human-computer interaction philosopher Blackwell (2006) raises alarms about metaphors in user interfaces, pointing out numerous similar examples that “cast doubt on the common engineering perception that an instructional metaphor might be retro-fitted as a communication channel that explains existing system functionality, rather than being the core of the design.” The presentation of social media as metaphor therefore obscures the actual impact of posting. If the user confuses the metaphor with what it actually represents, he may think that his publication is less visible than it is. Facebook and Twitter are accessible to the entire world, not only to the people on a college campus who own a physical facebook, or to the small area of a forest where a bird’s tweet might be heard. Most social media companies – nowadays, at least – have built virality into their business model. There is a profit incentive to encourage shaming. Sam Biddle openly acknowledged that as his motivation, as we shall see. One way that the companies drive the virality is by employing deceptive metaphors. So: if a shamer insults the target in a real-life conversation over a dinner table with one listener, that shamer linguistically obligates only his dinner partner to experience the insult. However, if the shamer posts the insult on social media or a website, he obligates every person who sees it to experience the insult. In major shaming cases – like Sacco’s, where #HasJustineLandedYet became the number one hashtag on Twitter – literally millions of people can be obligated to experience a punisher’s message. To call it a “tweet” is to use a poor metaphor; a better metaphor is speaking into a microphone at a huge, packed stadium. Shamers may know this intellectually, but the benign metaphorical presentation of Twitter or Facebook provides cover for their power – even, perhaps, to them. Although after his conviction Turner was linguistically obligated to being presented

11 as evil in the minds of probably millions of people, “it appeared to the students, and to many in the public, that Turner was being treated with unearned deference” (emphasis mine) because his prison sentence was for six months (Rennison, 2017). I am not making a claim that his shaming and his sentence were morally unjustified. What I am saying is that the common statement that Turner was not punished very severely is open to question. Although his prison sentence was short, his online shaming is uncountably vast. The target can himself become a metaphor; Turner has become a metaphor for rape. His picture stands beside the definition of rape in a criminology textbook. Therefore: to properly evaluate the morality of a shaming case, we must take into account how the metaphor the social media employs does (or does not) deceive users into misunderstanding the obligative power of their posts. Furthermore, we may need to evaluate if and how the target has become a public metaphor for the perceived injustice. Persuasive technologies, as intended by one of their father figures Fogg, engender mass interpersonal persuasion, without coercion (Fogg, 2008). They can linguistically obligate a user to reconsider an online action. When we discuss engineering solutions, we will include the option of a persuasive technology offering shaming warnings. In analyzing a shaming case, we can ask if technology was used to try to combat the shaming. Blended spaces obligate a target to re-experience shaming throughout physical space. A physical place is “deliberately integrated in a close-knit way with a digital space” to create a conceptual space which is very much a real user experience (Benyon, 2019). “The spatial geometries of the local space continue coherently across the distributed boundary into the remote site, providing the illusion of a single unified space,” explain O’hara et al. (2011). Online vigilantes exact punishment by putting insults and threats into as ubiquitous a physical space as possible, even to the point of, for example, encouraging people to visit the homes of the targets. The target’s whole environment can become a blended space of online and offline punishment, impossible to escape. In Benyon’s (2019) terminology, “topologically” the target’s punishment surrounds and overwhelms him, and is “volatile” and ever-changing. For example: erected a memorial of the Turner case on the spot of the assault, a metal plaque bearing a quote from Miller’s online victim impact statement, which calls for Turner’s punishment, and which itself had gone viral online (Bae, 2020). [Fig. 5] Miller’s book is sold in stores throughout the nation. It is not just Turner who is overwhelmed. Such a high-profile case of gender-based violence, publicized ubiquitously, risks retraumatizing victim-survivors. Therefore, our moral tools include considering how shaming is embedded in the blended space of physical and online environment. Ubiquitous embedding can emotionally overwhelm a target, or victim- survivor, so that he has trouble reasoning morally, and therefore becomes unable to act according to good will. This concludes our presentation of Korsgaard’s Kantian ethical theory. We looked at how it is related to human-computer interaction concepts via Wittgenstein’s private language argument. From this, we derived useful conceptual tools for designers to evaluate online shaming. In Section 5, we will use these tools to evaluate arguments from other traditions against and for online shaming. First, however, we need to learn what exactly those arguments are, and also more about the empirical data we have on shaming.

12

4. Related Research For most of this paper, I have been primarily using the term online shaming, but in this section, I will mostly be using online vigilantism. That is because the authors I cite here are discussing vigilantism specifically, emphasizing its extrajudicial nature. How is our research question positioned with regards to this related research? Our research question is: what tools can we develop to help human-computer interaction designers build systems that are ethical with regards to online shaming? Our related research provides data to answer this question directly. First, we present empirical findings about characteristics of online shaming. Then, we look at how academic philosophers so far have evaluated the ethics of online shaming. All this data will then be ripe for application of our research methodology, Korsgaard’s Kantianism.

4.1 Empirical Observations about Online Shaming

4.1.1 Norm Enforcement, Anonymity, and Virtue Signaling Gallardo and Trottier explain how online vigilantism can involve norm enforcement, anonymity, and virtue signaling. Norm enforcement is the vigilante’s ostensible attempt to punish the target in order to promote a social norm that the target has broken. “[I]ndividuals seek to correct behavior that does not comply with the perceived norm, or rule of conduct in society” (Gallardo, 2016, p. 726). Trottier (2016) describes a similar concept to norm enforcement, but with a moral layer: online vigilantism “seeks to identify and shame unmitigated evil in embodied contexts … [it] is similar to cyber-bullying [but] framed in terms of a moral compass.” The norm is a just world that the vigilante imagines. Vigilantes can use anonymity to escape being held responsible for threatening behavior. “The anonymity that the Internet provides enables some individuals to dodge accountability and any kind of tangible fallout from derisive statements they make about other individuals online.” (Gallardo, 2016, p. 728) Is not just assignment of responsibility, but real-world backlash, that the vigilante may try to avoid through anonymity. Virtue signaling, in contrast, is participating in online vigilantism un-anonymously, so that other internet users think the vigilante stands up for a generally-accepted social norm. Gallardo does not actually use the term virtue signaling, but it is widely used elsewhere. (Jordan & Rand, 2019) “By reprimanding offensive behavior, individuals can effectively and easily signal to others that they are trustworthy because they punished a norm violator” (Gallardo, 2016).

4.1.2 Psychological Characteristics of Online Vigilantes Chang and Poon (2016) attempt to describe some of the psychological characteristics of people likely to engage in online vigilantism. A survey of Chinese online vigilantes showed that they tend to: 1) perceive the criminal justice system as ineffective; 2) possess high “self- efficacy” – that is, “online disinhibition”, wherein they are “more likely to express negative emotions, harsh criticism, anger, hatred, and threats” online than in real life; and 3) perceive online vigilantism to be a means to social justice.

13

Chang and Poon hypothesize that online vigilantes possess strong psychological empowerment: a “personal belief in control” combined with “involvement in community activities”. This leads to a snowballing cycle of achieving a goal, perception of control, and further participation in online vigilantism.

4.2 Other Philosophers’ View of Online Shaming There is very little academic philosophical literature on the ethics of online shaming. Billingham and Parr provide the only real comprehensive analysis from a normative ethical tradition, social contract theory. We want to draw from as much published wisdom as possible, though, so we will consider a few cogent arguments from non-peer-reviewed philosophical sources, namely Wynn, R.Dave, and Goguen. Their analyses stand on their own merits, and are worthy of attention.

4.2.1 The Stages of Canceling Wynn (2020) breaks down the various stages of the canceling version of online vigilantism, all of which she finds ethically problematic. All quotes that follow in this section are from Canceling. 1) Presumption of guilt. Online vigilantism skips the process of hearing the target’s side of the story. Wynn locates presumption of guilt in politically progressive circles where the slogan “believe victims” has currency. “It’s a norm that was put into place in progressive spaces because out in the world at large, people generally don't believe victims.” Online vigilantism ignores what in law is known as due process for the accused. 2) Abstraction. Online vigilantes abstract out broadly negative general claims about the target from specific details. Wynn points out “beauty guru” James Charles, who was accused of “trying to trick straight men into thinking they’re gay.” Regardless of this claim’s unlikeliness, an online vigilante quickly rewrote it as “James Charles is a sexual predator.” The changing involves a “linguistic shift”, says Wynn, wherein “the verb in the sentence is is or to be. … it’s not now his actions we’re criticizing, but his personality.” Abstraction usually falsely attributes an undesirable general character trait to the target, and is usually thus a form of lying. 3) Essentialism. Nuances of the target’s life and character are lost, and he is reduced to symbolizing an essential vice. Wynn thinks that “sexual predator” is not a fair summary of the original, already doubtful, accusation against Charles. Essentialism “happened instantaneously” and it “dominated the conversation in a community of millions of people for weeks.” Charles was defenseless against it. With full humanity denied, targets become a metaphor for a certain type of norm violator. 4) Pseudo-Moralism or Pseudo-Intellectualism. Some online vigilantes might say that they just want an apology, criticism, or for the person to be reformed, but this is a smoke screen for attacking a person. “[A] lot of my YouTube friends were like, ‘Did you see James Charles was canceled? L-O-L-O-L-O-L-O-L,” Wynn recalls. 5) No forgiveness. Charles posted an apology which Wynn says “could not be better”, but “[c]ancellers will often dismiss an apology as insincere, no matter how convincingly …

14 delivered. … [as] further proof of what a Machiavellian psychopath you really are.” (Note: not all apologies are, of course, sincere.) 6) The Transitive Property of Cancellation. A target’s associates are punished along with the target. Wynn’s observed that she became “radioactive” to her colleagues.

4.2.2 A Kantian View (R.Dave) In a thorough review of the literature, I found only one single Kantian analysis of online vigilantism. This analysis appears in a blog comment on a feminist philosophy website. It is not peer-reviewed, but it is literally all we have, and it raises an important argument. The commenter, with the handle R.Dave, sees online vigilantism as treating persons as a mere means, and so violates formulation two of the Categorical Imperative. Vigilantism uses targets as mere means “by punishing them to a greater extent than their individual actions warrant in order to achieve a broader goal of societal modification.” (Goguen, 2013)

4.2.3 A View from Social Contract Theory Billingham and Parr’s (2019) analysis is the most detailed ethical breakdown of online vigilantism in the academic literature, and is drawn from Locke’s social contract theory. They find that Locke can come down in favor of online vigilantism, but with many complications. Shaming makes an offender aware he has violated a social norm; deters others from violating it; demonstrates solidarity with those wronged; and promotes public expression of what is right. “Ideally,” Billingham and Parr write, the target will “recognize that she has acted wrongly, feel remorse, apologize, seek to make appropriate amends, and commit to complying with the norm in future.” That is the ideal. As we have seen from our three case studies, however, such outcomes sometimes fail to occur. Locke (as did Kant) argued that anyone who punishes another should always be accountable. The shamer must therefore also be able to held to account. “The decentralized nature of online public shaming,” however, “means that no particular participant can be held to account for the full extent of an instance of shaming” (Billigham & Parr, 2019). Since with distributed cognition comes distributed responsibility, an accountability threshold is eventually crossed. Punishment by individuals scales up to shaming when the sheer mass of shamers makes determining anyone’s relative accountability practically impossible. Here are Billingham and Parr’s (2019) three complications. 1. A group of potential vigilantes can disagree about whether a social norm has been violated. “This is because the nature of an individual’s conduct is sometimes unclear, such as in the case of Sacco, whose comments were intended as ironic.” 2. Informal punishment (sometimes like formal) tends to favor socially advantaged individuals. It is “likely to reflect racist, sexist, and classist biases, and so be more targeted against members of marginalized groups” who do not have the resources to defend themselves. 3. Disproportionate punishment. The pain inflicted by online shaming could be much greater than deserved. “We hope that an increased awareness of the risks of disproportionality will lead people to be more hesitant to engage in online public shaming. (ibid)”

15

4.2.4 Feminist Views Feminist views are crucial to our analysis, because the primary ethical defense of online vigilantism can be constructed from a feminist theory of pre-emptive self-defense. However, no academic work of feminist writing addresses the ethics of online vigilantism as directly and thoroughly as, for example, Billingham and Parr do. We have to piece together the feminist analysis from various sources.

4.2.4.1 For Online Vigilantism The most direct feminist philosophical defense of online vigilantism is in a blog post by the feminist philosopher Goguen (2013). She sees societal modification as extremely important. Problems like racism, transphobia, or lack of justice for gender-based-violence survivors are massive, she argues, and a person who contributes to these problems – even in a small way – is contributing to a great evil. Therefore, responding to them with massive online vigilantism is actually a proportionate response. She uses this analogy:

If I’m in a boat with another person, and they are bored and carelessly poke a single hole in the boat’s hull, I might be justified in getting mad, but throwing that person overboard (which could possibly do real harm to them) would not be warranted. But if we are in a boat where there are already lots of holes in the hull, and we are in serious danger of sinking, and they poke another hole in the boat (also: they don’t think we will actually sink, and even if I yell at them they will probably do it again), I think I am warranted in throwing them overboard in order to safeguard myself (Goguen, 2013).

The person who writes a racist tweet, or a YouTuber who associates with a transmedicalist, or a sexual assailant who does not apologize, or the judge who under-sentences him, may be perpetuating a great evil, according to this argument. Therefore, that person deserves to be removed from the community, in order to protect it from that evil developing further. This is, in essence, a utilitarian-feminist stance. Utilitarianism seeks to maximize the greatest utility for the greatest number. The boat hole-poker, or any norm violator, must be removed “in order to safeguard” the punisher and her cohort. This improves the chances of survival – and therefore, utility – for the other sailor or sailors. Not throwing the hole-poker overboard means a bad outcome for everyone.* D’Amore (2017) takes a related stand, in favor of – or, at least, not opposed to and can be read as romanticizing – what she calls vigilante :

the performance of vigilantism by girls and women who have undertaken their own protection, and the protection of others, against violence—such as sexual assault, abduction, abuse, and trauma—because they have been otherwise failed

* One of my correspondents, Kalle Grill, points out that Goguen’s argument also has a deontological flavor, in that “those who think that morality is about not harming others, rather than about maximizing the good, typically make exceptions to the rule against harming, with self-defense a central exception.”

16

Vigilante feminism finds theoretical backing in some feminists’ approval of preemptive self-defense, which permits someone at danger of domestic violence to kill her partner, 1) even if the threat is not imminent, 2) if she has no means of leaving the relationship, and 3) by using overproportionate violence, like a weapon against an unarmed man. (Hartline, 1997, p. 166-167) D’Amore identifies vigilante feminism in fairy tales and movie characters, who use extrajudicial, often violent, means to punish men who commit gender-based- violence. They “track and kill serial rapist/murderers because law enforcement failed to act on behalf of the victims.” Vigilante feminism “does not dismantle patriarchy, but rather uses patriarchal means to undergrid its own takeover. (D’Amore, 2017)” D’Amore defends this violence by illustrating how serious the problem of gender- based-violence is. She quotes feminist popular author Valenti (2017), who describes:

a constant fear of rape (conscious or not), [when] women do things throughout the day to protect themselves. Whether it’s carrying our keys in our hands as we walk home, locking our car doors as soon as we get in, or not walking down certain streets, we take precautions.... It’s essentially like living in a prison—all the time (Valenti, 2017, p. 63, quoted in D’Amore, 2017)

Responding to this ever-pervasive fear of rape warrants justice “by any means necessary” (White & Rastogi, 2009). Women using vigilante violence to right the wrong of gender-based-violence does not only occur in fiction. Jane (2016) points to the case of female gangs in India. “[T]he major targets … are corrupt officials and violent, immoral husbands: ‘their activities range from beating up men who abuse their wives to shaming officials with whatever weapons are available including walking sticks, iron rods, axes, and even cricket bats’” (White and Rastogi, 2009, p.313) Jane says that this is “not representative of ‘irrational, spontaneous mob violence’ but of ‘grassroots feminist sensibilities’ which offer ‘psychological, social, and justice-related assistance to … women who have been failed … by the judiciary system.’”

4.2.3.2 Against Online Vigilantism The major feminist argument directly against online vigilantism comes from Powell (2015). She defends gender-based violence victim-survivors telling their stories on social media, without necessarily naming abusers. Powell calls for “technosociality”, whereby informal justice can be carried out online. “[T]echnologies become embedded in our experiences of the social world at the same time as they contribute to new social and cultural practices.” Along with the usual concerns about due process, attacks on dignity, and disproportionate punishment, Powell points out the potential for catastrophic outcomes from online vigilantism.

High profile cases of rape victim suicides following abuse on social media (such as Rehtaeh Parsons and Audrie Pott) are tragic examples of the extent of the additional harm and trauma experienced by victims when the evidence of an assault never goes away—and when the response online via social media and the public sphere is all too often negative and victim-blaming (Powell, 2015).

17

Justice does not involve “focused action against accused perpetrators”, but rather victim- survivors “voicing personal narratives of sexual violence to be acknowledged by a trusted audience”. Justice means “information, participation, voice, validation, control, and offender accountability,” and Powell hopes that “social media, blogs, and other online communications increasingly [mediate] informal justice for rape.” This would require more research into how informal justice functions. “[T]here is a comparative underdevelopment within theoretical criminology regarding the nature and mechanisms of such practices.” In 2019, Miller showed the power of the personal narrative view of justice, when she went public as the victim-survivor in the Turner trial. She, like Goguen, saw her case as one instantiation of a wide-ranging problem, and therefore continued to feel that Turner’s sentence was too short, and that Persky’s recall was justified. “I just want people to know that it’s not happening in a vacuum, in the neat perimeter of college campuses. It is not young people being too silly and too reckless,” she told CNN. “There are greater patterns of male sexual entitlement playing out.” (Sangal, 2019) Miller does not defend shaming, or vigilantism, as such. However, going public and publishing a bestseller about the case necessarily had ramifications for Turner and Persky. It turned a harsh public lens toward them once more. Meanwhile, however, Miller also took control of the narrative of the case. Instead of public discussion of who Turner and Persky are, what they did, and whether or not they were properly punished, her book addresses who Miller is, what she experienced, and how gender-based-violence can be prevented and how victim-survivors can be supported. She has thereby raised awareness of gender-based- violence. Is this vigilantism? Not necessarily, according to our definition – Miller may not intend to further punish Turner and Persky in order to achieve societal modification. Their punishment is a by-product, although one might argue that they punished themselves in their original actions. (Kantians will have something to say about this later.)

5. Data Analysis 5.1 Kantian Responses to the Various Arguments For and Against Online Shaming We now look at the various arguments against and for online shaming in 4.2 above, and discuss how Kantians might respond. We will use that response to try to develop a set of ethical guidelines for human-computer interaction designers to follow. As before, these new tools for you will be in underlined. In this section, I present a radically new conception of the moral obligations of the human-computer interaction designer – obligations that may seem impossible to fulfil. Until today, the platforms where shaming take place, such as Twitter and Facebook, engage in very little content regulation. They might defend this as a free speech policy, but Kantians would retort that by not stopping some online shaming, these platforms are complicit in its spread. Bear with me, because later, I will suggest some technologies that might make fulfilling these obligations possible.

18

5.1.1 Response to Wynn Wynn (2020) described six morally problematic stages in online vigilantism: presumption of guilt, abstraction, essentialism, pseudo-moralism or pseudo-intellectualism, no forgiveness, and the transitive property of cancellation. Let us address each. Presumption of guilt. Presuming a person’s guilt – not giving them some kind of due process in justice – involves failing to consider the whole context of their perceived unjust action. It denies the target the chance to comment on the case, and perhaps establish her innocence. Every justice system has its flaws, and due process is meant to try to ensure that innocent people are not punished, whether that punishment is capricious, abusive, or just scapegoating. Presuming guilt therefore uses the target as a means to an end, and violates the second formulation of the Categorical Imperative. It follows for human-computer interaction designers that any shaming you allow should only be targeted at a person whose guilt has been proven. How much proof is required? That is a topic for future research. Abstraction. As pointed out above, abstraction is usually a form of lying, which Kant forbids. Therefore, your platform as a designer should find a way to prevent people from abstracting out false character traits about a person based on a few details about them. Essentialism. Essentializing a target turns her into a metaphor for a particular kind of norm violator. The person’s name, picture, and so forth are then being used as vocabulary elements in a language game (Sacco for racist, Wynn for transmedicalist, Turner for rapist). If someone consents to being used as a metaphor, this poses no ethical problem because it is a free choice. For example, Liszt might have consented to be a metaphor for “virtuoso piano player” (which he is, among pianists). Without consent, however, the person is being used as a means to an end, and we have a violation of the second formulation. It follows that people should only be used in your system as a metaphor when they have given consent to be used that way. Pseudo-moralism or pseudo-intellectualism. Another form of lying. You need to find a way for your platform to prevent fake demands for apologies, criticism, or reform. No forgiveness. Kant was clear that he believed repentant wrongdoers should be forgiven (Kant, 1797). When we forgive a wrongdoer – when we overcome negative emotions toward her, such as vindictiveness, anger, and resentment – we cease obligating those emotions in our interaction with that person. When the wrongdoer does not feel those emotions, she is more likely to be able to reflect rationally on her actions and thereby grow in moral conscience. Maintaining negative emotions in this case would, conversely, deprive the target of the ability to act according to conscience. Therefore, your platform must have some mechanism for preventing automatic dismissals of a target’s apologies – if they are genuine apologies, that is to say, not lies. (Victim-survivors of gender-based-violence report that they commonly receive insincere apologies from assailants (Marsh & Wager, 2015).) The Transitive Property of Cancellation. We have seen how we must evaluate the how shaming is embedded in a physical environment. When the transitive property applies, associates of the target are shamed. The emotion of shame is obligated across a community. The transitive property is really the trickiest of Wynn’s concepts to analyze, because in real-life cases, people have felt very justified in shaming associates of targets. The Turner case provides a clear example. During the trial, 39 people wrote character statements for

19 the court supporting Turner. Two of them, childhood friend Leslie Rasmussen and high school guidance counselor Kelly Owens, faced public backlash for their letters. Rassmussen’s rock band was subject to a “viral mob” (Cuevas, 2016) and subsequently dropped from a New York music festival (Levin, 2016). Both women retracted their statements. If responsibility is distributed among shamers, it is also distributed among norm violators. Meanwhile, if a target becomes a metaphor for a perceived injustice, then so can associates. So, on one hand we have an argument for blaming associates, and another saying that they should only be used as a metaphor with their consent. Our Kantian analysis leaves us in a bind here as to the moral acceptability of the transitive property of cancellation. I will leave this here as a limitation of Kantianism. This section contained a number of guidelines which seem to require constant human monitoring to enforce, and so are unrealistic for large platforms like Twitter or Facebook. However, Powell (2015) will proffer a partially automated solution for large platforms.

5.1.2 Response to R.Dave R.Dave argued that online vigilantism treats the target as a mere means to the end of societal modification, and therefore violates the second formulation of the categorical imperative. Here, I will argue that this is not always the case; that there may be a tipping point at which too many ill-intentioned vigilantes can overwhelm well-meaning ones who are respecting dignity; and whether the target actually turns out being treated as an end in herself is unpredictable and may only be determinable long after the vigilantism stops. We begin with the question of whether targets are indeed only used as mere means. This would imply that in no cases of online vigilantism, targets are ends in themselves. To treat someone as an end in herself means to support her freedom to make moral decisions according to her conscience. Is it ever the case that online shamers are doing this? It is not impossible. Without a doubt, in Sacco, Wynn, and Turner’s cases, some, if not most, shamers were trying simply to restrict the target’s freedoms – freedom from emotional pain, freedom of speech, even freedom of movement or from physical pain. Some vigilantes may, however, have genuinely been attempting to raise the targets’ moral awareness – treating them as an end in itself. We have defined shaming as an attempt to punish a person. The goal – and effect -- of punishment can absolutely be to raise a person’s moral awareness. Offenders are separated from society in prisons, in part, so they can use the time to reflect upon their actions and realize how they were violations of moral law. So, when a vigilante calls for a target to be ostracized, they may be doing so in the hopes that the target will reform his thinking. Online vigilantism involves, however, a large amount of minds. A single genuine attempt to raise moral awareness may be surrounded by a dozen messages intended only to attack, linguistically obligating emotional and therefore bodily impacts on the target, preventing the target from reflecting rationally upon his actions. So, any well-meaning vigilante may be drowned out by others who are punishing the target as a mere means. In any given instance of online vigilantism, then, there is a balance, a tipping point, at which too many vigilantes would be punishing with ill intent, disabling the target from

20 being able to make sound moral reflection. The precise location of that tipping point cannot be said generally; it would seem to vary from case to case and target to target. Therefore: design your platform such that ill-intended shamers cannot overwhelm well-meaning ones. It is here that we must distinguish between short-term and long-term outcomes. In our three examples, the targets were stunned in the short-term, unable to effectively defend themselves. However, we have three very different long-term outcomes. As for Sacco: Biddle met with her some six months after he retweeted her. As he recounts the conversation, she said, “I was so naïve.” (Biddle, 2014) “She was serene, decent, and despite the continued existence of Twitter, hopeful: ‘Someday you’ll Google me and my LinkedIn will be the first thing that pops up.’” Her conclusion on how to handle being a target was: “Just don’t engage.” Keep silent, and do not try to defend oneself, for doing so feeds the mob (Ronson, 2015). Meanwhile, after actually meeting Sacco, Biddle engaged in his own moral reflection about being a vigilante:

Would I post the tweet again? Sure. Would I post the tweet knowing it's going to cause an incredibly disproportionate personal disaster for Justine Sacco? No. Would I post the tweet knowing it could happen? Now we're in dicey territory

So, the vigilante process ended up causing the vigilante to regret being a vigilante. Wynn, in contrast, did exactly what Sacco does not advise: Wynn engaged. She engaged massively, in the form of Canceling. Wynn, unlike Sacco, defends her original action that triggered the shaming:

trans Twitter won't forgive me unless I condemn Buck Angel as [a transmedicalist] and apologize for ever working with him. But I'm not going to do that because I don't think it would be the right thing to do, and doing it just to get myself uncanceled would be cowardly (Wynn, 2020).

So, if Wynn’s well-meaning vigilantes wanted her to examine her conscience and discover that employing Angel was morally wrong, they did not succeed. Furthermore, the ill- intentioned vigilantes who wanted to deplatform Wynn also failed. As for Turner, little is known about his long-term outcome. Turner has not spoken out publicly in years. I attempted to reach him for an interview for this thesis, without success. Therefore: you do not know the long-term effects of online shaming; they could be very opposite from the short-term effects. This is where Kant’s anti-consequentialism comes in. This can be understood in a brief analogy, which is about murder so it may feel a bit jarring, but we shall use it because it is the analogy Kant himself famously used. He argued that if a murderer at the door demanded to know if your spouse was at home because he wanted to murder her, and your spouse was indeed at home, you should not lie. Why? It is possible your spouse saw the murderer from, say, an upstairs window and went out a back door. If you lie, the murderer then might not go inside, but rather see your spouse on the lawn and kill her. Kant would hold you partly morally responsible for that murder. Similarly, we never can be sure of the consequences of our actions so the only (!) measure of whether we have acted rightly can be in the action itself, not its consequences (Kant, 1785).

21

5.1.3 Response to the Social Contract Theory View Billingham and Parr are concerned first with disproportionate punishment. From a social contract perspective, they identify ways shaming may be valuable, but raise major concerns about lack of accountability, disagreement among vigilantes, and punishment of socially disadvantaged individuals. Here, I will argue that these concerns are all related, and indeed justified, but that in some cases, accountability can be traced to an instigator vigilante. We can address the concerns by starting with disproportionate punishment. It is best handled with the first formulation of the Categorical Imperative. The potential shamer, before punishing the target, should ask himself: if everybody punished this target for the same reason I am, could the outcome be a universal law, or would it yield a logical contradiction? We can consider the Sacco case as an example. Biddle, seeing Sacco’s tweet, is considering whether to retweet it. He explained his reason for retweeting as: “Twitter disasters are the quickest source of outrage, and outrage is traffic.” (Biddle, 2014) So, his maxim could be formulated as: “I am reposting Sacco’s (or: a social norm violator’s) tweet, for prestige and career advancement.” Universalizing this maxim yields a contradiction: Sacco herself cannot perform the same action on that maxim. Sacco retweeting her message destroys her career standing, at least in the short term. So, Biddle does not pass the first formulation test.* More generally, online vigilantes tend to fail this test because their maxim is not universalizable to the target of their actions. The test exposes the fundamental contradiction of online shaming, articulated by Wynn: shamers say their goal is societal modification and norm enforcement, and it very well may be, as Chang and Poon have documented; but in reality, it may rather be an act of distributed cognition wherein each repost of the punishment draws attention to the vigilante who is now virtue signaling or simply gaining online reputation, even if anonymously. Norm enforcement ends up as a metaphor for a more general phenomenon of a distributed cognitive act. That is to say, vigilantes might say, “I am shaming this person to combat transphobia (or racism, gender- based violence, or otherwise)”, but actually the cause itself is only a metaphor for the motivation of a collective action driven, in reality, by simple hope for positive notoriety. As the cognition driving that storm is distributed, its movement is controlled by thousands, if not millions, of individual shamers, all guiding it along as do multiple hands guide an Ouija board pointer (or so say the makers of Ouija boards). Each shamer is a variable factoring into the direction of movement, and so, like other complex chaotic systems wherein we know neither the initial conditions nor reasoning of each node (each mind), the future is unpredictable. As there are so many participants, all of them share a small part in collective, distributed responsibility, which cannot be traced to a single person… except, in some cases, the instigator vigilante, the Sam Biddle, who was the first

* Grill insightfully argues back: “I can eat chocolate cake in order to have a pleasant experience, but not everyone can, because some are allergic to chocolate. That does not make my eating the cake for that reason immoral.” My retort is that eating chocolate cake differs in a relevant way from reposting Justine Sacco’s words. Chocolate cake, being an ethically inert non-agent, cannot eat itself. Justine Sacco, however, can do great harm to her own dignity if she follows Biddle’s maxim. In any case, Grill and I agree that this case also probably fails to pass muster of the second formulation, because Biddle is treating Sacco as a mere means, to the end of his own prestige and career advancement.

22 to violate the Categorical Imperative. Therefore: in some cases, you can assign moral responsibility for a shaming campaign to the instigator vigilante, who originated it with the first shaming post. The argument comes from the first formulation. The instigator vigilante says, “I will start a shaming campaign against anyone who does something unacceptable, on the maxim that doing so will improve my reputation (through virtue signalling, or virality and generating traffic, etc.).” The contradiction arises when the instigator vigilante inevitably does something unacceptable, as we all do at some point in our lives. Applying the maxim would ruin his reputation. Remember: online shaming is a process which stretches over time, and one case sets the precedent for future cases. Just as the Categorical Imperative harkened back to one principle from the Sermon on the Mount, it mirrors another: “Remove the log from your own eye, before you take the speck out of your brother’s” (Matthew 7:5).

5.1.4 Response to Feminist Views

5.1.4.1 Feminist Views For Online Shaming I will now analyze Goguen’s, and D’Amore and Jane’s feminist arguments in favor of online vigilantism. I hope to show that Kantians have a firmly grounded opposition to preemptive self-defense. Kantians would also oppose Goguen’s argument, but in their opposition, the limitations of Kantian ethics begin to take shape. Goguen’s argument in favor of online vigilantism is, in essence, that major punishment is justified for anyone who furthers a social injustice, so as to protect disenfranchised humans subject to that injustice. Goguen wants to create a spiral of silence. She wants the target to be stuck in a blended space of overwhelming topology and volatility, and furthermore be on display there, so that any future potential social norm violators are scared away from acting. It is a feminist argument, and also a utilitarian one: a target is made an example of, with great pain, in order to protect many more people from pain. At first glance this may appear to be a harsh infringement of the second formulation, using the target as a mere means, and indeed R.Dave worried that it was. A quick glance at historical parallels, however, shows Goguen to be more reasonable than she may seem. In Germany, it is highly punishable to display any Nazi symbol – even ironically – under section 86a of the Strafgesetzbuch, the criminal code; violators can be imprisoned (Bundesminsiterium, 2020). The argument behind this is that these symbols show support for an evil, highly unjust, and dangerous ideology. If someone tweets a Swastika, he risks having his agency revoked by the state. Just after World War II in 1945, however, there was a blanket forgiveness for soldiers who fought for the Nazis; only leaders were punished, with 97 convicted and sentenced. In Goguen’s defense, one might argue that with the current unregulated state of the internet, socially damaging views that promote, for example, racism, transphobia, or gender-based-violence can easily be expressed, and people holding them can build dangerous communities; the situation is analogous to pre-1945 Germany. In the absence of a state which might imprison someone who supports these views, we can use the mechanism of online vigilantism to silence him. Online vigilantism then, has an important regulatory function in the state’s absence. If it established that hate speech will provoke

23 automatic vigilante sanctions, then ideally no one would ever use hate speech, and so they would never be subjected to vigilantism, and therefore our worry about infringing the second formulation disappears. Indeed, in the perfect Kingdom of Ends for which Kant calls in the third formulation, hate speech would never occur, so he shares this ideal with Goguen. Kant was a staunch defender of free speech, but Varden (2010) makes a strong case that Kant would outlaw hate speech in order to remedy social inequalities. Let us, then, put a pin in Goguen’s argument, and return to it after we have considered some of the other feminist arguments for online vigilantism. D’Amore and Jane describe two very different phenomena – fairy tales versus Indian female gangs – but they are making the same point: women are not defended from abuse - -- above all, gender-based-violence – by formal justice systems, so they have the right to be extrajudicially physically violent offline, in order to bring about multiple types of justice: retributive, deterrent, and perhaps rehabilitative. Kantians have a more direct complaint about this kind of physical vigilantism in preemptive self-defense. It is the following: the target of the vigilantism has not agreed to a political system that would bring about that particular punishment if he broke the law. Remember that Kant insists that a person is never treated as a mere means, but always as an end in herself. Just violently punishing someone, in order to satisfy some other person’s desire (for justice, or safety, or perhaps even that person’s rehabilitation), is treating her as a mere means, if she has not rationally consented to that punishment. In a well-functioning justice system, participants consent to a system of punishment for laws; this sacrifice allows participants “to enjoy the benefits that a legal system makes possible” (Murphy, 1979, p. 83). Therefore, if they break the law, they have already consented to their own punishment. (Here, Kantianism merges with social contract theory, its deontological cousin.) So, offline violent vigilantism is forbidden by Kantians on the grounds that the target, regardless of the horror of his action, has not agreed to the vigilante system (by definition of vigilantism; it is justice carried out informally, without organized community agreement). Vigilante feminism, no matter how satisfying it is to our desire to see perpetrators suffer, remains a violation of a fair political order. These observations allow us to return to Goguen. Kant would certainly admire the end that Goguen imagines: a political order that preserved all kinds of social justice, either by law, or by the unlegislated choice of all citizens simply to conform to the ideals of social justice. If it were possible to evolve to that end without violence, Kant would absolutely support that evolution. Any kind of organized rebellion to that end would be impermissible. Goguen does not call for that organized rebellion; rather, she calls for online vigilantism to lead to that evolution. Unfortunately for Goguen, we are still stuck with the problem that the targets did not consent to their punishment. Although the punishment is online and not offline physical violence as in vigilante feminism, the facts of embodied cognition and linguistic obligation mean that verbal punishment is an act of physical punishment. Earlier, we asked if Turner and Persky punished themselves in their original actions. They might certainly have been able to predict that there would be a major public outcry over their actions (in Turner’s case, maybe as soon as he was caught by the Swedish students). However – regardless of how unfair the criminal justice system is to gender-

24 based-violence victim-survivors – Turner and Persky did not beforehand consent to a system of online vigilante punishment, and so from a Kantian perspective, that vigilante punishment violated the moral law. This shows either that 1) the reaction to Turner and Persky was immoral, despite Shame Nation calling it the most moral example of online shaming, 2) the Kantian perspective is morally inadequate, or 3) in the online era, we have a new judicial system, however disorganized, to which we all automatically consent by logging on. Here is where Powell (2015) is instructive.

5.1.4.2 Feminist Views Against Online Shaming Powell’s (2015) conception of online justice is not the absence of victim-survivors speaking publicly. Powell is highly attuned to the inadequacies of the legal system to deal with gender-based-violence, and so she calls for technolsociality for victim-survivors, with technology embedded in a social world, and voicing of “narratives to a trusted audience”. What is a trusted audience, from the point of view of human-computer interaction? It is one in which the victim-survivor knows that the distributed cognition, of the group to which she is telling her story, treats it with the principle of generosity. The audience gives a safe emotional space to the survivor. This means that the victim-survivor and her audience have developed certain rules for the use of their language, such that the audience does not linguistically obligate the victim-survivor to experience shame in her embodied interaction with that community. Rather, the opposite: linguistic obligation creates embodied self-esteem, self-acceptance, and healing, metaphorically and literally. This may even be a blended space, online and offline. Therefore: can you design a system that replaces disproportionate shaming of an individual with focusing on the lived experience of a real-world victim-survivor of injustice?

5.2 Engineering Solutions

5.2.1 Online Justice Systems A moment ago, I suggested that perhaps we may have a new judicial system – for which Powell calls – to which we consent by logging on. Kant’s insistence on universalizability and human dignity could be satisfied, if non-state online judicial systems were clearly organized to provide a consistent path of consent to the system by the user. Then, punishment could be permitted, on grounds that a “criminal”* would have already consented to it. We noted earlier that the Internet can be modelled as a Kantian Kingdom, and to create a Kingdom of Ends, a society needs moral organization. Powell’s insight is that the Internet community spans borders fluidly, which makes it very hard to regulate under the laws of a country with a functioning justice system – and even less so, with a poorly functioning one.

to take seriously the uptake of communications technologies for informal justice across diverse online spaces and spanning the boundaries of western democracies, is to displace fundamentally geo-spatial and conceptual divisions

* I put ”criminal” in scare-quotes here because in such a justice system, the notion of criminality may be very different from that of the traditional state.

25

between the formal justice of the ‘successful’ West and the informal, community or traditional justice of emerging, post-conflict or ‘failed’ states (Powell, 2015)

Powell does not propose a detailed model for this justice system, but she believes it would exist in subaltern counter-publics (Fraser, 1990), “in which culturally and discursively marginalized or silenced groups engage in resistant and/or critical speech that is ordinarily delegitimized and excluded from the public sphere.” (Powell, 2015) There would have to be a means for offender accountability, as this is also necessary to justice for victim-survivors. Social media critical theorists like Fuchs (2019, p.219-220) also call for such subaltern environments, although with intent to secretly organize rebellion against the state, which Kant forbids.* Kant’s argument goes like this: for a maxim to be effective, it must be universalizable, and a secret maxim is impossible to universalize (else it would cease to be a secret). Rebels who organize in a cognitively distributed counter-public, as Fuchs would have it, plan their rebellion in secret. “The illegitimacy of rebellion is thus clear from the fact that its maxim, if openly acknowledged, would make its own purpose impossible. Therefore [the maxim to revolt on occasion] would have to be kept secret” and would therefore be un-universalizable (Kant quoted in Beck, 1971). Therefore: an online justice system that you might build would contain a non-secret space of distributed cognition, wherein the language games agreed upon afford: 1) an honoring of victims’ narratives (giving them information, participation, voice, validation, vindication, and control), and 2) prior knowledge of the type and scale of punishment that an offender faces, so that every participant can consent to the system. Phillips and Chagnon (2018) hope that thereby, “rape justice can be achieved through solutions that meld the shared interests of antirape culture advocates, criminal justice reformers, victims/survivors, and the communities in which violence occurs.”

5.2.2 The Persuasive Technology of Shaming Detection Gallardo (2017) has a very different practical solution for human-computer interaction designers: shaming warnings. Acknowledging that social media and blog hosting platforms are not legally responsible in the United States for removing vigilante content, and that doing so might undermine their commitment to free speech on their platform, she suggests that ISPs create a choice architecture that forces the user to reflect upon the moral implications and consequences of shaming. This is therefore a persuasive technology. Hosts, she says, could “use algorithms and other filters to detect certain keywords or usernames that indicate that users are contemplating engaging in shaming activities”. Then, the user would receive a message such as:

Hi, [user]. It looks like your post is directed towards [norm violator]. Shaming can impose permanent and disproportionate consequences upon this individual. You can read [norm violator]’s public apology here. Please consider these implications before posting. (p. 742-743)

* Kant, however, famously admired the French and American Revolutions. Beck (1971) endeavors to work through this apparent contradiction.

26

Shaming warnings, then, linguistically obligate users to consider the reasons for their actions, rather than acting out of impulse – something of which Kant would approve. These warnings, although generated by a computer, nonetheless become a force for moral reflection in the distributed cognition of a social media or blog platform. The very refined natural language or image detection would be a difficult engineering challenge, although I do imagine that an artificial neural network, fed with enough data about what how shaming posts are written or appear as images, could be trained to detect many instances of shaming. In a slightly more futuristic scenario, Gallardo’s solution could be extended to detect the various unwanted phenomena we identified in our response to Wynn, such as whether guilt has been adequately proven, abstraction of false character traits, use of targets as metaphors, fake demands for apologies, criticism and reform, or automatic dismissals of apologies. Therefore: if you develop a platform allowing shaming to flourish en masse, train it also to detect shaming en masse, to raise public awareness of shamings’ moral dubiety.

5.2.3 The Shaming Circuit Breaker Your third engineering solution is entirely original to this paper. The kernel of the idea was suggested by my thesis advisor, Torbjörn Nordström. I expand upon it here. Stock markets have a mechanism officially known as a trading curb, but informally as a circuit breaker (Lee, 2020). When a market average starts to plummet suddenly as in a stock market crash, once it crosses a certain pre-determined threshold, trading is automatically stopped. The purpose of these circuit breakers is to stop panic selling, give traders a chance to rationally reflect on their actions. Trading then resumes after a few hours or days. Nordström’s suggestion is that viral shaming could also have a circuit breaker. I would propose this: when you, the human-computer interaction designer, are creating your platform, see if you can engineer a natural language processing mechanism that detects acts of shaming (or abuse of any kind, really) as Gallardo discusses above. Set a threshold for the number of posts and reposts that you will permit; one hundred, one thousand, 10 thousand – the best exact figure will depend upon the topology of your platform space, that is to say, how a target might experience the shaming in your particular platform (through push notifications? Emails? Augmented reality? etc.). The exact threshold might be something of a human judgment call, wherein you say, a person who experienced this much punishment would start feeling overwhelmed and unable to act freely on her moral conscience. The threshold might also depend on the exact content of the messages; more injurious punishment might be weighted more than less harmful. (Again: think about the balance of well-meaning versus ill-intentioned messages.) The fact that a circuit breaker can be turned off, when emotions have cooled, makes this solution palatable to those who think it might stifle free speech. You would need to determine a certain length of time before you again permit posts on the topic. I have not found any research in psychology that would be instructive here, but perhaps you could borrow timing from stock market circuit breakers, and not have to reinvent the wheel. Of course, we have a problem here that the shamers could just move to another platform, which is more committed to virality than to user safety. On the other

27 hand, as Gallardo points out, given the increasing social awareness of the dangers of shaming, having an anti-shaming mechanism may make your platform attractive to consumers. Therefore: Consider a shaming circuit breaker for your platform.

5.3 Conclusion: All Your Tools We can now summarize, in paragraph form, all the underlined tools this paper has provided for you, the designer of a social network, blog, or other potential online shaming host. The key concepts you can use for evaluating the ethics of shaming can be phrased as the following questions: Could the maxim behind a shaming action be willed to be a universal law? Are the target, or the shamers, being treated as ends in themselves, or mere means? Are autonomy and dignity preserved? Are shamers and targets following rules that everyone could follow in a Kingdom of Ends? Is the shaming, perhaps because it is ubiquitous in a blended space, causing the target or shamer to lose his free moral choice? What emotions and physical experiences is the shaming evoking in all parties? Is your platform using a metaphor to describe itself which deceives users into misunderstanding the power of their actions? Is the target becoming a metaphor for the perceived injustice? The key guidelines can be phrased as imperatives: Any shaming you allow on your platform should be target at a person whose guilt has been proven. You should prevent shamers from abstracting out false character traits about a target based on a few details. People should be used as a metaphor only if they have consented to this. Fake demands for apologies, criticism, or reform from the target should be stopped, as should automatic dismissals of targets’ apologies. Do not permit ill-intended shamers to overwhelm well- meaning ones. Consider the long-term as well as short-term effects of the shaming. If you need to assign moral responsibility, one possibility is the instigator vigilante. Above all, try to design a system that replaces disproportionate shaming with a focus on the lived experience real-world victim-survivors of injustice. The key engineering solutions are three. 1) Consider participating in a project to build a new, online justice system, which honors victims’ narratives, and to which everyone consents to the type and scale of punishment of norm violations. 2) Try developing natural language and image detection algorithms that detect en masse typical patterns of shaming, and then prompt the potential vigilante to reflect first upon his moral obligations. 3) See if you can build into your platform a shaming circuit breaker. Guide your users, with their consent of course, back to a path of rational thinking, if a case of online shaming breaks out.

6. Discussion The primary aim of the paper was to provide tools to help human-computer interaction designers build systems that are ethical with regards to online shaming. We summarized these tools in the conclusion. How is this advice, and this entire thesis, situated in the greater dialogue about online vigilantism? Outside of the academic research we referenced in this paper, there is a much larger popular dialogue about online vigilantism, online shaming, and related phenomena.

28

Much of that popular dialogue is motivated by peoples’ rough personal interests, rather than nuanced application of ethical reasoning. Netizens cheer when a social norm violator – whether a public or a private figure – is brought down by an internet mob (Spaeth, 2016). Or, they curse the mob, reading the worst of human nature into the actions of online vigilantes (Molloy, 2018). This thesis showed that the type of online shaming we addressed is not the black-and- white, good-or-evil phenomenon that popular dialogue tends to call it. It is a complex, chaotic set of events, in which many people engage in good conscience, with the aims of improving the world, protecting themselves and particularly socially disenfranchised groups, and even perhaps creating moral reform in the target. Other shamers have abysmal malintent. When we draw from the profound ideas of a great Enlightenment philosopher, and state-of-the-art human-computer interaction theory, we have a much richer model of online shaming, as a moral object and process. So, we looked at online shaming using Korsgaard’s Kantian methodology, informed by Wittgenstein’s private language argument and how it helps to model several concepts in human-computer interaction theory. This approach had multiple limitations. First of all, it did not provide a solution to the problem of the transitive property of cancellation. Our conclusions also conflicted with current general public moral sentiment about the shaming of Turner.* The rigorous demands for control that our Kantianism put on the designer in the response to Wynn are impractical for large platforms, and may conflict with any philosophy insistent upon freedom of speech. Like most papers on ethics, this one was not intended to solve all its problems permanently. Rather, it proposed some possible solutions, grounded in a moral theory justified in solid meta-ethics which share characteristics of human-computer interaction theory. My hope is the analysis spurs free moral reflection in the reader about online shaming. I also hope that it can be used as a reference point for a Kantian view of online shaming, for future academic research into the phenomenon from other ethical traditions such as consequentialism, virtue ethics, more feminist and critical theoretical scholarship, and non-Korsgaardian interpretations of Kant.

* There are nonetheless a few commentators who oppose Turner’s shaming; see Vitiello (2017) and Sprecher (2016).

29

7. References

Aune, B. (1991) Knowledge of the External World, London: Routledge.

Bae, G. (2020) Chanel Miller plaque installed at Stanford after 4 year delay. The Campanile, April 15, 2020. https://thecampanile.org/2020/04/15/chanel-miller-plaque- installed-at-stanford-after-4-year-delay/

BBC (2019) Aaron Persky: Brock Turner judge fired from coaching girls. Sept. 12, 2019.

Beck, L. (1971) Kant and the Right of Revolution, Journal of the History of Ideas, Vol.32:3.

Benyon, D. (2019) Designing User Experience: A guide to HCI, UX, and interaction design, 4 ed., Pearson, 644 p.

Biddle, S. (2014) Justine Sacco Is Good at Her Job, and How I Came To Peace With Her, , https://gawker.com/justine-sacco-is-good-at-her-job-and-how-i-came- to-pea-1653022326

Billingham, P. and Parr, T. (2019) Online Public Shaming: Virtues and Vices, Journal of Social Philosophy, Journal of Social Philosophy, Vol. 51:1.

Blackwell, A. F. (2006) The Reification of Metaphor as a Design Tool, ACM Transactions on Computer-Human Interaction, Vol. 13:4, pp. 490-530.

Bundesministerium der Justiz und für Verbarucherschutz (2020), Strafgesetzbuch, https://www.gesetze-im-internet.de/stgb/

Carson, T. (2010) Kant and the Absolute Prohibition against Lying, Lying and Deception: Theory and Practice, Oxford University Press.

Chandler, S. (2016) How Social Media Policing and Online Vigilantism May Increase Wrongful Convictions, Truthout, https://truthout.org/articles/how-social-media- policing-and-online-vigilantism-may-increase-wrongful-convictions/

Chang, L. and Poon, R. (2016) : Attitudes and Experiences of University Students Toward Cyber Crowdsourcing in Hong Kong, International Journal of Offender Therapy and Comparative Criminology, Vol. 61:16, p. 1912- 1932.

30

Cuevas, M. (2016). Musician who defended Brock Turner feels backlash. CNN, June 10, 2016. https://edition.cnn.com/2016/06/08/us/stanford-rapist-defender- backlash/index.html

D’Amore, L. (2017) Vigilante Feminism: Revising Trauma, Abduction, and Assault in American Fairy-Tale Revisions, Marvels & Tales, Vol. 31:2, p. 386-406.

Deruy, E. (2018) Meet Michele Dauber, the woman who won the Persky recall, , June 10, 2018.

Dourish, P. (2004) Where the Action Is: The Foundations of Embodied Interaction. MIT Press: Cambridge. 233 p.

Fogg, B.J. (2008) Mass Interpersonal Persuasion: An Early View of a New Phenomenon, International Conference on Persuasive Technology, Vol. 5033

Fraser, N. (1990) Rethinking the Public Sphere: A Contribution to the Critique of Actually Existing Democracy, Social Text, No. 25:26

Fuchs, C. (2017) Social Media: a critical introduction, SAGE, 386 p.

Gallardo, K. (2017) Taming the Internet Pitchfork Mob: Online Public Shaming, the Viral Media Age, and the Communications Decency Act, Vand. J. Ent. & Tech. L., Vol. 19 p. 721-1085.

Goguen, S. (2013) The Ethics of Public Shaming, Feminist Philosophers, https://feministphilosophers.wordpress.com/2013/03/22/the-ethics-of-public- shaming/

Guyer, P. (2006) Kant, London: Routledge.

Hartline, S. (1997) Intimate Danger: The Case for Preemptive Self-Defense, Feminist Ethics and Social Policy, Bloomington: Indiana University Press.

Hou, Y., Jiang, T., & Wang, Q. (2017) Socioeconomic status and online shaming: The mediating role of belief in a just world, Vol. 76 p. 19-25.

Huffman, E. (2016) Call-out culture: How online shaming affects social media participation in young adults, Thesis, Gonzaga University.

Hursthouse, Rosalind & Pettigrove, Glen (2018) Virtue Ethics, The Stanford Encyclopedia of Philosophy (Winter 2018 Edition), https://plato.stanford.edu/archives/win2018/entries/ethics-virtue/

31

Jane, E. (2016) Online misogyny and feminist digilantism, Continuum: Journal of Media & Cultural Studies, Vo. 30 Issue 3, p. 284-297.

Jordan, J. & Rand, D. (2019) Are You “Virtue Signalling”? , March 30, 2019.

Kant, I. (1785) Grundlegung zur Metaphysik der Sitten, Projekt Gutenberg, https://www.projekt-gutenberg.org/kant/sitte/sitte.html

Kant, I. (1797) Die Metaphysik der Sitten, Zeno.org, http://www.zeno.org/Philosophie/M/Kant,+Immanuel/Die+Metaphysik+der+Sit ten

Korsgaard, C. (1996) The Sources of Normativity, Cambridge University Press, 217 p.

Korsgaard, C. (1996b) Creating the Kingdom of Ends, Cambridge University Press, 442 p.

Laidlaw, E. (2017) Online Shaming and the , Laws, Vol. 6:1.

Lee, Y. (2020) ‘Circuit breakers’ are being triggered across Asia — here’s how they differ across major markets. CNBC, March 13, 2020, https://www.cnbc.com/2020/03/13/what-are-circuit-breakers-and-how-they- differ-across-asian-markets.html

Levin, S. (2016) Stanford sexual assault: woman who blamed victim for drinking apologizes, , June 9, 2016.

Marsh, F. & Wager, N. (2015) Restorative justice in cases of sexual violence: Exploring the views of the public and survivors, Probation Journal, Vol. 62:4.

Molloy, M. (2018). Online shaming: The dangerous rise of the internet pitchfork mob. The Telegraph, June 25, 2018, https://www.telegraph.co.uk/news/2018/06/25/online- shaming-dangerous-rise-internet-pitchfork-mob/

Murphy, J. (1979) Kant’s Theory of Criminal Punishment, Retribution, Justice, and Therapy, Dordrecht: Reidel.

Noelle-Neumann, E. (1993) The Spiral of Silence. University of Chicago, 269 p.

O'hara, K., Kjeldskov, J., & Paay, J. (2011) Blended interaction spaces for distributed team collaboration, ACM Transactions on Computer-Human Interaction, 18(1), 1-28. doi:10.1145/1959022.1959025

32

Phillips, N. & Chagnon, N. (2018) “Six Months Is a Joke”: Carceral Feminism and Penal Populism in the Wake of the Stanford Sexual Assault Case, Feminist Criminology, Vol. 15 p. 47-69.

Powell, A. (2015) Seeking rape justice: Formal and informal responses to sexual violence through technosocial counter-publics, Theoretical Criminology, Vol. 19(4). p. 571- 588.

Rennison, C. (2017) I’m the professor who made Brock Turner the “textbook definition” of a rapist, Vox, https://www.vox.com/first-person/2017/11/17/16666290/brock- turner-rape.

Rohlf, Michael (2020) Immanuel Kant, The Stanford Encyclopedia of Philosophy (Spring 2020 Edition), https://plato.stanford.edu/archives/spr2020/entries/kant/

Ronson, J. (2015) How One Stupid Tweet Blew Up Justine Sacco’s Life, The New York Times Magazine, Feb 15, 2015, p. 20.

Sangal, A. (2019) Chanel Miller says her assault by Brock Turner reflects a culture of ‘male sexual entitlement’ that goes beyond colleges, The Mercury News, https://www.mercurynews.com/2019/11/29/chanel-miller-says-her-assault-by- brock-turner-reflects-a-culture-of-male-sexual-entitlement-that-goes-beyond- colleges/

Scheff, S., Schorr, M., & Lewinsky, M. (2017) Shame Nation : The Global Epidemic of Online Hate, Sourcebooks.

Scruton, R. (1982) Kant, Oxford University Press, 99 p.

Scruton, R. (1996) Modern Philosophy: An Introduction and Survey, London: Bloomsbury.

Sommer, A. (2015) Facebook Shaming-turned-suicide Isn’t Just for Teenagers Anymore, Haaretz, https://www.haaretz.com/.premium-suicide-by-facebook-shaming- 1.5366184

Spaeth, E. (2016) We gathered the best of Brock Turner burns on social media, The Tab, September 12, 2016, https://thetab.com/us/pitt/2016/09/12/gathered-best- brock-turner-burns-social-media-3316

Sprecher, T. (2016) Has the Shaming of Brock Turner Gone too Far?, The George Washington University https://historynewsnetwork.org/article/163043

33

Sussman, D. (2008) Shame and Punishment in Kant’s Doctrine of Right, Philosophical Quarterly, Vol. 58:231.

Timmermann, J. (2000) Kant’s Puzzing Ethics of Maxims, The Harvard Review of Philosophy, Vol 8, p. 39-52.

Trottier, D. (2016) Digital Vigilantism as Weaponisation of Visibility, Philosophy & Technology, Vol. 30, p. 55-72.

Valenti, J. (2017) Full Frontal Feminism: A Young Woman’s Guide to Why Feminism Matters, Berkeley: Seal Press.

Varden, H. (2010) A Kantian Conception of Free Speech, Freedom of Expression in a Diverse World, Springer.

Vingiano, A. (2013) This Is How A Woman's Offensive Tweet Became The World's Top Story, BuzzFeed.News, December 21, 2013, https://www.buzzfeednews.com/article/alisonvingiano/this-is-how-a-womans- offensive-tweet-became-the-worlds-top-s

Vitello, M. (2018) Brock Turner: Sorting Through the Noise, The University of the Pacific Law Review, Volume 49:3, p. 631-659.

White, A. & Rastogi, S. (2009) Justice by Any Means Necessary: Vigilantism among Indian Women, Feminism & Psychology, Vol. 19:3, p. 313-327.

Wynn, N., (2020) Canceling, Contrapoints, https://www.contrapoints.com/transcripts/canceling.

8. Appendix Figure 1. Justine Sacco’s original tweet.

34

Figure 2. Brock Turner’s photo next to the definition of rape in a criminology textbook.

35

Figure 3. A May 14, 2020 public death threat against Brock Turner.

Figure 4. Tweets calling for Wynn’s cancelation.

36

Figure 5. Plaque installed at Stanford with quote from Chanel Miller’s victim impact statement. (Source: Stanford Daily, February 24, 2020)

37