1

“Automated Meme Magic: An Exploration into the Implementations and Imaginings of Bots on Reddit”

Jonathan Murthy | [email protected] | 2018 2

Table of Contents

Acknowledgments...... 3 Abstract...... 4 1.2 Research Questions...... 6 1.2.1 Why Reddit...... 7 1.2.2 Bots...... 9 1.3 Outline...... 10 2 Bot Research...... 11 2.1. Functional Bots...... 13 2.2 Harmful Bots...... 14 2.2.1 The Rise of Socialbots...... 16 2.2.2 The Rise of Political Bots and Computational Propaganda...... 19 2.3 Sockpuppets and Cyborgs...... 21 3 Reddit...... 23 3. 1 Background on Reddit Structure, History, and Culture...... 23 3.2 Reddit’s Toxic Technocultures, Gamergate, The Fappening...... 26 2.2.2 /r/The_Donald...... 28 2.2.3 Reddit Bots...... 29 3.2 Independent Research by Redditors...... 31 3.2.1 Reddit Transparency...... 37 4 Methodology and Findings...... 38 4..1 Google Search Findings...... 40 4.2 Bot Lists...... 51 5 Discussion...... 65 5.1 Automation, Technology, Culture, and Economics...... 65 5.2 Taxonomy of Bots...... 66 5.2.1 Visible bots...... 67 5.2.2 Invisible Bots...... 68 5.3 Suggestions...... 68 6 Conclusion...... 70 Works Cited...... 72 3

Acknowledgments

There are several people I would like to acknowledge and thank for their support, encouragement, and insights. First, I would like to thank all the professors and students I have had the pleasure of working with over the course of this program. Marc Tuters, my thesis supervisor, who encouraged me to pursue this research. Sal Hagen, for writing various SQL scripts which helped me find interesting data. My mom, for the lifelong, love, and support she has given me over the years. To my sisters and brother who encourage me to follow my dreams. To my father, who is no longer with us and the unconditional pride he had for me. And to Laura, for talking to me everyday, for challenging me, loving me, being patient with me, and the amount of growth I have experienced because of you. 4

Abstract

The allegations of bots being used as deceptive, persuasive, manipulative, unseen, networked machines, seeded inside digital environments to control, guide, subvert, or otherwise alter the public discourse, is a prevalent topic around the areas of new media, political science, human-computer interaction, journalism, computational propaganda, science and technology studies, and many other areas of interest. Recent instances where bots have arisen and have caused alarm are typically situated around political elections, but also have been seen in some areas related to cryptocurrencies.

The effects of bots are most commonly seen via social networking sites where they are capable of exploiting homophilic algorithms and direct content toward particular groups of people. Essentially, visibility is a means toward shifting normative discourse, maintaining popularity or controlling the circulation of a particular piece of information is susceptible to manipulation. Visibility is also directly linked to profit as well.

This thesis will be an attempt to present the history of bot research, classifications for different bots which display specific attributes, and background on the content aggregator site Reddit.com. A first focal point of this thesis will be the use of social bots, political bots, and Search Engine

Optimization (SEO) strategies which entails the use of marketing techniques which seem at odds with

Reddit’s behavioral policy. The second focus will revolve around particularities of Reddit and how bots are used on the site.

Discussion points will focus on a taxonomy of the observed bots on Reddit, comments and

SEO models, Reddit’s culture and internal governance, and the political, economic, and cultural implications of bot and bot like activity.

Less research has been considered for Reddit as a point of investigation. Hopefully, this thesis will act as a stepping stone into further research on an increasingly prevalent online environment and topic. 5

1 Introduction

Is it possible that I did not write this thesis? I, Jonathan Murthy? Or, perhaps, is it possible that someone else wrote it? Is it possible that something else wrote it? If you had an infinite amount of monkeys typing at an infinite number of typewriters, typing words at random, could they not write this thesis? Could my digital profile be fabricated and used to gain credibility? Could my style of writing be derived from a corpus of previously consumed works around a particular area of interest in order to mimic natural knowledgeable language? How much would it take to convince you that I am a human being presenting credible information? In thinking about these questions (despite their hyperbolic nature), we can then think, ‘what is required, technically speaking, to mediate exploitable abstractions between you and I’, and (perhaps more importantly) ‘why would I do this?’ Despite these ponderings, sowing seeds of doubt toward me and this thesis’ authenticity is not what this thesis is about, but attempts to acts as an image to enter into a world of algorithmically mediated communication, automation, and online identities. This is then compounded by the circulation of misinformation, fake news, visibility manipulation, directed marketing, and other issues which concern public discourse around digital media. While there are other factors that can contribute to these same issues, I will be focusing on what is colloquially referred to as ‘bots’, ‘botnets’.

But what exactly are bots and what are they capable of doing? In computing, a bot is “an autonomous program on a network (especially the Internet) that can interact with computer systems or users, especially one designed to respond or behave like a player in an adventure game

(Google.com). The term’ bot’ is a shortening of the word robot, derived from the Czech ‘robota’ meaning “forced Labor” (Google.com), and this seems to be in reference to the ability programmability of bots (Geiger, 2014). These three notions (that bots interact with both humans and computers, that they are designed to mimic human behavior, and can automate tasks) makes for a precarious state of affairs regarding what we see online and what information get circulated (Wooley and Howard, 2016;

Howard et al., 2017; Forelle, 2015). Various industries and institutions recognize this reliance on automated tasks within networked systems, and what seems to be a growing department in every market for the ability to automate data heavy tasks (Geiger, 2014). This translates into how visibility, 6 virility, and amounts of engagement are transformed into profit by marketing firms described as Search

Engine Optimization (SEO) (Heder, 2018). Also, It is not solely economic reasonings for the deployment and use of bots, as there are social, political, experimental, and other reasons for their use. According to the 2016 Incapsula Bot Traffic report, bots makeup 51.8% of internet traffic where

22.9 % of total traffic is classified as “good” bots and 28.9% are classified as “bad bots”

(www.incapsula.com). But what determines the moral signifier of ‘bad’ or ‘good’?

This thesis will be an attempt to present to the best of my knowledge the ways in which bots are implemented and deployed in online environments. Information regarding bots on Twitter and

Facebook will be presented and I will contribute original research involving bot activity on Reddit.

Allegations of bots being used to control and manipulate the normative discourse and circulate misinformation is a prevalent topic in many different areas of research and study (Woolley and

Howard, 2017). A majority of that research has been focused on Facebook and Twitter especially when we consider events like the 2016 US Presidential Election, where Donald Trump relied heavily on an online campaign and the use of social media (politico.com). Other online spaces, such as

Reddit, have not been given the same amount of attention despite there being what seems to be equally dubious activity and even what many consider to be the largest gathering space of Trump supporters in the subreddit /r/The_Donald (Zannetou, 2017; qz.com; thehill.com ). Reddit has also been observed to have a techno-libertarian sentiment, a particular culture of self-governance, a controversial history, and what many consider to be an easily manipulable voting system, all of which make it an interesting point of observation (Massanari 2015). For these reasons, I have chosen to focus on bot activity Reddit in order to explore its particularities and bring an area of study to a less publicized site of observation. The rest of this introduction will cover some of the research questions and objectives of this thesis, as well as an outline for the following chapters and sections.

1.2 Research Questions

While investigations into various methods of manipulation on Facebook and Twitter are more prolific, other sites that have equally, if not more, precarious policies and governance systems are 7 overlooked. The research presented in this thesis is investigative and exploratory in nature, first attempting to answer the question of ‘how do bots operate on Reddit as opposed to Facebook and

Twitter?’ This breaks down into what are the capabilities, functionalities, and imaginings of bots in a broad view. From there we will move into the particularities of how Reddit as site functions both technically and culturally. What is it about Reddit and bots in general that allows them to operate in the way that they do? What are the implications of their activity and what can be done to lessen harmful abuse? I will now present brief summaries for the theoretical reasonings for choosing Reddit as a site of observation and bots as an object of investigation. Longer, more in depth backgrounds will be given on both Reddit and bots in late chapters.

1.2.1 Why Reddit

While Reddit does not seem to receive as much attention as social networking sites such as Facebook and Twitter, it is still the 4th most visited site in the U.S. and the 6th most visited site worldwide with

58.5% of traffic coming from the U.S. followed by 7.6% from the UK and 6.1% from Canada

(www.alexa.com). Reddit is also mired in controversial events, including but not limited to Pizzagate,

Gamergate, and The Fappening (thenewyorker.com; Masanari, 2015).

Pizzagate consists of a conspiracy theory accusing the Clinton’s of being part of a child sex trafficking operation centered around a Washington D.C. pizzeria by the name of Planet Ping Pong.

The theory developed on 4Chan’s /pol/ board after leaked emails from the Clinton campaign circulated and eventually made its way onto /r/The_Donald in November of 2016, leading up to the US presidential election (Malmgren, 2017; digitalmethods.net). The event culminated when an unidentified man in his late 20’s opened fire on the pizzeria, and where it was subsequently revealed that the theory was false (snopes.com). This event demonstrated how the spreading of misinformation can have violent implications as well as examining the role anonymous and pseudonymous sites like

4chan and Reddit play in the spread of misinformation. Reddit in particular is considered to have a playful and facetious but scientific and logical sentiment, where the information is shared quite virially, but can be erroneous, antagonistic, and even unlawful (Masanari, 2015; Milner, 2015). 8

Gamergate refers to the harassment and alienation of women in the video games industry which initially began as the harassment of Zoe Quinn, but took on the sentiment of misogyny and sexism underneath a growing distrust in video game journalism and the industry as a whole.

(Masanari, 2015). The Fappeing refers to the circulation of private celebrity photos which which were stolen from Apple’s iCloud service (Masanari, 2015). Those photos circulated on Reddit with such a high amount of activity that one subreddit moderator described “insane traffic ”due to the hack.

(Masanari, 2015). Both Gamergate and Fappening will be elaborated in a later chapter as well, but what these event demonstrate is a proclivity toward spreading information with an ideological identity, one most closely associated with the American alt-right. All three of these events had dedicated subreddits where they were discussed and where information particular to the event was shared.

These subreddits have since been banned, but until after a substantial amount of time.

/r/The_Donald is considered to be the largest pro-Trump online community and, along with other subreddits such as /r/bitcoin, have been accused of vote brigading and vote nudging, a phenomena where a group of individuals vote in a particular way in order to gain visibility on Reddit’s message board (gizmodo.com; medium.com). Vote brigading is strictly against Reddit’s policy, while vote nudging is a more common practice which exploits how Reddit weighs earlier votes against later votes, enabling content which is voted favorably earlier a longer lifespan (reddit.com). By abusing this exploit, a piece of content’s lifespan can increase exponentially and garner a great deal of attention.

The /r/The_Donald subreddit has also been at odds with the Reddit administration for some time where a tenuous relationship regarding free speech, hate speech, censorship, and authenticity hang in the balance. Many calls for the banning of the /r/The_Donald have circulated, but Steve Hoffman

(/u/spez), current CEO of Reddit, has been reluctant levy the punishment despite having banned other problemantic subreddits (vox.com). /u/spez has also received a large degree of hate from

/r/The_Donald’s community involving censorship, the suspension of accounts, the purposeful downvoting of /r/The_Donalds content, and the editing of user comments. This last event involved an script which swapped /uspez’s account name with the name of the person making the comment. This resulted in anger directed toward /u/spez being redirected to the person who first posted, but the event left a bad impression on an already tenuous relationship. The Reddit administrative team has changed some of their interface and homepage to include /r/popular alongside the previous /r/all page in order 9 to satisfy the calls for removing /r/The_Donald, and /r/The_Donald’s claims of censorship. Other filtering options like ‘best” and “hot” have been implemented for similar reasons in order to limit or better curate the content that rises to the top of other pages (reddit.com).More on Reddit’s background, history, and structure will be presented in a later chapter.

These examples demonstrate the technical aspects of Reddit’s infrastructure and cultural sentiment, how information circulates and how content can be controlled. Reddit has a history of controversy and seems to be an intermediary space between mainstream media sites and the darker corners of the web such as 4chan (thehill.com; Malmgren, 2017). But it is important to understand how events similar to these occur on Reddit, and what functionalities are vulnerable to exploitation. How does Reddit’s ranking algorithms, technical nature, and culture contribute and influence the way information is shared? This leads me into the next aspect of this thesis: the role of computation and automation (i.e. bots).

1.2.2 Bots

In order to begin exploring the operationalization of bots in digital spaces, a distinction should be made here about what bots are, what they do, what they are capable of, and where they are placed in the discussion over social media, misinformation, automation, data, and networked activity. Bots are capable of performing infrastructural tasks. These functional bots are helpful in their imaginings Some of the ways in which bots are used include scraping, crawling or cleaning websites, capturing and performing web analytics, assisting in customer services (Geiger, 2014), assisting in data research and many other administrative tasks. But in their utile capacities and given the right environment, their abuse seems to almost inevitably follow, especially in cultures where the hacking of systems is a pillar of that culture.

This is where the notion of harmful or malicious bots can begin to be explored within the context of Reddit. The use of automated systems on social networking sites seem to be responsible for purposeful swaying, manipulating, guiding, drowning, or altering the public discourse of political elections online (Woolley, 2017). Even functional bots have been seen erring on the side of spreading 10 misinformation during the Boston Marathon Bombing (Cassa et al, 2013) Those bots were considered benign and were more a product of a faulty architecture, spreading false information about suspects.

Similarly, Twitter’s experimental chatbot, Tay, after digesting a corpus of tweets from other users, began tweeting hateful, racist, misogynistic, pro-Hitler content (Neff and Nagy, 2016). The more deliberate bot which attempts to hide its identity in order to astroturf, and present itself as contributing to a normative discussion. Social Bots are by definition deceptive, seeming human while in actuality being automated (Ferrara et al., 2014, Boshmaf, 2011). When Social Bots take on a specifically political sentiment they are considered to be political bots. Political bots have been seen to populate online spaces with partizan ideologies governing what kinds of things they post (Howard and Woolley,

2016). Numerous investigation and research into the influence political bots in the wake of the U.S.

Presidential election have been undertaken to gauge the extent with which their environments were infiltrated, as well in other countries around the world (Forelle et al., 2017; Howard et al., 2017

Schafter et al., 2017, Wooley, 2017).

Bots are not just capable of circulating misinformation, but are also capable of reinforcing cultural sentiments. In what Lawrence Lessig refers to in his book Code and Other Laws of

Cyperspace, the code acts as an infrastructure that has the ability to enforce social normative and governmental rules that is omnipresent, omnitemporal, and automated (Lessig, 1999). Because an automated process can run indefinitely, their presence can allow for a normative discourse to emerge that is encouraged by that process. We will see examples of this in a later chapter on /r/The_Donald.

1.3 Outline

The following chapters and sections of this thesis will review some of the classifications and methodologies of bot research, namely: functional bots, harmful bots, social bots, political bots, and sockpuppets. This chapter on bot research is informed by publications from accredited academic institutions. The sites of investigation where bots are present in include Wikipedia, Twitter, Facebook, and Reddit. The section on sockpuppets will elaborate on SEO strategies and involve various social media accounts which are operated by humans, but where activity is bought and sold. 11

The following chapter will present background and history on Reddit as well as its infrastructure. This will include aspects of Reddit’s site specificity, content policies, and controversial events. Specifically, research regarding Gamergate, The Fappening, and /r/The_Donald will be presented. Another section involving an ethnographic studying on the requesting and creation of bots will be presented

Reddit is considered to have a high degree of internal governance, and has produced non- accredited research regarding bots on the site. Reddit has also released its 2017 Transparency Report where it documents many suspicious accounts and the subreddits they were seen most active in.

Original research is also presented in a Methodology and Findings chapter where particular subreddits are scraped in order to identify bot accounts. Google BigQuery is used to perform the data scrape where a data set of Reddit posts and comments dating back to 2007 is available. Those accounts are then looked at individually in order to gain a sense of the type of activity they partake in as well as to if there are specific attributes to the community those bots are found in.

The final two sections of this thesis will include a discussion and conclusion where a classification and taxonomy of the bots found on Reddit will be presented. I will also address some of the more problematic aspects of Reddit’s culture and structure as well as how SEO and growth hacking business models encourage the exploitation of digital architectures.

Little research has been done regarding the research of bots and bot like activity on Reddit. It is highly encouraged that future research build off what is presented here in order to gain better insight into an increasingly prevalent digital environment and equally prevalent digital artifact. Due to the subject matter and Reddit as a site of observation, some of the content present will have explicit language or subject matter. The names of the accounts that will also be mentioned have been removed for privacy, unless they are immediately related to a bot account.

2 Bot Research

This section of the thesis will cover a broad range of topics surrounding the research of bots, the methods used for detecting them, and some of socio-technological and political implications 12 therein. This section will contain a collection of papers that have been published with academic accreditation in order to present background on the topic as well as situating the work done in this thesis among the academic community. This is an important aspect of this thesis because there will be independent research presented later on.

Some of the earlier methods for detection and classification have proven over the evolution of these technologies to not have the same kind of efficacy they would have had in the early years

(Ferrera, 2014). There seems to be what Ferrera et al. refers to as a technological “arms race” between developers, academics, and those who work to detect and prevent the abuse of these systems against those who wish to abuse those systems(Ferrara, 2014; Boshmaf, 2011). Because of this, methodologies for bot detection have progressed hand in hand with methods for concealing bot activity.

The following section will review some of the literatures put forth researching, classifying, detecting, and observing the effects of bots in different online environments. This will include the distinction between benign or functional bots and the deployment of malicious or harmful bots. From there we will move into research done on the specific categorization of socialbots and various methodologies for determining whether certain accounts are in fact automated. From social bots, we will look at the deployment of bots that seem to be related to political elections or politics in general.

These are referred to as political bots (Woolley, 2015). The last section of this chapter will focus on accounts that are operated by human users. While this seems at odds with the topic of this thesis, these sockpuppet accounts (accounts which are owned and operated by humans, but produce inauthentic activity) are instrumental for astroturing campaigns, the concealment of the manipulation of visibility, and the encouragement of such activity online in what Herder refers to as emerging “black markets” for the buying and selling of likes, comments, upvotes, and other activities. This chapter represents a small amount of the research done on bot related issues and encourages further investigation. 13

2.1. Functional Bots

Within the potential of automated software lies a seemingly inherent threat that automation can not only replace us, but trick us into not knowing who or what we are interacting with. Alan Turing remarked on the capabilities and requirements necessary for an automated piece of software, mediated through communicative technologies, to deceive humans into thinking they too were humans through his eponymous “Turing Test” (Turing, 1950). But while the Turing Test was a specific type of imagined technology, Turing is also responsible for using automation to decipher German communications during WWII. Like any technology, automated software agents are tools, much like the systems they inhabit. The system will rarely be unexploitable, if ever, and the tools which constitute that system may be used against it. It is for the sake of not completely vilifying the system or the digital entity therein, but rather to present some of the information and implementation of bots that contribute and assist in problem solving, information gathering and system maintenance.

Automation allows for a greater number of tasks to be performed with speeds orders of magnitude greater than humans are capable. This includes assisting data researchers, data scientists, and other data intensive professions in their work. The affordances and capabilities of bots allow researchers to scrape vast datasets, scanning for particularities in keywords or content, clustering associated entities, and overall allowing for the refinement of seemingly unintelligible data. This allows researchers to perform analysis and visualize information in comprehensible ways that, if those same researchers were to do manually, looking for patterns, would take many times over the amount of work an automated script could accomplish.

Bots are also capable of performing administrative tasks for particular websites, provide chat supported customer service, and aggregate content. The administrative capacity of bots allows us to see the ways certain websites are able remain functional through the flagging of illicit content, performing updates, and overview of websites operations. Chatbots provide customer service by responding through natural language processes in order to assist customers in matters related to website. Content aggregates are capable of collecting news stories and related pieces of media for personal or analytical purposes. Looking at how and in what capacities functional bots operate gives 14 us a better sense of the capabilities of automated software and the tasks that they are most commonly associated with doing.

An example of this taking place is how Wikipedia operates (Geiger, 2011). Bot accounts for a large portion of edits to Wikipedia pages and have even been seen to play a larger role in less developed countries (Niederer and van Dijck, 2010). The extent of which bots perform edits was on full display when in late 2006, the english version of Wikipedia held its third annual election for its

Arbitration Committee. An editor bot by the name of AntiVandalBot, was nominated to the committee due to its larger number of edits, much to the chagrin of the other members (Geiger, 2011).

AntiVandalBot’s candidacy was later revoked, but this brought into the question the role bots play within these systems. This brings Geiger, in a paper titled “Lives of Bots”, to consider the bot not only as a digital artifact or object, but as an actor as well (Geiger, 2011; Hegelish and Janetzko, 2016). In

Niderer and van Dijck’s 2010 “Wisdom of the Crowd or Technicity of Content? Wikipedia as a

Sociotechnical System” also calls into question the scale or extent editors on Wikipedia frame that information, as well as what kind of geographical, political, and economic influences can be derived from using editor bots.

While these technologies are becoming more rooted in the infrastructure of various systems, it then becomes imperative to understand how they can be used to exploit that system. From here, the potentialities of automated software agents become more precarious and imaginings for the abuse and deployment of malicious or harmful bots begin to take shape.

2.2 Harmful Bots

While the functionalities of bots allow them to contribute to research, information gathering, and the general upkeep of massive repositories and resources like Wikipedia, those same functionalities also allows for a degree of abuse. Even in Wikipedia, where in certain countries the contribution or amount of bot traffic is staggeringly larger than that of its human counterparts, the role and influence of these digital entities is questioned (Niederer and van Dijck, 2010). As has been noted, content aggregator bots on Twitter have been responsible for spreading misinformation and machine learning chatbots have also exhibited a disposition for society’s more hateful nature (Cassa et al., 15

2013; Neff and Nagy, 2013). But these are still examples of benign bots producing unintended and potentially harmful consequences due to environmental and functional design. There are also instances where abuse by automated software agents is far more intentional and deliberate. Examples of malicious bots include distributed denial-of-service (DDoS) attacks, botnets used for identity theft,

“click fraud, cyber-sabotage,” malware, etc (Howard and Kollanyi, 2016).

The line between intentional and unintentional abuse becomes blurred when it comes to classifications and implementation of bots especially considering the ability of bots to mimic human behavior. This is where we enter into a classification of bots by Boshmaf et al. of social bots. Social bot is a general term that connotes a piece of software that serves the function of interacting with humans which occupy the same online spaces as the bot on some socially related way (friends, followers, etc.). An example of these are chatbots, but also political bots (Ferrara et al., Howard and

Woolley, 2017). While chatbots can be used by a variety of different companies to interact with customers, when dealing with socially networked systems like Twitter and Facebook, accounts that post under the guise of human authorship, especially when relating to contentious public discourses like politics or economics, nefarious motivations can begin to be surmised. These bots exploit our own in our desire for sociality and familiarity, working in tandem with with an algorithmic infrastructure which tends to feedback upon itself. Bots that take part in politically motivated activity have been referred to as political bots (Howard and Woolley, 2017).

In a 2000 paper titled “Mindlessness and Machines: Social Responses to Computers”, Nass and Moon performed a study to see what kind of social signifiers are projected onto computers and how unconscious that projection is. Nass and Moon observed social signifiers like gender, race, ingroup/outgroup formation as playing a role in the ways machines were perceived and detected a level of “mindlessness” when interacting with computers (Nass and Moon, 2000). When we take bots into this kind of consideration and ways in which media technologies have changed the landscape of how information is disseminated (Marwich and Lewis, 2017), the role social and political bots play in an increasingly homophilic and information saturated environment becomes a much more tenuous one. 16

2.2.1 The Rise of Socialbots

Prevalence of social media as an arena of discussion in the last decade also coincides with social media become a space where information is shared. Various actors including governmental and non-governmental organizations have made use of social media. In the 2008 US Presidential Election where the Obama Campaign raised 500 million dollars online (Vargas, 2008). This section will cover a particular type of bot that has emerged in the last several decade, coinciding with the prevalence and increased usage of social media networks: the Socialbot (Boshmaf et al, 2011). In 2010, Ratkiewicz et al. published a paper titled “Detecting and Tracking the Spread of Astroturf Memes in Microblog

Streams”. The paper highlights Twitter’s uniqueness and data rich environments and its susceptibility to political astroturfing (astrofuring being the appearance of spontaneous grassroots behavior orchestrated by an individual or organizations) (Ratkiewicz et al., 2010). In this paper, Ratkiewicz et al. outline a method for detecting the “truthiness” of a meme. By meme Ratkiewicz et al. refer to a pieces of content contained in a tweet (e.g. URL, hashtag, body, etc). From there, they created network graphs detecting the diffusion of those memes. This research is largely influenced by Metaxas and

Mustafaraj’s work on Twitter accounts which artificially inflated the occurrences of specific URLs in the

2009 Massachusetts special election. Other work cited includes work detecting spam on Twitter by

Greier et al., Boyd et al. and their work on manually classifying Twitter accounts, and Benvenuto et al.’s work a machine learning systems that, at the time of publishing, had a 87% accuracy rate for detecting good or bad accounts.

This research showed that bots were instrumental in circulating content on Twitter and delineate a uniformed pathology for bot activity (Ratkiewicz et al., 2010). A particular meme which was circulated involved Chris Coons, a Democratic candidate for the U.S. Senate in Delaware, where a network of about 10 bot accounts were found. Mind you that this was in 2010. We shall see that since then both the number of accounts in bot networks has increased rapidly and the topologies of their activities becomes less patterned.

The work done by these studies of looking at astroturfing, spamming, and the ways in which information circulates in various online spaces, provides a ground work for the study of Socialbots.

There are some key features to the socialbot which allow it to be an effective, albeit insidious tool to 17 deploy on whatever online social networking site the perpetrator so chooses. The primarily factor is that it is designed to interact with human users and is meant to conceal its identity as an autonomous being (Boshmaf, et al., 2011). Interactions can include directly posting comments, sharing links, liking content, and any number of other actions on a given social networking site. These sites can be used to influence the human users and mimic activity to an effort to fabricate grassroots digital activism

(Misener, 2011; Boshmaf et al., 2011; Ratkiewicz et al., 2010).

In 2011, Boshmaf et al. at University of British Columbia at Vancouver, articulates the socialbot and socialbot network with a particular consideration to social networking sites or “Online Social

Networks (OSNs) where socialbots are considered most prevalent and dangerous (Boshmaf et al.,

2011). In order to test their hypothesis, a network of socialbots were deployed to gauge their effectiveness. They first identified 4 vulnerabilities in these sites: Ineffective CAPTCHAs, fake user accounts and profiles, crawlable social metrics, and exploitable application programming interfaces

(APIs) (Boshmaf et al., 2011). Socialbots are defined as “computer programs that control OSN accounts and mimic real users.” (Boshmaf,et al., 2011). Ideas that socialbots control the systems that they emerge in and that they do this by pretending to be human users makes for a large portion of current dialectic of how information is presented and shared online.

By revealing the vulnerabilities of OSNs, the Boshmaf et al. sought to see how effective a concentrated, “large-scale” attack by a Socialbot Network (SbN) could be (Boshmaf et al, 2011). At the time of the study, the deployment of bots in order to gauge the effects and depth of penetration, infiltration, or other propagation throughout a social networking site proved to an effective method for exploring the implications and effects of bots activity. Facebook reported to have an 80% infiltration rate. But, there is also the matter of being able to classify, identify, and detect bots. However, as methods for bot detection continue to become more complex, primarily through various data driven methodologies including machine learning and network analysis, the deployment of bots too becomes more complex and multifaceted, leading to sustained efforts to develop new methodologies (Ferrara et al., 2014; Boshmaf et al., 2011).

In 2014, researchers at the University of Indiana developed the BotOrNot? protocol (later named Botometer) in order to gauge the likelihood of a particular account on Twitter, aptly, as a bot or not (Ferrara, et al., 2014). This method was built on some of the methods and discussion put forth by 18

Boshmaf and his group, specifically remarking on the interaction with humans, the emulation and manipulation of behavior, as well as their history (Lee et al., 2011; Boshmaf et al., 2011). Botometer breaks down the socialbot into 6 different attributes to investigate: network, user, friends, timing, content, and sentiments (Ferrara, 2014). In order to test this method, Ferrara’s team build a corpus off of a 2010 study at the University of Texas A&M, where a honeypot method was implemented in order to obtain and attract bot accounts on Twitter (Ferrara, 2011; Caverlee, 2010). The honeypot method employed the use of fake Twitter accounts which post nonsensical tweets that would deter a seemingly normal human from following those accounts (Caverlee et al.). This method also quickly became outdated as socialbots started being able to mimic a greater range of human behavior including circadian rhythms and accounts which were piloted by both automated scripts and human users (Ferrara et al., 2011). These accounts are referred to as cyborgs by Ferrara due to the use of human and automated components. Ferrara’s team collected 200 of the most recent tweets by these previously identified group of bots, and 100 tweets which mentioned them from the Twitter Search API

(Ferrara et al., 2011). This resulted in over 2.9 million tweets with 15 thousand manually verified socialbot accounts, but also produced 10 thousand human accounts (Ferrara et al., 2011).

Some of the more recent efforts at classifying and detecting bots can be seen in different universities, research teams, and other organizations. An example of a current project aimed at bot detection is a small group of collaborators of DiscoverText users who have assembled a Bot Or Not: A

Briefing Book which acts as a repository for people to share and contribute work to a growing repository of bot detection (Shulman et al., 2018). Discovertext allows for a textual analysis of suspicious accounts by human coders, who are briefed with methodologies found within the Bot or

Not: Briefing Book (Shulman et al., 2018). Some of these methodologies are derived from DFRLabs,

MIT Technology Review, How-to Geek , The New York Times, botcheck.me, and Mashable. The majority of the work done by Discovertext relies on manually coding suspicious tweets or accounts with an emphasis on metadata and the use of linguistic or rhetorical abnormalities which is (Shulman et al., 2018). From the data generated by the human coders, various data driven and machine learning methods can be used to create visualizations and perform analysis. Shulman et al. published their findings to datadrivenjournalism.net, remarking on how “bots are not as easy to spot as many may 19 assume” and how continued research is still necessary to reduce the effects of harmful or exploitative algorithms in digitally mediated society (www.datadrivenjournalism.net).

2.2.2 The Rise of Political Bots and Computational Propaganda

The use of socialbots to exploit or abuse vulnerable social media ecosystems and social media users is nowhere more widely discussed and deliberated than in the political realm (Woolley et al.,

2016). The implementation of this type of bot with this particular intention is more commonly captured in the term ut forth by Samuel Woolley: the political bot. Both Woolley and Howard classify the use of political bots as computational propaganda in order to emphasis the production and circulation of content through digital communicative technologies. Political bots are considered to be “algorithms that operate over social media, written to learn from and mimic real people so as to manipulate public opinion across a diverse range of social media” (Howard and Woolley, 2016). Bots can also hinder the advancement of public policy by creating what seems to be grassroot movements, deploy various

‘bombs’ to occupy various search engine spaces or control a piece of contents visibility (Ratkewicz et al., 2011b), and polarize the political discussion (Conover et at., 2011).While the use and deployment of automated agents to influence the outcomes of political elections is not entirely a new phenomenon

(e.g. the 2010 US Primaries and 2008 US Presidential election mentioned above), the 2016 US

Presidential election and elsewhere around the world has caused a shift in the research of automated software agents in political sphere (Shulman et al., 2018).

One example of the use of political bots comes from the first 2016 U.S. Presidential Debate, where Kollanyi et al. collected pro-Trump hashtags and pro-Clinton hashtags and analyzed the amount of traffic generated by each of those hashtags on Twitter (Kollanyi et al., 2016). This method generated over 9 million tweets with the purpose of “discerning how bots are being used to amplify political communications” (Kollanyi et al., 2016). Some of their findings showed pro-Trump hashtags nearly doubled that of pro-Clinton hashtags, roughly one third of the traffic related to pro-Trump hashtags were from automated accounts (i.e. bots) compared to about one-fifth of the pro-Clinton hashtags (Kollanyi et al., 2016). Their findings also remark on how Twitter appears to be an overall 20 pro-Trump space, rather than pro-Clinton where many human users engaged in the political discussion (Kollanyi et al., 2016).

Another research study conducted in 2017 over the tweeting habits of voters in the state of

Michigan considers the same political and technological implications (Howard et al., 2017). In this study, Howard et al., were looking for how misinformation and more extremist, sensationalist, conspiratorial or legitimate misinformation spread and compared to that of professionally researched and legitimate political news (Howard et al., 2017). Their findings showed that fake news, or “junk news”, outperformed that of other more credible news sources (Howard et al., 2017). Interestingly enough, the occurrences of content being shared from credible news sources reached “its lowest point the day before the election: (Howard et al., 2017). In the UK, the referendum on EU membership using the hastags #Strongerin and #Brexit (Howard and Kollanyi, 2016). The results of this study showed that while there was not a distinctly or discernibly high amount of automated accounts, those accounts were still strategically used (Howard and Kollanyi, 2016) Specifically, they showed that less than 1% of the accounts that were tweeting with the particular hashtags associated with the UK referendum generated roughly “32% of all Twitter traffic about Brexit” (Howard and Kollanyi 2016). The implementation of political bots is not only a European or American phenomena, but has been researched in other countries as well. Political bots have been observed as influencing the Japanese

2014 general election (Schafer et al., 2017). The methodology used in this research took the posting patterns as the primary attribute to focus on in their sampling and analysis (Schafer et al., 2017).

Some of the findings they point to involve the use of cheap technologies to influence social media and a difficulty in identifying bot accounts using purely statistical analysis (Schafer et al,. 2017).One more example of research done on political bots involves the tweeting habits of Venezuela (Forelle et al.,

2015). Much like the UK example earlier, bots tweeting about Venezuelan politics only account for a small percentage (10%) of overall political communication on Twitter but are used strategically and by the radical opposition (Forelle et al., 2017).

Howard and Woolley explore some of the contemporary research issues surrounding political bots in their paper “Political Communication, Computational Propaganda, and Autonomous Agents”.

This article is a derived from papers submitted to the 2016 International Communication Association meetings in Fukuoka, Japan. The burgeoning networked apparatus, rise of big-data infrastructure, the 21

Internet of Things (IoT) and growing ubiquity and use of digital media for news and information all contribute to the spread of damaging political communication online (Woolley and Howard, 2016;

Cisco, 2014). Some of the ethical deliberations regarding how to combat the use of political bots and lessen their impact are included in this article as well. Marechal proposes the normalization of state- sponsored auditing at the algorithmic level while Mittelstadt considers that the “burden of auditing these systems for political bias lies on the shoulders of the platforms themselves” (Marechal, 2016;

Mittelstadt, 2016; Woolley & Howard, 2016). Guilbeault sees the emergence of political bots as an opportunity to open discussions about policy and the theoretical implications of both the “innovation and intervention” of digital communicative technologies (Guilbeault, 2016; Woolley & Howard, 2016).

Sandvig argues that algorithmic auditing will become an increasingly important area of research in both social and scientific sciences (Sandvig et al., 2016).

2.3 Sockpuppets and Cyborgs

A common thread through all of the studies presented in this chapter thus far delineate an analytical difficulty in the ways bots are researched, namely that of the human element and topography of behavior. This last section will present some of the research done regarding “black markets” that are built off of marketing techniques to boost engagement metrics, and thus visibility, of various pieces of content on different online platforms (Heder, 2018).

In paper titled “A Black Market for Upvotes and Likes”, Heder explores the microtask/freelancer website microworkers.com (Heder, 2018). In that paper, Heder looks at listings posted to the site, where payments are made in exchange for likes, comments, upvotes, watching videos, and other microtasks (Heder,2018). Microtasks are menial tasks performed online where microworkers are compensated from anywhere between 0.15 USD to 1 USD (Heder, 2018). It should be noted that microworkers.com is not exclusively used for the buying and selling of likes and upvotes (some of the tasks conducted include research surveys, software testing, and data processing). However, Heder shows that 89.7 % of microtasks on the site were related to online promotion (Heder, 2018). Heder collected 1,856,316 microtasks and 7,426 campaigns between February 22, 2016-17, categorized by 22 platform and specific activity. Some of the platform categories include Amazon, Instagram, Alibaba,

Reddit, Facebook, Smartphone (iOS and Android), Twitter, and Google among others. There are also

23 categories of activity including: Following, Like/Upvote, Sharing, Tweet/Retweet, Comment, Signup,

Vote, Solving CAPTCHAs and other actions that are associated with online activity and social media.

An example of an actual job posting, where the client requests the creation of 230 Gmail accounts via www.fakenamegenerator.com can be seen in Image 1.

Image 1: Posting from microworkers.com for generating Gmail Accounts

The total amount the client of this job posting pays in order to generate these accounts is $40.48

(Heder, 20118).

Heder then looked at some of the totals of these campaigns including how much money was spent on specific platforms. The Smartphone platform had the most campaigns at 1102, 231,892 tasks, and a total budget of $27,702.07 (Heder, 2017). Reddit came in at number 7 behind Google+,

Twitter, Facebook, Youtube, and others with 539 campaigns, 77,104 tasks, and a budget of $5,434.15

(Heder, 2017).

Heder uses the term sock puppet in order to describe the phenomenon of when many accounts are owned and operated by one or several human users. One of the ways in which Heder 23 describes the hypothetical detection of these methods is by looking how many accounts are logged in order being used by a single serve, IP address, or other location based data.

3 Reddit

This section of the thesis will cover some of the academic contributions as well as some independent research done on Reddit.com. One of the reasons for choosing Reddit has to do with what seems to be a lack of research done (or at least attention given) regarding bots on Reddit as well as Reddit’s culture, politics, or technicity (Massanari, 2015). It should be noted that while the literature regarding bots on Reddit, is smaller in comparison to Twitter and Facebook, there have been some papers dedicated to Reddit as a primary site of observation involving Gamergate, The Fappening, the

2016 U.S. presidential election, and the requesting and creation of bots on Reddit. It is unclear why the subject of bots on Reddit is not a more prevalent topic in the literature, and perhaps some of it has to do with the technical difficulties in studying bots, generally speaking, and particularly aspects or

Reddit which makes it a difficult platform to study. One such aspect is referred to as ‘vote fuzzing’ which obfuscates the actual number of votes from the ones depicted (Reddit.com). This is done to prevent spam bots, but also prevents the observation of voting habits. Despite these reflections, there is indeed research done on Reddit, specifically regarding to the rise of the alt-right, pizzagate, gamergate, toxic technocultures, the circulation and spread of memes, images, and sentiments delineating an anti-feminist, anti-globalist, and technocratic worldview (Massanari, 2015). As we shall see later, Reddit has a system of self governance which can result in slow or nonexistent administrative action, but also allows for Reddit users (redditors) to act on their own.

3. 1 Background on Reddit Structure, History, and Culture

Reddit was founded in 2005 by Steve Hoffman and Alexis Ohanian. In 2006, Reddit was sold to to Conde Nast Publications and in 2011 became a subsidiary of Conde Nast’s parent company 24

Advance Publications. Reddit later became independently operated although Advance Publications was still the primary shareholder.

Reddit is a message board styled content aggregator where users are able to post links in the form of photos, videos, web pages, etc from other content hosting sites on the Internet. Users are also able to post plain text posts as well, which is considered to be the only content that is originally hosted or produced by Reddit. Users are able to vote up or down on a particular piece of content in order to show favor, support, agreement etc. Users then can comment on the original post or reply to other users’ comments where the parent comment can branch into different threads of replies. Users are able to form communities around particular areas of interests in the form of a subreddit. As of

November 17, 2017, Reddit has 1,179,342 subreddits (statista.com) and as of February 2018, 542 million monthly visitors and 234 million unique users (alexa.com).

In the introduction to her 2015 book Participatory Culture, Community, and Play: Learning from

Reddit, Adrienne Massanari examines how Reddit functions as a participatory space with particular affordances that make it unique from other social networking sites like Facebook and Twitter. One of the primary aspects of Reddit (and in general, research around participatory cultures) which

Massanari elaborates on, involves an inaccurate representation for the democratic potential in online participatory cultures that researchers in the past have seemed to think too highly of (Massanari,

2015). That research does not properly describe the ways in which space and attention is negotiated in complex and nuanced ways and is simultaneously co-created by a combination of users, designers, administrators, and moderators (Massanari, 2015).

Another aspect which sets Reddit apart from other social networking sites is the account creation and verification. Both Facebook and Twitter aim for their users’ accounts to have some semblance to a real person in real life where their actions, check-ins, and other methods of engaging their users and the real world are recorded. While Reddit is not completely anonymous (as opposed to

4Chan and 8Chan) it is pseudonymous, where the account and user do not share the same the level of verisimilitude. Furthermore, where Facebook and Twitter operates through the network created through socializing (e.i. followers and the friends), Reddit does not facilitate the same network of users and accounts, although some form of following is available, it is not the primary metric from which a (to borrow a term from Bruno Latour) network of associations forms. The way information or content flows 25 through the site is also different. Reddit was originally designed as a content aggregator as opposed to sites which host original content (e.g. video or images). It also allows for text submissions alongside the aforementioned submission types. From those submissions, then that content is voted upon in order to determine what content is considered noteworthy or what should be seen by the larger audience.

Reddit imagines itself as a democratic space, where speech is not suppressed, users have the ability to decide what content rises to the top of an otherwise ubiquitous banality, everyone and anyone can find or build their own community, and everyone has an equal voice. After a piece of content is submitted to Reddit, users are then able to upvote or downvote that submission as they see fit. The post then is given a ‘karma’ score which is an approximation of upvotes minus downvotes.

Posts that are voted on approvingly remain on the front page of the subreddit which makes their way onto Reddit’s frontpage, which is a collection of the top posts from various subreddits. Karma is also perceived to be a form of validation, or that if enough people voted positively for whatever that post is, it must be true, valid, or otherwise good. This ideology is supposed to allow for remarkable, unique, or newsworthy content to get pushed to the top of a subreddit’s page, where it will be seen by a greater number of people, and less worthy content gets relegated to the unseen. The same is true of comments although there are filters which allow the user to select ‘controversial’, ‘hot’, ‘new’, or ‘best’ comments. Despite this seemingly democratic meritocracy, where only worthy content and commentary can survive, there have been examples of how the technical affordances of Reddit, alongside its user base and culture, has propagated problematic information, materials, and images.

In Reddit’s content policy, there are several key aspects which describes allowable content in legal albeit laissez faire system of governance. It prohibits content that is illegal, contains involuntary pornography, is “sexual or suggestive” of minors, “incites violence”, “threatens, harasses, bullies or encourages others to do so”, share personal information, “impersonates someone”, solicits goods and services, or is spam (reddit.com). The prohibited behavior section only cites “asking for votes or engaging in vote manipulation”, “breaking Reddit or doing anything that interferes with normal use of

Reddit”, and “creating multiple accounts to evade punishment or avoid restrictions” (reddit.com).

Reddit’s enforcement policy includes warnings, “temporary or permanent suspension of accounts”,

“restrictions of accounts and communities, “removal of content”, and the “banning of Reddit 26 communities”. Reddit’s sentiment of self-governance can be seen expressly in the section on moderation within communities where both rules and enforcement are decided upon by a subreddit’s moderators.

In the following sections, several case studies will be presented which represent controversial events in Reddit’s history. These events also represent instances where Reddit’s policies and culture allowed for the circulation of sensitive information and emerging toxic ideologies.

3.2 Reddit’s Toxic Technocultures, Gamergate, The Fappening

In a 2015 paper by Massanari titled “#Gamergate and The Fappening: How Reddit’s Algorithm,

Governance, and Culture Support Toxic Technocultures”, Massanari explores how the Reddit ranking algorithm, Reddit’s relatively low barrier to entry in terms of account creation, a history of message boards sites, geek masculinity and culture, and the way in which Reddit governs its own internal politics contributed to a massively anti-feminist movement (Massanari, 2015). It is also arguable that these methods of harassment became appropriated by online cultures to assist in the harassment of the other individuals, primarily that of doxxing (the publishing of personal information, which is expressly against Reddit’s content policy). Massanari is also interested in how non-human agents act in these spaces.

There are several components to the Reddit platform which have contributed to the type of cultures emergent on the site. Firstly, the ability to create pseudonymous accounts with little verification encourages a playful demeanor in how users engage with each other. Second, the ability to create subreddits, allowing for niche communities to form. It is from the last point which Massanari considers nerd or geek culture to come into the foreground as well as Reddit’s message board aesthetics like The WELL (Whole Earth ‘lectronic Link) (Massanari, 2015). Aspects of nerd and geek culture along with ideas of geek masculinity, which are embedded in technological expertise, the valorization of knowledge, and the sharing of information play a large role in the circulation of information regarding both Gamergate and the Fappening (Massanari, 2015; Bourdieu, 1977;

Coleman, 2013). Massanari also cites the work of Lori Kendall (2011) in terms of how nerds view themselves as adept and joyful in their mastery of computer related activities (Kendall, 2011). This 27 sentiment is then embodied in revenge fantasies as seen in the 1984 film Revenge of the Nerds, and, as Ryan Milner observed, spaces like Reddit are radicalized and gendered, assuming a white male centrality (Milner, 2013).

Gamergate began when a jilted ex-lover, Eron Gjoni, posted on the SomethingAwful forums accounting the break up between he and game developer Zoe Quinn. The post included the defamation of Quinn, who had recently published a game titled Depression Quest to the Steam

Greenlight service as well as the accusation that Quinn’s game only received attention and positive reviews due to an intimate relationship with other games journalist (Massanari, 2015). The post also cited alleged Facebook messages exchanged between Quinn and Gjoni. The original post was removed, but was revived on 4Chan where it took on a new life as an argument which called into questioned the ethics of games journalism, but soon thereafter became associated with the harassment of women in different aspects of the video game industry (Stuart, 2014). Much like

Gamergate, the Fappening helped merge gaming, geek, and technocentric cultures with anti-feminist, misogynist and men’s rights activist (Massanari, 2015).

The Fappening was the result of a huge data breach where many celebrities had their personal photographs hacked from Apple’s iCloud service and posted to 4Chan (Massanari, 2015). Within the first 24 hours, /r/thefappening had received 100,000 new subscribers until the subreddit was redacted.

The images circulated through different subreddits even going so far as Reddit administrators commenting on the substantial increase in Reddit’s traffic due to the propagation of intimate celebrity photographs.

Massanari uses both Gamergate and The Fappening as an entry point into a toxic technoculture that has fomented on Reddit as well as a technical structure which is susceptible to abuse. Because Reddit is a pseudonymous space, where groups can create their own communities, content is voted upon and in that process visibility and attention are prized, and lastly, moderation is kept to a minimum in order to preserve an idyllic sense of authenticity, groups engage in homophilic activities which tend to support a white male point of view (Massanari, 2015; Auerbach, 2014). There is little support from the moderators or administrators to levy punishments on anybody who violates

Reddit’s policies, and often, if a complaint is received, it is met with the counsel of ‘if you don’t like the way things are on this subreddit, make your own and enforce your rules there.’ 28

2.2.2 /r/The_Donald

This coalescing of anti-feminist, misogynists, and men’s rights activists, coupled with a techno- libertarian sentiment that apotheosizes networked technologies, added to an emerging alt-right movement on Reddit which converged on the controversial subreddit /r/The_Donald (/r/The_Donald).

Tim Squirrell, in an article for Quartz.co, uses a massive dataset of Reddit comments and posts from the Google BigQuery service to create, what he calls a “Taxonomy of Trolls” on /r/The_Donald

(www.qz.com; Google). Squirrell classifies the following smaller groups of communities as composing the alt-right and gathering on /r/The_Donald: 4Chan shitposters, anti-progressive gamers, men’s rights activists, anti-globalists, and white supremacists. Squirrell then conducts a study similar in scope to one which took place, and larger influenced Squirrell’s study, by the Alt-RIght Open Intelligence

Initiative at the University of Amsterdam, where they tracked the usage of the word ‘cuck’ on

/r/The_Donald (wiki.digitalmethods.net). In the study by UvA, the implementation of the Google

BigQuery service along with the observation of Reddit as an open-source, data-rich resource, growing in popularity, and which seems to be a gathering place of the alt-right, has allowed for a particularly interesting site of observation.

Other deliberations for Reddit’s consideration, in particular /r/The_Donald, has to do with the positioning of /r/The_Donald within Reddit as an aggregator, incubator, and disseminator or misinformation, propaganda, or fake news (Zannettou et al., 2017). In an article titled “The Web

Centipede: Understanding How Web Communities Influence Each Other Through the Lens of

Mainstream and Alternative News Sources”, published in 2017, a study which tracked URLs from 99 different news sites on Reddit, 4Chan, and Twitter, showed the tendencies and capabilities of these sites to circulate information and to what extent their influence is. /r/The_Donald was chosen by

Zannettou et al. due to its positioning and history as a hub of alt-right activity. The research reported that 99% of URLs shared on /r/The_Donald were from alternative news sources (Zannettou et al.,

2017). Using the Hawkes Process (a point process method which allows for temporal consideration), the report shows that /r/The_Donald “is the only platform that has the greatest alternative URL weights for all of its inputs” (Zannettou et al., 2017). 29

Reddit is considered to act as an intermediary space between the deeper and darker corners of the internet (e.g. 4Chan) and mainstream social networking sites (e.g. Facebook and Twitter)

(Bradshaw, 2017). In a 2017 article for theHill.com, Samantha Bradshaw, a member of the

Computational Propaganda Project at the University of Oxford, considers Reddit to be a fertile space in which to test the virality of content and a space where “coordinated information campaigns” occur

(www.thehill.com). The article goes on to explain that most of the US investigation of Russian manipulation on social media is focused primarily on ad space, which Reddit manually verifies so as to not have prohibited content including contentious political views, making it less likely for automated agents or manipulators to buy ad space (reddit.com). In a post in /r/announcements, /u/spez shares thoughts about Russian influence on Reddit and Reddit’s internal investigation on these matters

(reddit.com). In the post /u/sez remarks on how “ads from Russia are blocked entirely” (reddit.com).

He then mentions that indirect propaganda is a more contentious issue particularly because of

Reddit’s stance on authentic content.

However and despite this, automated software agents populate a large amount of the Reddit space. And because Reddit seems to have a strong white male geek culture, along with subreddits like /r/The_Donald which posits an extremely partizan political ideology, the experimentation and implementation of automated software agents can still allow for abuse. When this is taken into consideration

2.2.3 Reddit Bots

This last section on Reddit will cover an ethnographic study in which the sentiments surrounding the requesting and creation of bots on Reddit is discussed. While there does seems to be a well-deserved alarmist response to the use of bots on Reddits, the ways in which Reddit both mediates the request/creation of bots and interaction with its API, allows for an encouraging space for programmers and amateurs to exchange information and share ideas (Long et al., 2017). In a paper titled “‘Could You Define That in Bot Terms?’: Requesting, Creating and Using Bots on Reddit”, a 30 researcher study was conducted analyzing the sentiments of “2,284 submissions [and 14,822 comments] across three [subreddits] dedicated to the request, creation and discussion of bots on

Reddit (Long et al., 2017). In this paper, some of the key finding for how bots are requested and created mostly delineate a misunderstand of the capabilities of bots between developers and users on

Reddit (Long et al., 2017). One of the main purposes of this study is, while there is much research done investigating the effects of bots in different online spaces and the public discourse, little is done which investigates the why and how bots are created (Long et al., 2017). Reddit provides an interesting opportunity to research these questions because there is an activity community which openly discusses the creations of bots (Long et al., 2017).

In this research /r/requestabot, /r/botrequests, and /r/botwatch were selected for observation due to their subject matter (Long et al., 2017). The data collected ranged from 2012 to 2016 and made up the entire dataset. Two stages of qualitative analysis was then undertaken in order to discover the types of topics and functionalities involved in the request of bots and then a thematic analysis to see what predicates the larger discussion of bots on Reddit.

One of the key findings that the majority of bots requested (705 out of the 2,284 submissions) were administrative with the second most request type of bot request was Play/Humour (278) and the third being functional/quality (206) (Long et al., 2017). The most common functionality of requested bots involve querying and responding to particular keywords (686) followed by automatically posting content from other sources (220), and “querying user accounts” (105) (Long et al., 2017).

The thematic elements that come out of the second stage of this research largely has to do with knowledge and is broken into five themes: “knowledge and kills, technical infeasibility, legitimate and valuable bots, inappropriate and annoying bots, and the value of building a bot” (Long et al.,

2017). Out of these categories, the first two primarily deal with technical aspects of bot creation and implementation (i.e. how does one go about doing this or that, and is ‘x’ possible). Overall, the findings show that there is a disconnect between requests and developers, where requesters do not possess the ability or knowledge to manifest their bot idea, and in turn over estimate the resources and feasibility of their ideas. A particular example are bots that are able to respond to comments in natural ways. This requires a more sophisticated bot that, rather than being triggered by a keyword and posting an automated response, would be able to understand the contextual elements in a post, and 31 respond appropriately. An example cited in this study is the request for a bot that is able to respond to fallacies (Long et al., 2017). What is refreshing, though, is that there is an awareness of exploitative or malicious bots (Lang et al., 2017). Requests that have ambiguous motivations or do not fully describe the intent of a bot are called into question. In one instance, a redditor asks another “Please explain why you want this? It can be easily abused” (Lang et al., 2017). As far as malicious bots go, anything that seemed to violate Reddit’s terms, policies, or norms would be identified as such with warnings of bans and other punishments by the more seasoned bot creators.

Lang concludes by stating that the study of bots, and especially a space where bots are requested, created, and openly discussed is a valuable resource for understanding an online community (Lang et al., 2017). Building bots require a working knowledge of that a platform is capable or doing and be able to observe the requests of users allows us to see what a community desires in about.

This last section will cover independent research done by members of the Reddit community.

As observed above, there is a degree of technical expertise regarding the request and creation of bots on Reddit. However, in that study the sentiment toward malicious bots is somewhat overlooked as well as the fact that, for the most part, they are only cover bots which are discussed openly on the platform, rather than ones that could possibly already be running. The articles covered in this section while be mostly focused on /r/The_Donald and /r/bitcoin.

3.2 Independent Research by Redditors

The use of bots or sockpuppets for nefarious activity on Reddit has not gone unnoticed by academics, but it is also a prevalent topic and point of discussion for many Redditors. We have seen that there is an active community centered around the requesting, creation, and discussion of bots, but there are also independent investigations into how Reddit’s ranking algorithm is manipulated by malicious software agents and those who deploy them.

In an archived post from 2016 on the subreddit /r/TheoryOfReddit, an account named

MittRomneysCampaign (name is left because it seems to not be a primary account), discusses some 32 findings and observations from two experiments conducted in 2012 and 2008 (www.reddit.com). Both experiments were to see if a particular type of post with a particular type of sentiment could make it to the top of certain subreddits and ideally to the frontage. The 2012 experiment was organic, using the

MittRomneysCampaign handle to lament Barack Obama’s re-election.

Image 2: MittRomneysCampaign top comment

The poster extrapolates that the account name, along with the content of the comment, allowed the comment to reach the top of the thread. In the 2008 experiment, the content of the comment dealt with a scientific observation that appears adversarial to the general culture (as the poster sees it) of Reddit as whole, and even questions the collective intelligence of the site as a whole. In this situation, the poster boosted the post and used proxies to hide the manipulation. From the standpoint of the original poster, upvotes represent a truth or approval gauge rather than the support of quality content. The poster goes on to claim that most redditors vote with this in mind, where, in simple terms, highly upvoted content is good or true, downvoted content is bad or false. But the truth of the matter is, according to the original poster, quality content can get pushed to the bottom and never see a true day in the light. And it does not take much to determine the fate of a post in its nascent stages.

/u/MittRomneysCampagin uses these examples to illustrate the types of SEO tactics involved in gaming Reddit’s ranking algorithm. The user also identifies several different ways the voting system can be gamed: forced front paging, vote nudging reverse nudging, and vote brigading/puppeteering.

Forced front paging seems like an obvious activity to detect if it is particularly suspicious to the poster, but vote nudging, reverse nudging, and brigading/puppeteering seem more common and less easy to identity.

The posters posits a corollary relationship to upvotes, views, and subsequently economic incentive, and points to how Reddit weighs recent submissions and upvotes higher when compared 33 with later votes, voting tendencies of redditors, and Reddit’s enforcement of its policies allow for an easily manipulable system.

In another post, this time on /r/WikiLeaks, the redditor provides examples of websites such as redditbot.com and upvotes.club as services readily available for the buying of tools, accounts and services related to votes and cites several more in the post (reddit.com; redditbot.com; upvotes.com).

The user goes on to describe several different known bots that they were able to create a bot, referred to as a Markov Chain Bot which takes strings from other comments, splices them together, and posts a new comment.

Image 3: Diagram of Markov Chain Bot

This post concludes by not having substantial evidence of malicious bot activity or vote manipulation, but rather serves to show the ease and accessibility of bot creation and the buying of votes.

The next submission, posted to /r/subredditcancer, contains several different links to different subreddits and websites (reddit.com). There are several different aspects of the post which lend it toward a methodology for the detection of bots on Reddit: lists of banned/shadowbanned accounts associated with /r/The_Donald (reddit.com), links to an image by Ryan Compton which shows upvotes over time on different subreddits which can be seen in Image 3, presents a script supposedly circulated on 4Chan which allows the user the ability to massively upvote or downvote content on a subreddit, an example of ‘Markov Chain Bots’ as described previously, upvote to comment ratios, calculations for the amount of upvotes on /r/The_Donald subreddit compared to the frontpage, and comparisons to other subreddits (reddit.com). 34

Image 4: Upvotes overtime per Subreddit. Provided by Ryan Compton. http://ryancompton.net/2016/08/07/upvotes-over-

time-by-subreddit-or-why-the_donald-is-always-on-the-front-page-of-reddit/ 35

Image 5: Script For The Massive Upvoting or Downvoting of Content on Reddit 36

Image 6: Screenshot of Markhov Chain Bot 37

Image 7: A Screenshot to Highlight Comment to Upvote Ratio

Most of the subreddits linked to this submission came from anti-Trump subreddits such as

/r/EnoughTrumpSpam and /r/MarchAgainstTrump. While these findings are enticing, they seem more speculative in nature. For example, the ratio of upvotes to comment is an interesting metric, but can easily prove false positives by returning posts with a few upvotes but no comments or low ratio.

Perhaps putting a high a limit on the vote score could return interesting results, but it can only be so accurate. Vote nudging and brigading are against Reddit’s policies, but this simple prevents the public requesting of votes and does little for coordination elsewhere. Reddit is also not without its own awareness and efforts against this type of behavior and we can see some of that in their yearly

Transparency Reports.

3.2.1 Reddit Transparency

On April 18, 2018 Reddit released its 2017 Transparency Report (redditinc.com). This report catalogs details of government and law-enforcement requests for information regarding Russian influence on Reddit (reddit.com). He elaborates on the removing of “a few hundred accounts that were of suspected Russian Internet Research Agency origin” and reports to having found 944 other suspicious accounts at the time of the post (reddit.com). The report shows that 70% of the suspected 38 accounts had zero karma, 1% had negative karma, 22% had 1-999 karma, 6% had 1,000-9,999 karma, and 1% had over 10,000 karma (reddit.com). A link to those 944 accounts can be found here.

In the comment thread that follows the post /u/spez makes note of the top ten communities where the suspicious accounts posted the most numerously in: /r/funny (1,433), r/runcen (1443),

/r/Bad_Cop_No_Donut (800), /r/gis (553), /r/PoliticalHumor (545), /r/The_Donald (316), /r/news (306), / r/aww (290), /r/POLITIC (232), /r/racism (214). /u/spez also notes that the subreddits have been left available so that so that they can be investigated independently.

Suspended or banned accounts have been observed through the methodology of the original research presented in this thesis which will be elaborated upon in the next chapter. These accounts, when viewed, no longer contain their prior activity.

4 Methodology and Findings

In order to better understand the ways in which bots are used and discussed on Reddit, qualitative as well as some quantitative research was taken to observe the sentiments in different subreddits, along with other online communities, where the discussion of bots or vote manipulation on

Reddit occurs as well as what kind of bots populate these spaces. The qualitative research centers around a snowballing method of research where Google is queried in order to see how easily and how discussed Reddit’s content can be gamed. The Query Design for the Google search engine uses

[“buying Reddit upvotes”], and [“how to build a Reddit bot”] followed by investigations into the top ten sites returned from those queries. In order to remove digital artifacts which may sway the search results, a research browser was implemented as well. These queries were chosen in order to see what kind of tools or services exist that are actively trying to manipulate the visibilty of content and submissions on Reddit.

Observations were also made on the subreddit /r/The_Donald, /r/bitcoin, /r/botwatch,

/r/EnoughTrumpSpam, and /r/politics. /r/The_Donald was chosen because of its partizan ideology, its position on the Reddit 2017 Transparency report has having a high amount of banned accounts, and for its uniqueness of community (Reddit.com). /r/bitcoin was chosen because it has also been 39 observed to have a more recent occurrence of vote manipulation (medium.com; bitcoinmagazine.com) especially considering the prevalence of cryptocurrencies and fintech this last year (forbes.com).

/r/botwatch houses a discussion board on different types of bots and typically seems to have an amiable sentiment toward then, but do recognize the damage their abuse may cause. Requests for information surrounding known bots and bot detection protocols were also submitted to /r/botwatch in order to further investigate how Reddit as a community views bots and what measures are in place to prevent harmful activity. /r/EnoughTrumpSpam was chosen because of its relationship to

/r/The_Donald and the findings provided by independent researchers. Lastly, /r/politics was chosen as well because it us considered one of the larger and more general subreddits as well as being related to politics.

The quantitative analysis comes from using the database provided by Felipe Hoffa on Google’s

BigQuery. Using this resource, I queried for users that have ‘bot’ in the name as it seems to be a standard practice on different sites to label a bot with ‘bot’ in the account name.

Image 8: Google BigQuery Script for Accounts with the String ‘bot’ in it. Provided by Sal Hagen

This does not detect bots where the user does not specify it as a bot, and would require a detection protocol or tool similar to BotOrNot? where the content of the post and posting habits are analyzed in order to gauge its authenticity. What is possible with this method, is a list on known bots that are particular to the subreddit. From there, the top accounts are looked at in order to determine what kind of socially normative or governing rules that particular account enforces (Lessig, 1999).

These methods are experimental do not necessarily represent a way with which to detect malicious bot activity. Rather, they act as an entry point into observing how bots operate on Reddit with specific examples to draw off of. 40

4..1 Google Search Findings

The most significant findings from this experiment shows the ease with which to find resources to generate engagement for a particular piece of content. The first query [“buying Reddit upvotes”] can be seen below.

Image 9: Buying Reddit Upvotes Google Search

The first result, “Upvotes.club”, we have seen before in section 2 of chapter 3. When we go to the homepage of the site, we immediately see the options for selecting how many upvotes or downvotes we would like to add to posts or comments on Reddit. 41

Image 10: Homage page of Upvotes.club

The site specifies that all votes are performed by accounts which the site owns. Under the “Why work with us” tab to the left, the site supposes discretion between itself and the customer, ensuring that transaction records will be regularly wiped (upvotes.club). The site then states that all tasks are completed manually without the use of bots. Despite this, the use of aged accounts, operated by a human with an active history, and specific location makes the activity harder to detect. The site also employs the use of VPNs (Virtual Private Networks) in order to further obfuscate the activity. The buying of upvotes violates Reddit’s user policy, but due to the human component and lack of transaction history, detection and evidence of such an offense becomes incredibly difficult. What is more unsettling is the option for downvoting as well. Rather than only supposedly providing growth (as most of these services claim to be doing) they also provide the ability to prevent growth.

The next result from the Google query links to a post on Quora.com asking for the “best place to buy

Reddit upvotes.” While the comments mostly have suggestions for what to look for in choosing where to buy upvotes, it does recommend Appsally, a growth hacking service. Some of the suggestion are related to the guarantees of upvotes.club (e.g. the use of humans and not bots). A disturbing sentiment that these comments have are in that they describe the use of “quality” accounts

(quora.com). This is mostly likely related to the use of aged, active, and geographically diverse accounts operated by humans, as Heder observed in their research, most of these tasks get delegated to poorer countries (Heder, 2018). 42

The next result redirects to a page called boostupvotes.com. Here we can see the acquisition of between 10 and 200 upvotes; the cost of 200 upvotes being costs $39.99. The interface is simple, presenting a 3 step process where one selects the quantity, inputs the like, and pays via paypal. The site also mentions the importance of early voting on Reddit, and how the first few views can determine how well the post can stay visible.

Only two of the top ten results either do not present a service or support the buying and selling of upvotes, and in fact admonish it. Out of the 8 that are actively exchange various forms of currency for upvotes or downvotes, one that stands out is blackhatworld.com . In the image below we can see a submission from Jun 6, 2016 containing instructions for Reddit marketing strategies without the use of the of the service as seen earlier and suggesting that to vote nudging is simple and does not the require more than 3-5 votes.

Image 11: Blackhatworld.com

What stands out from this post is when the poster says “I am thinking about automating this stuff via a Multi Account Tool* automation API” (blackhatworld.com). This stands out because of the implementation of technology to aid in and further prevent the detection of this activity. For this reason blackhatworld.com stands out.

According to webopedia.com “Black Hat SEO refers to the use of aggressive SEO strategies, techniques and tactics that focus only on search engines and not a human audience, and usually does not obey search engines guidelines” (webopedia.com). Blackhatworld.com is forum (much in the same 43 message board style as Reddit) which is centered around online marketing strategies, technical strategies and tools for concealing one’s identity and ones marketing activity.

On the homage, to the left of the most recent discussion threads, show a sidebar with some of the main topics and forums which this site has to offer.

Image 12: Blackhatworld.com Sidebar 44

Image 13: Blackhatworld.com Sidebar

It should be notes that blackhatworld.com also provides services related to web design, web hosting, graphic design, and other legitimate endeavours. By looking at and interacting with the page, certain vernaculars like “Shit list”, recent members labeled as “newbies”, the amount of technical information that is passed through, suggest a very techno-libertarian sentiment not unlike the one found on Reddit. My interpretation is derived from limited experience on the site rather than in-depth studies and given some of tools and services being shared on this site, further research is encouraged. When we visit the ‘Cloaking’ forums, the description at the top of the page shows that the topics of discussion revolve around “Cloaking, Doorway pages, and Automatic content generators”

(blackhatworld.com). For those who are not as well versed in SEO nomenclature (myself included), cloaking refers to a technique in which the content presented to the search engine spider is different 45 from that presented to the user’s browser” (Wikipedia.org). What is went by this is essentially, cloaking allows a user to bypass or deceive search engines in order to display pages that would otherwise not be see. This is related to “Doorway Pages” which are “low-quality pages (or groups of pages, or entire websites) optimized to rank well for specific keywords that act as a door between users and content... they offer little value to visitors and serve the sole purpose of boosting SEO value” (brafton.com).

Under the “Black Hat Tools” section, the description at the top of the page is as follows, “You can’t GO it alone...have an arsenal of programs to help you out!” (blackhatworld.com). While this sounds helpful for legitimate businesses to grow online, the moniker of black hat still presents this information as secretive and inherently deceptive. Some of the thread topics include ‘CAPTCHA solving software’ (a method used in bot deployment (citation)), VPNs, various scrapers related to

Instagram, YouTube, Twitter, Twitch, among others. Below is an image of a reply to a request for software that can automatically detect CAPTCHAs and solve them.

Image 14: Reply to Request for CAPTCHA Solving Software

The first of tools provided by the user is an application called CapMonster. 46

Image 15: Posting of CapMonster

Here we can see the details, benefits, and limitations of the software along with the chance to demo or buy it.

In order to return to our object of focus, the term ‘reddit’ is queried in the keyword search bar of the blackhat tools forum.

Image 16: Threads with the keyword ‘reddit’ in the BlackHat Tools forums 47

Immediately we can see that Reddit bots are the most prevalent topics in the discussion threads. The first post is from Aug 21, 2017 and is related bots that can upvote on reddit, and requests verification of whether or not a particular bot is “legit” (blackhatworld.com). A reply suggests redditdominator.com which is an “all in one Reddit marketing tool for Reddit marketers” (redditdominator.com).

Redditdominator.com seems to be a marketing service much in the same way upvotes.club, boostupvotes.com, and appsally. Some of the features can be seen in Image 17.

Image 17: Redditdominator Features

The site also bolsters +155 Countries Served and +7000 customers served (redditdominator.com).

The majority of submissions are related to bots that can upvotes, bots that can submit, and bulk creation of accounts on Reddit. The age range of these posts various from 2018 to 2011 at least on the first page of search results. Some users suggest contacting another person who might be able to create or claim to have been able to create the bot themselves.

Image 18: User Claiming to have Used an Upvoting Bot 48

Image 19: User Claiming to have Used an Upvoting Bot

Image 20: User Suggesting Other User For Bot

Image 21: Comment Suggesting to Contact another user for bulk account creation tool

Image 22: Prices for Bulk Account Creation

Some other points of discussion that happen on blackhatworld.com, related to Reddit and the implementation of these tools, methods, and services, includes what the repercussions, punishments, policies, and detection methods of Reddit are, and how many accounts can a user own per IP. The most common punishments levied against the users are “shadowbans” which prevents other users 49 from seeing content from a particular account as well as disabling the account in question. Most of the detection methods discussed involve being able to see how many accounts are signed in at particular times from particular servers or IP Address. These topics are not only in regards to Reddit, but include

Twitter, Facebook, Instagram, YouTube etc…

Blackhatworld.com is one of the more interesting findings from this research. It seems to be derived of marketers and software engineers, and place where people discuss various techniques, tools, or methods for controlling content visibility openly. While it can be argued that the intent of this site and what is shared on it is not overly for the manipulation and controlling of a piece of content’s visibility, or expresses insidious intent, there seems to be an awareness that certain activities are deemed objectionable or in violation of whatever platform’s policies. More research into the activities, tools, and methods shared on blackhatworld.com is greatly encouraged.

The next query involves the building of Reddit bots. Reddit has already proven to be open when it comes to the creation and discussion of bots (Lang et al., 2017), but how easy is it to accomplish this, and what kinds of bots are being created. The first 10 results are all tutorials or step- by-step instructions for how to create a bot on Reddit and includes YouTube videos as well. 50

Image 23: How to Build a Reddit Bot

The first result directs to the website pythonforengineers.com and presents a tutorial for how to make what the creators call “Marvin the Depressed Bot”. Marvin is a bot which responds to a keyword and

(as long as the program is running) is able to make a reply based off of a predetermined list of randomized sardonic quips from movies (pythonforengineers.com). The instructions are easy to follow given that the person has some prior programming experience.

All of the bot tutorials present follow the same kind of format as “Marvin the Depressed Bot”, where the bot looks at comments and posts, identifies some sort of keyword, and performs some 51 logic. That logic might be to respond with a comment of its own, collect data, or any number of other operations.

Nothing about these results are unusual or, at a glance, implies malicious intent. Rather, this fits with some of the sentiments of Reddit as an open discussion place where one is capable of learning new skills. Subreddits are also set up for users to test their new bots in controlled environments and are encouraged to experiment. However, we have seen how some of these textual bots can be used to harass users or derail discussions, and in these cases Reddit has been quick to suspend or ban accounts. This is still a very surface level view of what types of bots are created on

Reddit as there are more elaborate bots such as /u/TicTacToebot among others perform different tasks. We shall later on that these text based bots can be implemented in strange and interesting ways, especially on /r/The_Donald.

4.2 Bot Lists

This section will present curated lists of identified bots on Reddit. I use the word ‘identified’ here for several reasons: 1) the method of scraping that I use strictly returns bots that have identified themselves as such by including the term ‘bot’ in their name. 2) Rather than attempting to uncover hidden bots, this is a simple method in order to generate lists of bots to observe within particular subreddits. Because of Reddit’s philosophy of self-governance and its community orientated distribution, seeing how and for what purpose bots operate on particular subreddits can allow for interesting insights into how that culture functions.

To begin, an SQL script was used on Google BigQuery service to return a list of 20 account names with the term ‘bot’ in them along with the number of times it has posted. The first subreddit that will be looked at is /r/The_Donald which you can see below. All searchers are cleaned in order to remove accounts that do not explicitly mention bots/robots/automation. After this step, each account was observed in order to see whether or not it exhibited bot like behavior or a deleted/suspended account. The criteria for whether or not an account exhibited bot like behavior relied on if there was a notable format with which the account posted. Not all accounts seemed to have this noticeable format, 52 but retained other attributes such as having a low post karma pscore while having a very large comment score. While there were more account names with the term bot in them, not all are presented due to low count scores.

count author post karma comment karma 10249 MAGABrickBot 363 120,195 8207 TrumpTrain-bot 1 11,042 3745 trumpcoatbot 1 88,913 2400 patriobot 1 8,002 1260 DrRubotnik deleted deleted 1017 HelperBot_ 11 111,321 747 gifv-bot 1 335,516 362 RemindMeBot 67 221,441 226 AnindoorcatBot suspended suspended 224 image_linker_bot 5 195,890

Figure 1: List of Accounts with the Term ‘bot’ in their Names from /r/The_Donald

Already there is a level of community specificity regarding the names and sentiments of these bots. At the time of writing this, the first account, MAGABrickBot, has been reported as down to an expiration of an Amazon AWS account. Two images of a typical post by MAGABrickBot can be seen below.

Image 24: Typical Posting by MAGABrickBot 53

Image 25: Typical Posting by MAGABrickBot

The operation of this bot is fairly straight forward. The MAGABrickBot continuously runs and looks for comments that contain the word ‘brick’ in them. Depending on whether or not the original comment specificies a single brick, two or other numbers, the MAGABrickBot replies with one response of several depending of the number of bricks specified. MAGABrickBot also seems to keep a tally of how many times the word ‘brick’ has been mentioned or at least how many times the bot has been activated based off of these parameters. The bot then performs some arithmetic to determine the theoretical percentage of bricks laid down between the Imperial Beach, California and Brownsville,

Texas. We will see later that several bots on this subreddit operate in this way.

This bot is in reference to the Trump border wall between the U.S. and Mexico which acted as a rallying cry during Trump’s presidential campaign, which makes it standout from some of the other bots that follow this procedure in that it is in direct reference to a Trump Administration effort. MAGA is also an acronym for a popular Trump slogan “Make America Great Again”. This account has posted over 10,000 and has a karma score of over 120,000. Karma does not translate into anything other than a metric of how well received an accounts contributions are and if anything we can see that this account is very well received. There are also particularities to the language of the bot, primarily the word ‘centipede’, a term used to connote a Trump Supporter and a reference to the “You Can’t Stump the Trump” YouTube video (attn.com). As we will see in some of the other bot posts, reference to memes and the deep vernacular web are common themes.

The next bot we will look at is the TrumpTrainBot. This bot functions similarly to the

MAGABrickBot in that it runs continuously, replies to certain keywords and posts with exclamatory zeal. An example can be seen below. 54

Image 27: Typical Posting of the TrumpTrainBot

The Trump Train is in reference to the a video retweeted by Trump where a cartoon train hits a CNN

Reporter, much in the same vein as other videos depicting Trump bullying members of the press,

Trump during the presidential campaign, or assaulting various people with the CNN logo superimposed over their faces (snopes.com; knowyourmeme.com). TrumpTrainBot is operated by a user by the name BotsByLiam. This allows for a degree of transparency and despite sharing similar formats with other bots, it is unknown whether or not /u/BotsByLiam is responsible for other bots.

The trumpcoatbot operates similarly and is in reference to a January 2016 rally in Burlington, Vermont where protesters were removed to which Trump responded, “ throw them out. Throw them out in the cold… don’t give them their coat.” (attn.com).

Image 28: Trumpcoatbot post

What is interesting about this post is that includes a citation to the meme it references as opposed to some of the others bots. 55

Patriobot operates slightly differently in there is less of a discernible related meme or reference, and is more of a general ‘shitposter’.

Image 29: PatrioBot

Image 30: PatrioBot

Image 31: Patriobot

The remaining accounts, if they are bots, do not share the same posting format. An interesting observation is the way in which redditors discuss these bots. They are generally regarded as humorous and amicable, but the keywords which cause the bots the reply to a comment are referred to as “triggering” or “summoning”. Triggering can refer to the technical operation which cause the bot to post, but in the context of this socio-political-digital space, where terms like ‘snowflake’ and ‘libtards’ is used regularly, “triggering” seems to take on a double meaning and seems to be used with this double meaning. “Summoning” can be seen as related to the idea of Meme Magic

(motherboard.vice.com). The invocation of these bots can be seen as an automation of memes, which 56 many consider to be a significant factor in the spread and influence of the alt-right’s ideologies

(iolab.net, citation). The remainder of accounts in the list do not exhibit the same kind of behavior as the meme posting bots.

There are some accounts which occur in other subreddits such as /u/RemindMeBot,

/u/HelperBot, /u/gifv-bot, and /u/image_linker_bot. Most bots that have this cross subreddit occurrence tend to exhibit some kind of reasonable function with a recognizable format. For example,

/u/HelperBot converts wikipedia pages which were posted via a mobile link to non-mobile links. /u/gifv- bot and /u/image_linker_bot perform similar functions. /u/RemindMeBot (when it is triggered) will remind the the poster of the comment it replies to a year from the posting date.

There are accounts which are in reference to bots or robots, but are not included in this tables because upon further inspection, they did not appear to bot despite their names. For example, an account by the name /u/captcha_bot had a post karma of over 12,000 and did not seem to follow a recognizable format for posting.

The next two subreddit we will be looking at are /r/bitcoin and /r/botwatch. content author post karma comment karma 230 mr_robot-sh deleted deleted 216 TweetsInCommentsBot 1 1,033,652 95 RemindMeBot 67 221,441 39 sneakpeekbot 31 458,368 31 HelperBot_ 11 111,321 20 yourewelcome_bot 1 10,535 15 gifv-bot 1 335,516 15 FallacyExplnationBot suspended suspended 12 could-of-bot suspended suspended 7 image_linker_bot 5 195,890 Figure 1: /r/Bitcoin

There are several accounts here that are suspended or deleted and it is unclear why this is so. There does not seem to be a noticeable sentiment in the functionalities or in the naming of these bots. Most of these bots can be seen in other subreddits and sevre reasonable functions.

/u/TweetsInCommentsBot replies to comments that include links to twitter with a post which includes 57 the content of the post, the post time, and other information related to the tweet. The

/u/yourewelcome_bot is a good example of a different type of bot that does not serve a true function other than replying with the text “you’re welcome” anytime “thank you” or “thanks” appears in a comment. What makes this interesting is that opposed to the more functional bots, this bot is novel and humourous. And while some of the more functional bots have methods to disable in hopes of limiting their postings and allowing users more control, this bot is less controlled and can be considered annoying (Long et al., 2017). It also seems like this is behavior that leads to suspension and banishment.

count author post karma comment karma 90 blackjack_bot 30 5,367 11 TrollaBot not found not found 8 MapLinkerBot 1 1 3 thank_mr_skeltal_bot 82 184,030 1 QuoteMe-Bot suspended suspended 1 HelperBot_ 11 111,321 1 FrugalPriceBot 1 1 1 Google-It-Bot 1 4

Figure 3: /r/botwatch

In /r/bitcoin and /r/botwatch there is a significant drop off in the occurrence of these types of accounts

There can be several different reasons for result including that most likely when bots are created they do not explicitly identify themselves as such. This is also a community dedicated to actual discussion of bots and takes that sentiment seriously. There even seems to be a sentiment of actively removing annoying of harmful bots, while preserving and encouraging experimentation and creation of bots.

There also seems to be a sentiment of protecting bots and Reddit being a space where regulation and administration should not get in the way of deployment and concealment (if the user so chooses not to explicitly identify their bot). This can be seen in the comment below. 58

Image 32: Comment from /r/Botwatch

All that being said, the accounts that appear are interesting and do reflect something of the subreddit’s sentiments. /u/blackjack_bot is a bot that plays blackjack with users, using ascii art in order to display cards. It seems like a legitimate experiment in programing and design. /u/TrollaBot does not have an actual user account page that it redirects to, however, it does have a dedicated subreddit with 715 subscribers. From looking at posts to the subreddit, it is hard to understand what the intent of the subreddit and the (what at least seems to be) a disabled bot account. The subreddit claims that the function of the bot is that when summoned, it will “tell you things about yourself” (reddit.com). There are also posts to the subreddit lamenting its shadowbanning with many posts having “RIP” in them. It seems like the community expresses remorse for bot accounts that are not functional, but still seem interesting. Or perhaps the entire page is an ironic troll on the bot community (as suggested by the name), by convincing the community that a bot existed, that could do this tasks, but it never really did.

We can see other mixtures of functionality, experimental, and humor in some of these other bots including /u/MapLinkerBot, /u/thank_mr_skeltal_bot, /u/FrugalPriceBot, and /u/Google-It-Bot.

/u/MapLinkerBot, /u/FrugalPriceBot, and /u/Google-It-Bot are all functional, but seem to be in early testing phases. Their function is related to their name as /u/MapLinkerBot posts a google map link if requested, /u/FrugalPriceBot compares prices of a particular item, and /u/Google-It-Bot posts a google link with the requested query. /u/thank_mr_skeltal_bot is a humorous bot, posting words like “updoot” and praising bones and calcium. 59

Image 33: /u/thank_mr_skeltal_bot

These bots seem to have a high degree of transparency and again this is because these accounts have identified themselves as such. Also, in light of Reddit 2017 Transparency Report, many accounts have been suspended and banned. Out of the data that we have looked at so far, the number of occurrences of accounts in /r/The_Donald is significantly higher than /r/botcoin and

/r/botwatch. Because of similar scrapes were performed on the subreddits /r/funny, /r/gaming,

/r/KotakuInAction, /r/btc, /r/politics, /r/racism, and /r/EnoughTrumpSpam. These subreddits were selected due to their presence in Reddit’s 2017 Transparency Report and their socio political ideologies. Similar results were found in that the account occurrences were surprisingly low with the exception of /r/EnoughTrumpSpam and /r/politics which are presented below. count author post karma comment karma 25979 SnapshillBot 257 286,672 271 JoeBidenBot 1 94,653 259 youtubefactsbot 1 237,887 137 HelperBot_ 11 111,321 39 SupremeRedditBot 13 3,960 32 FallacyExplnationBot suspended suspended 30 Mock_Salute_Bot 1 21 28 sneakpeekbot 31 458,368 23 TrollaBot not found not found 22 AlternateFactsBot 1 4,062 18 could-of-bot suspended suspended 17 thank_mr_skeltal_bot 82 184,030

Figure 4: /r/EnoughTrumpSpam 60

count author post karma comment karma 19458 PoliticsModeratorBot 678,947 100 211 Mock_Salute_Bot 1 21 210 jetez_vos_sabots deleted deleted

Figure 5: /r/politics

An interesting comparison can be seen between these two tables, mainly the diversity between them. / r/politics has nearly 4 million subscribers, while /r/EnoughTrumpSpam has around 86 thousand. There can be several reasons for the difference in bot accounts. A possible reason is that since /r/politics is so much larger, active banning and moderation is more frequent. /r/EnoughTrumpSpam is more of a niche community, acting as an antithesis to /r/The_Donald. It's more niche community might be what allows it to have at least more account occurrences, and because its subject matter is so closely related to the activity of the alt-right and /r/The_Donald, crossover between communities can be accepted.

Some noticeable bots from /r/EnoughTrumpSpam are /u/SnapShillBot and /u/JoeBidenBot. Bot accounts express sentiments that are related to the cultural ideologies of their subreddit.

/u/SnapShillBot is able to archive posts and also has a subreddit where the bot can be discussed.

‘Shill’ is in reference to various characters in the media, politics, business etc… who are lakies, puppets, plants, or stooges (wikipedia.org) The overall function of bot seems to be geared toward the documentation of interactions on Reddit. /u/JoeBidenBot is a humorous bot and one of the few we have seen outside of the /r/The_Donald that is community specific and highly used. It is difficult to know exactly if /u/JoeBidenBot is an actual bot because it does not follow noticeable reoccurring format, at least in the same way that we have seen. The bot appears to respond to mentions of Joe

Biden or Barack Obama and responds with some text. It is not unreasonable that that text is informed by a collection of quotes from Biden and used as a source for its replies. This account also has a subreddit in where the creators address in an FAQ “Q: Is this really a bot, A: If you can’t tell, why should we.” This expresses a sentiment of playfulness, non-seriousness, and technical knowledge 61 which seems to be indicative of Reddit as a whole (Massanari, 2015). Unfortunately, at the time of writing this, /u/JoeBidenBot is offline.

The other bot accounts that we see in /r/EnoughTrumpSpam follow recognizable formats of posting behavior. /u/youtubefactsbot is similar to the twitter bot we saw earlier in that it responds to youtube links by posting a comment with the title, runtime, and views of the video. This bot also has a subreddit. /u/SupremeRedditBot congratulates a user who has had a post reach the /r/all page.

/u/AlternateFactsBot seems to respond to a comment which contain the phrase “alternative facts” or some similar phrase, in which it replies with an exact regurgitation of the post its replying to or “Nah.

Lies”. There are different replies, however, suggesting that either the architecture of the bot changes or that a human operator occasional responds to comments.

Reddit is an interesting environment with its techno-libertarian ideology as well as its allowance to let communities cluster around themselves. The open discussion of bots in different subreddits

(both for a specific and bots in general) allows for a degree of transparency from the administrative side of Reddit, but allows redditors to express their feelings on bots as well. An example of this is the / u/Goodbot_Badbot bot. This bot was recommended through the botwatch community and is a bot that is able to record the submission of user on whether a bot is good or bad. The results are collected on a website https://goodbot-badbot.herokuapp.com/ where those bots are then rated. 62

Image 34: GoodBot_BadBot Rankings

On the homepage we can see honorary good bots and honorary bad bots as well as the top ten bot accounts as rated by the Wilson score (evanmiller.org). Below is an image of the top 25 accounts. 63

Image 35: Top 25 Goodbots

While this is a good example of how the Reddit community and Reddit administration record sentiments of bot accounts, there is evidence of this method being abused as well. In a post on the /r/

TheoryOfReddit subreddit, a submission from 9 months at the time of writing goes into the history and 64 functionality of the bot shows examples of how, after a few days of being online, different subreddits started creating threads with the sole purpose of calling a particular bot and boost its rating.

Image 36: Example of Goodbot_BadBot Boosting

The bot in question here, /u/The-Paranoid-Android, is found on a subreddit called /r/SCP (Secure

Contain Protect). The bots post links to a fictional inter-governmental foundation’s website in which it accounts various phenomena in an online collaborative writing project. (spc-wiki.net). The author of the post examining /u/GoodBot_BadBot’s flaw also points to the easy spamability of the voting system as the as the primary flaw and point of abuse. /u/GoodBot_BadBot also has a subreddit where many users have expressed both disdain for the bots spam and confusion over whether or not it is still running.

The example of /u/GoodBot_BadBot presents an interesting microcosm for Reddit and its efforts to be transparent, remove harmful or malicious bots, and certain communities’ abuse of the voting system. Reddit’s open environment encourages experimentation, but also allows for a vulnerable network of individuals and information. It is extremely difficult to detect bots that either do not identify themselves as such or exhibit recognizable bot behavior, and it becomes even more 65 precarious when we start to consider a market of selling and buying upvotes from accounts that are operated by humans. The methodology presented in this section was primarily used to identify bots in particular communities in order to see what kind of bots are being made. The next section of this thesis will present a theoretical discussion on how bots seem to operate on Reddit and what SEO growth hacking tactics.

5 Discussion

After having observed bots on Reddits over the course of this thesis, there are several different threads through discussions of Reddit and the way automation, technology, culture, and economics affect the ways information flows on the site. This chapter will explore some of these concepts through the lens of Reddit and offer several different points to how bots seem to be operationalized. First, the classification of behavior and implementation of bots on Reddit in a taxonomy. This taxonomy will be broken into two main branches of divergence (visible and invisible) which will have dedicated sections in this chapter. Second, SEO strategies which seem at odds with Reddit’s content policy. And lastly, comments regarding the culture of Reddit both as a whole and specific communities. Suggestions will also be made regarding methodological limitations and future avenues of inquiry into bot research on

Reddit.

5.1 Automation, Technology, Culture, and Economics

The historical background for the use of simple computer programs to perform automated tasks becomes a difficult concept to reconcile with how this technology affects the network of individuals and computers they interact with. This is most prevalent on social media sites, but when we consider a content aggregator like Reddit, it would seem like the use of bots is overall a more accepted practice despite the calls of abuse. Reddit’s classification as a content aggregator even parallels a typical functionality of bots suggesting some sort of analogous relationship. Reddit’s culture is also problematic for several different reasons. One is Reddit’s tendency toward harassment. While not all of Reddit is overtly misogynistic or controversial, certain communities within seem to be 66 entrenched in particular problematic behavior and ideology. Second is how information flows through the site, what information receives more attention, and the authority derived from a mastery of information. Third is Reddit’s self-governance and overall structure. Reddit sees itself as an authentic space, where minimal moderation will allow for the community’s voice to ring true. Considering the use of bots seems to allow for a considerable degree of moderation as suspicious accounts are often banned or suspended, bots are not without their legitimate functionalities in order to quell the spread of controversial information although they are considered to be slow in the process.

When we consider how Reddit’s culture and infrastructure mitigates the flow of information on the site, control over that information is a valuable commodity. The cases where human operators use accounts in order to act in accord with financial incentive becomes equally as precarious as automation in part because it makes research methodologies difficult to implement and because it still exploits a vulnerability and signifies inauthentic intent. The cases where economics and politics are the primary catalysts for abuse indicate how control over the flow information online is sought after and manipulated is growing in more technological and sophisticated ways. Massanari seems to describe it most succinctly as a negotiating of space without romanticizing democratic potential. In the more homophilic centers of Reddit, the negotiation of space is largely skewed toward that of the primary thought collective, where users even refer to it as a hive mind. We have seen how bots collected in these spaces are imagined and implemented and I will elaborate on an attempt to classify those bots in the following section.

5.2 Taxonomy of Bots

The two primary bot types found within reddit can be imagined as visible and invisible. This this distinction may seem obvious at first, the presence of a voting system that acts as a black box to those who are not Reddit admins obfuscates traces of there activity. Given the importance Reddit’s ranking algorithm and the prevalence of vote manipulation, information on how content is ranked can prove to be useful to researchers. 67

5.2.1 Visible bots

Visible bots are typically textual. They post, comment, and reply if their parameters are met to trigger their programming. This is the most common type of bot strictly based on their observable nature. As we have seen in earlier chapters there are several different implementations of these programs.

First, we will consider the visible bots observed on /r/The_Donald. These bots seem to be memetic, referencing in-group colloquialism and vernaculars to create highly used and popular bots. In

Lessig’s book, he examine how the software of a program can act as an omnipresent enforcer of rules and normative codes of conduct (Lessig, 1999). Lessig primarily uses this example, “code is law” to describe the ways in which digital infrastructures regulate the activity found on them, but in this case, a normative code of conduct can be imagined in spaces that further entrench cultural tendencies. The notion that these bots also continuously run and respond to certain key trigger words allows them to persist and enforce cultural norms. These types of bots can be seen as cultural digital artefacts, encapsulating ideologies and sentiments of their communities.

Other examples of these memetic bots can be found in other subreddit like the JoeBidenBot on

/r/EnoughTrumpSpam. Joe Biden is a popular political figure in the U.S. and is typically seen as humorous by proponents of the left. There are also bots that seem to be memetic in that seem to be in reference to some other original lineage of socially transmitted activity or information. Examples include AlternateFactsBot, SnapShillBot, and thank_mr_skeltal_bot. Both SnapShillBot and thank_mr_skeltal_bot are slightly different. The use of the term shill implies a certain ideology toward crony capitalism, but performs an archiving function rather than posting jokingly. Thank_mr_skeltal_bot is flippant, but is not specific to any one community.

Lastly, there are the bots that act as moderators or administrative and functional tools. Most of the ones that we have seen thus far reposts metadata information of links from Twitter, Youtube, and

Wikipedia. Others act as bookmarks to remind users of information.

All things considered, the bots observed over the course of this research primarily dealt with bots that self identified. There are also unidentified bots that might be experimental or share similarities with the visible bots described above. 68

5.2.2 Invisible Bots

Visible bots are much more aligned with the pseudonymous nature of Reddit, in that they can be named, unlike their invisible, completely anonymous counterparts. Visible bots, despite running continuously, need to be triggered in order to perform their functions while invisible bots seem to deployed for specific purposes. Visible bots are concerned with content and text, while invisible bots are predominantly situated around Reddit’s voting mechanics. Visible are much more easy to detect, while invisible bots are far more difficult to observe.

The effects of invisible bots are most closely related to vote manipulation, and while vote nudging and SEO strategies do not necessarily require automated processes in order to be effective, it might seem to be a misnomer to refer to these entities as bots. However, I would still argue that the word bot, colloquially, implies concealing one’s identity in order to manipulate the system. Both Ferrara et al., 2014 and Boshmaf et al., 2011 refer to “cyberborgs” which are operated both by humans and automation in effort to further mimic human behavior online. This distinction, which Heder also notes that this demarcation of activity and intent is what makes SEO strategies potentially unlawful.

Because invisible bots are difficult to observe through this research, some points gleaned from

Heder’s work on social media black markets and black hat SEO forums might suggests key traits of invisible bots. While many postings on microworkers.com desired the creation of accounts, there are regulations in place to detect the creation and use of many bots at once. The age and activity contributing to the “quality” of accounts. Heder also notes that micro tasks are typically performed in undeveloped countries (Heder, 2018). And, if the voting habits of accounts are unrelated, this might also suggest that accounts are voting against their typically behavior.

5.3 Suggestions

This section will cover some suggestions for future research and comments on Reddit’s culture and infrastructure. The methodology present in this research only serves to identify bots which identity themselves as such. This leaves a problem for researchers interested in bots which are not as easily identified. As long as these bots are still mostly active in comment threads, comparing texts between 69 posts and against large corpuses of public text data might reveal unidentified bots. In fact, at the time of concluding this research, a user on Reddit’s BigQuery subreddit post this API Key called

“reply_delay”.

Image 37: Reddit Post Recommened Pushift BETA API Key ‘Reply_Delay’

This key is capable of returning comments which replied to a post within 10 seconds of posting.

Because one of the advantages of automation is there immediacy, this metric might be a tell tale sign of bot activity. Unfortunately, due to time constraints, this method was not operationalized.

At the moment information regarding Reddit’s vote data is only available to admins. This includes geolocation metadata which can detect how many accounts are operating on a particular server or IP address. Reddit has done a good job of being transparent with its own internal investigation and compliance with government bodies to produce information. Reddit even seems to be proactive when it comes to the suspension of accounts. However, it is less proactive in the suspension and removal of problematic content and accusations of harassment fail on the moderators of communities rather than

Reddit’s administrative team. This allows many instances to fall through the cracks. These kind of practices along with Reddit’s culture makes Reddit’s smaller communities especially vulnerable to abuse. Tools exist to assist community moderators, however, it seems there is a sentiment that Reddit is not as supportive to these communities.

Lastly, SEO strategies, while not explicitly unlawful are in violation of Reddit’s content policy which prohibits vote manipulation. A question brought up earlier in this thesis revolved around whether it was the responsibility of platforms or governments to to regulate bot activity, but in question of whether or not a service which violates another sites content policy, the responsibility seems to fall on 70 the shoulders of the platform, much in the same way illicit content is removed by the site. Perhaps greater transparency, including the requirement that creating a bot through Reddits’s API should always include the term bot, or that information that is problematic or suspicious becomes flagged. It is unclear what Reddit investigative methods are, but there is a greater amount of data that not open to the public.

6 Conclusion

It is difficult to know where the future of automation will lead us. Some think that it will lead to the liberation of of the workforce, others see it as a means to our end, and the rest agonize somewhere in between. When we consider bots on Reddit we see a community which prides itself on authentic and quality information, but shares illicit content, harasses groups of people who do not fit a white male centrality, and rampantly spread misinformation aligned with the culture’s ideology. In an article for the New Yorker titled, “Reddit and the Struggle to Detoxify the Internet” Andrew Marantz highlights key aspects of Reddit, namely its desire for authenticity and censorship, free speech and hate speech, and the way Reddit CEOs have tried to reign in some more vocal, but problematic communities (newyorker.com. Marantz ends his piece by presenting an experiment conducted in the

April 2017 on the subreddit /r/place. For several days, this experiment was left open, and users were able to simply change the color of a single pixel on a 1,000 X 1,000 pixel canvas. The only caveat, users could only change a color every 5 minutes.

The result is just as interesting as the process, where space is negotiated through collective effort, large groups of individuals rally together in order to take over large portions of the map, some efforts are futile, some efforts are successful. At one point, a black spot referred to as the Void engulfing previous works into its nothingness, but in its nothingness it allowed for new art to take shape (sudoscript.com). The result is magnificent thought. Negotiated space, free of hateful symbols or sentiments, representing different groups with different interests. Its an image and an idea that fills those who agonize in the middle with hope and determinations. 71

Image 38: Completed image from /r/place 72

Works Cited

2017 Will Be Remembered As The Year Of Bitcoin.

https://www.forbes.com/sites/outofasia/2017/10/25/bitcoins-ipo-moment-has-arrived/

#6dda8311c1f3. Accessed 27 May 2018.

A Primer on Political Bots - Part Two: Is It a Bot? It’s Complicated! | News & Analysis | Data Driven

Journalism.

http://datadrivenjournalism.net/news_and_analysis/a_primer_on_political_bots_part_two_is_it_a_

bot_its_complicated. Accessed 12 May 2018.

AltRightOpenIntelligenceInitiative < Dmi < Foswiki.

https://wiki.digitalmethods.net/Dmi/AltRightOpenIntelligenceInitiative. Accessed 10 June 2018.

Are Computers Merely “Supporting” Cooperative Work. https://dl.acm.org/citation.cfm?id=2441970.

Accessed 28 May 2018. arXiv, Emerging Technology from the. “How to Spot a Social Bot on Twitter.” MIT Technology Review,

https://www.technologyreview.com/s/529461/how-to-spot-a-social-bot-on-twitter/. Accessed 12

May 2018.

“Black Hat SEO Tools.” BlackHatWorld, https://www.blackhatworld.com/forums/black-hat-seo-tools.9/.

Accessed 21 May 2018.

BlackHatWorld. https://www.blackhatworld.com/. Accessed 21 May 2018.

“Bot Briefing Book.” Google Docs,

https://docs.google.com/document/d/1kx7E8TP4AzEd1n6ToBXVtVvs-FhOofW8u1_R2PKEA_M/

edit?usp=sharing&usp=embed_facebook. Accessed 28 May 2018.

Bot Definition - Google Search. https://www.google.nl/search?

q=bot+definition&oq=bot+definition&aqs=chrome.0.0j35i39j0l4.5349j1j7&sourceid=chrome&ie=U

TF-8. Accessed 27 May 2018. 73

Bot Etymology - Google Search. https://www.google.nl/search?

ei=AQ4LW_yqEIfVgAahxK7ACA&q=bot+etymology&oq=bot+etymology&gs_l=psy-

ab.3...119745.123657.0.123946.0.0.0.0.0.0.0.0..0.0....0...1.1.64.psy-ab..0.0.0....0.noHoWM1otyI.

Accessed 27 May 2018.

Bots, #Strongerin, and #Brexit: Computational Propaganda During the UK-EU Referendum by Philip

N. Howard, Bence Kollanyi :: SSRN. https://papers.ssrn.com/sol3/papers.cfm?

abstract_id=2798311. Accessed 28 May 2018.

Cassa, Christopher A., et al. “Twitter as a Sentinel in Emergency Situations: Lessons from the Boston

Marathon Explosions.” PLoS Currents, vol. 5, July 2013. PubMed Central,

doi:10.1371/currents.dis.ad70cd1c8bc585e9470046cde334ee4b.

“Cloaking.” Wikipedia, 16 Jan. 2018. Wikipedia, https://en.wikipedia.org/w/index.php?

title=Cloaking&oldid=820842646.

“Cloaking and Content Generators.” BlackHatWorld, https://www.blackhatworld.com/forums/cloaking-

and-content-generators.2/. Accessed 21 May 2018.

CoinMall. “How Easy and Cheap It Is to Manipulate Reddit Discussions.” Medium, 30 Nov. 2017,

https://medium.com/@coinmall/how-easy-and-cheap-it-is-to-manipulate-reddit-discussions-

4139a488542.

Coleman, Whitney Phillips, Jessica Beyer, and Gabriella. “Trolling Scholars Debunk the Idea That the

Alt-Right’s Shitposters Have Magic Powers.” Motherboard, 22 Mar. 2017,

https://motherboard.vice.com/en_us/article/z4k549/trolling-scholars-debunk-the-idea-that-the-alt-

rights-trolls-have-magic-powers.

Data-Memo-First-Presidential-Debate.Pdf. https://regmedia.co.uk/2016/10/19/data-memo-first-

presidential-debate.pdf. Accessed 28 May 2018. 74

Data-Memo-US-Election.Pdf.

http://blogs.oii.ox.ac.uk/politicalbots/wp-content/uploads/sites/89/2016/11/Data-Memo-US-

Election.pdf. Accessed 28 May 2018.

@DFRLab. “#BotSpot: Twelve Ways to Spot a Bot.” DFRLab, 28 Aug. 2017,

https://medium.com/dfrlab/botspot-twelve-ways-to-spot-a-bot-aedc7d9c110c.

Evans, Garrett. “Warner Sees Reddit as Potential Target for Russian Influence.” TheHill, 27 Sept.

2017, http://thehill.com/policy/technology/352584-warner-sees-reddit-as-potential-target-for-

russian-influence.

“FACT CHECK: Did Trump Retweet Cartoon of Train Hitting CNN Reporter?” Snopes.Com, 15 Aug.

2017, https://www.snopes.com/fact-check/trump-retweet-cartoon-train-hitting-cnn-reporter/.

“FALSE: Comet Ping Pong Pizzeria Home to Child Abuse Ring Led by Hillary Clinton.” Snopes.Com,

21 Nov. 2016, https://www.snopes.com/fact-check/pizzagate-conspiracy/.

Faq - Reddit.Com. https://www.reddit.com/wiki/faq. Accessed 20 June 2018.

Ferrara, Emilio, et al. “The Rise of Social Bots.” Commun. ACM, vol. 59, no. 7, June 2016, pp. 96–104.

ACM Digital Library, doi:10.1145/2818717.

Forelle, Michelle, et al. “Political Bots and the Manipulation of Public Opinion in Venezuela.”

ArXiv:1507.07109 [Physics], July 2015. arXiv.org, http://arxiv.org/abs/1507.07109.

#Gamergate and The Fappening: How Reddit’s Algorithm, Governance, and Culture Support Toxic

Technocultures - Adrienne Massanari, 2017.

http://journals.sagepub.com/doi/abs/10.1177/1461444815608807. Accessed 28 May 2018.

Geiger, R. Stuart. The Lives of Bots. SSRN Scholarly Paper, ID 2698837, Social Science Research

Network, 2011. papers.ssrn.com, https://papers.ssrn.com/abstract=2698837.

Get a Whole New Identity at the Fake Name Generator. https://www.fakenamegenerator.com/.

Accessed 28 May 2018. 75

“Google’s Shutting the Door on Doorway Pages for SEO.” Brafton, 27 Mar. 2015,

https://www.brafton.com/news/seo-1/googles-shutting-door-doorway-pages-seo/.

Héder, Mihály. “A Black Market for Upvotes and Likes.” ArXiv:1803.07029 [Cs], Mar. 2018. arXiv.org,

http://arxiv.org/abs/1803.07029.

How Not To Sort By Average Rating – Evan Miller. http://www.evanmiller.org/how-not-to-sort-by-

average-rating.html. Accessed 24 May 2018.

How to Check If a Twitter Account Is a Bot. https://www.howtogeek.com/325232/how-to-check-if-a-

twitter-account-is-a-bot/. Accessed 12 May 2018.

“In Response to Recent Reports about the Integrity of Reddit, I’d like to Share Our Thinking. •

r/Announcements.” Reddit,

https://www.reddit.com/r/announcements/comments/827zqc/in_response_to_recent_reports_abo

ut_the_integrity/. Accessed 27 May 2018.

“Infographic: The Explosive Growth of Reddit’s Community.” Statista Infographics,

https://www.statista.com/chart/11882/number-of-subreddits-on-reddit/. Accessed 26 May 2018.

Lartey, Jamiles. “Trump-Russia Investigation May Target Reddit Posts, Says Senator’s Aide.” The

Guardian, 27 Sept. 2017. www.theguardian.com,

http://www.theguardian.com/technology/2017/sep/27/trump-russia-investigation-reddit.

Lessig, Lawrence. Code: And Other Laws of Cyberspace. ReadHowYouWant.com, 2009.

Machines and Mindlessness: Social Responses to Computers - Nass - 2000 - Journal of Social Issues

- Wiley Online Library. https://spssi.onlinelibrary.wiley.com/doi/full/10.1111/0022-4537.00153.

Accessed 28 May 2018.

Malmgren, Evan. “Don’t Feed the Trolls.” Dissent, vol. 64, no. 2, Apr. 2017, pp. 9–12. Project MUSE,

doi:10.1353/dss.2017.0042. 76

Mccaskill, Nolan D. “Trump Credits Social Media for His Election.” POLITICO,

https://www.politico.com/story/2017/10/20/trump-social-media-election-244009. Accessed 27 May

2018.

[Method] How To Get REDDIT Upvotes For Almost Free | BlackHatWorld.

https://www.blackhatworld.com/seo/method-how-to-get-reddit-upvotes-for-almost-free.849749/.

Accessed 21 May 2018.

MrContent. “Accessible Marketing That Generates Sales from Day One ➜ Upvotes.Club.”

Upvotes.Club, https://upvotes.club/. Accessed 21 May 2018.

---. “What Can Be Done?” Upvotes.Club, 29 Apr. 2016, https://upvotes.club/what-can-be-done/.

Neff, Gina, and Peter Nagy. “Automation, Algorithms, and Politics| Talking to Bots: Symbiotic Agency

and the Case of Tay.” International Journal of Communication, vol. 10, no. 0, Oct. 2016, p. 17.

“Protect Your Account with Two-Factor Authentication! • r/Announcements.” Reddit,

https://www.reddit.com/r/announcements/comments/7spq3s/protect_your_account_with_twofactor

_authentication/. Accessed 27 May 2018.

Ratkiewicz, J., et al. Detecting and Tracking Political Abuse in Social Media. p. 8.

“Reddit Advertising Policy.” Reddit Help, https://www.reddithelp.com/en/categories/advertising/policy-

and-guidelines/reddit-advertising-policy. Accessed 27 May 2018.

Reddit AIO Bot - About Reddit Upvote Bot. https://redditbot.com/. Accessed 20 May 2018.

“Reddit Bots.” BlackHatWorld, https://www.blackhatworld.com/seo/reddit-bots.884728/. Accessed 21

May 2018.

Reddit Is Finally Fixing Its Trump Spam Problem. https://gizmodo.com/reddit-is-finally-fixing-its-trump-

spam-problem-1792061056. Accessed 26 May 2018.

“Reddit Upvoter Bot.” BlackHatWorld, https://www.blackhatworld.com/seo/reddit-upvoter-bot.965943/.

Accessed 21 May 2018.

Reddit.Com: Content Policy. https://www.reddit.com/help/contentpolicy/. Accessed 27 May 2018. 77

RedditDominator | Best Reddit Marketing Tool and Software. http://redditdominator.com/#. Accessed

21 May 2018.

Researcher, Tim Squirrell, and Alt-Right Open Intelligence Initiative. “Linguistic Data Analysis of 3

Billion Reddit Comments Shows the Alt-Right Is Getting Stronger.” Quartz,

https://qz.com/1056319/what-is-the-alt-right-a-linguistic-data-analysis-of-3-billion-reddit-

comments-shows-a-disparate-group-that-is-quickly-uniting/. Accessed 17 May 2018.

Search Results for Query: Reddit | BlackHatWorld. https://www.blackhatworld.com/search/26700339/?

q=reddit&o=relevance&c[node]=9. Accessed 21 May 2018.

Stokel-Walker, Chris. “There Are Some Major Issues with Claims Russian Bots Swayed the Election in

Jeremy Corbyn’s Favour.” WIRED UK, http://www.wired.co.uk/article/jeremy-corbyn-russia-twitter-

research. Accessed 20 May 2018.

The Socialbot Network. https://dl.acm.org/citation.cfm?id=2076746. Accessed 28 May 2018.

“The Types of Manipulation on Vote-Based Forums : TheoryOfReddit.” Archive.Is, 23 May 2015, http://

archive.is/RdPlQ.

Transparency Report 2017 - Reddit. https://www.redditinc.com/policies/transparency-report. Accessed

27 May 2018.

Trollabotting! https://www.reddit.com/r/trollabot/. Accessed 24 May 2018.

Turing, Alan M. “Computing Machinery and Intelligence.” Parsing the Turing Test, Springer, Dordrecht,

2009, pp. 23–65. link.springer.com, doi:10.1007/978-1-4020-6710-5_3.

“Vote Manipulation on Reddit? I Show Examples of Bots • r/WikiLeaks.” Reddit,

https://www.reddit.com/r/WikiLeaks/comments/5qxdwv/vote_manipulation_on_reddit_i_show_exa

mples_of/. Accessed 20 May 2018.

Webb, Steve, et al. Social Honeypots: Making Friends With A Spammer Near You. p. 11.

What Is Black Hat SEO? Webopedia Definition.

https://www.webopedia.com/TERM/B/Black_Hat_SEO.html. Accessed 21 May 2018. 78

“When Pixels Collide.” Sudoscript, 4 Apr. 2017, http://sudoscript.com/reddit-place/.

Why Reddit Won’t Ban The_Donald - Vox. https://www.vox.com/culture/2017/11/13/16624688/reddit-

bans-incels-the-donald-controversy. Accessed 27 May 2018.

Wisdom of the Crowd or Technicity of Content? Wikipedia as a Sociotechnical System - Sabine

Niederer, José van Dijck, 2010. http://journals.sagepub.com/doi/abs/10.1177/1461444810365297.

Accessed 28 May 2018.

Woolley, Samuel C. “Automating Power: Social Bot Interference in Global Politics.” First Monday, vol.

21, no. 4, Mar. 2016. Media reports on bots on social media for political means.,

uncommonculture.org, doi:10.5210/fm.v21i4.6161.

Woolley, Samuel C., and Philip N. Howard. “Automation, Algorithms, and Politics| Political

Communication, Computational Propaganda, and Autonomous Agents — Introduction.”

International Journal of Communication, vol. 10, no. 0, Oct. 2016, p. 9.

Zannettou, Savvas, et al. “The Web Centipede: Understanding How Web Communities Influence Each

Other Through the Lens of Mainstream and Alternative News Sources.” Proceedings of the

2017 Internet Measurement Conference, ACM, 2017, pp. 405–417. ACM Digital Library,

doi:10.1145/3131365.3131390.