sign in sign up
<<
Home , Online identity

Misleading Repurposing on

Tuğrulcan Elmas Rebekah Overdorf LSIR, EPFL LSIR, EPFL Lausanne, Switzerland Lausanne, Switzerland [email protected] [email protected] Ömer Faruk Akgül Karl Aberer Bilkent University LSIR, EPFL Ankara, Turkey Lausanne, Switzerland [email protected] [email protected]

ABSTRACT and even the style or language of the tweets. We refer to this type Twitter allows users to change their screen name and other profile of drastic shift in characteristics and subject as repurposing. attributes, which allows a malicious to change their account’s While not all instances of account repurposing are misleading or or purpose while retaining their followers. We present malicious, this behavior can be indicative of misconduct. Take, for the first large scale and principled study of this phenomenon of example, a malicious user who wants to spread political propaganda misleading account repurposing on Twitter. We analyze two large on Twitter. He creates an account that appeals to certain users, e.g. datasets to understand account repurposing. We find 3,500 repur- a catphishing account, with the goal of increasing the number of posed accounts in the Twitter Elections Integrity Datasets. We also followers. He uses the account only to maximize follows and other find more than 100,000 accounts that have more than 5,000 follow- types of engagements — an activity we refer to as grinding. Once ers and were active in the first six months of 2020 using Twitter’s the account has a sufficient number of followers, the malicious user 1% real-time sample. We analyze a series of common features of changes the characteristics of the profile to be a patriotic retiree repurposed accounts that give us insight into the mechanics of and uses the account to spread political propaganda. repurposing and into how accounts can come to be repurposed. In this example, the account itself never changes hands; the same We also analyze three online markets that facilitate selling and person always holds the account credentials. However, repurposing buying Twitter accounts to better understand the effect that repur- is common when account ownership is transferred. One way an posing has on the Twitter account market. We provide the dataset of account can change hands is by mutual agreement of the previous popular accounts that are flagged as repurposed by our framework. and new owners, which may be the result of commercial activity. We see this most often with accounts that have a high follower KEYWORDS count (i.e. popular accounts). The new owner bought the account for a reason (e.g. political propaganda, spam) so they then change twitter, sockpuppets, disinformation the account to serve the purpose for which they bought it. Such accounts can even be rented out temporarily to different clients at different times, with the account being repurposed with each 1 INTRODUCTION transaction. An account can change hands when it is compromised, "As Gregor Samsa woke one morning from uneasy dreams, he found at which point the new owner can remove all content of the previous himself transformed into some kind of monstrous vermin.” - Franz account owner so that they can use the account for another purpose, Kafka / Metamorphosis [18] again retaining the follower count. platforms allow users to change their profile in- It may seem more complicated to repurpose an account than formation in order to keep up with real world or online identity create a new one, but repurposing is more attractive for certain changes. For example, a user may change their real-world name adversaries. Repurposing helps a malicious user build in an arXiv:2010.10600v1 [cs.SI] 20 Oct 2020 and want their online identity to reflect that change, they may want account. Users may trust an older account with many followers to make their profile more anonymous, they may want to make more than a new one with few followers [23]. Similarly, fake account their profile harder to find via search, or they may make acareer detection algorithms are more lenient towards older accounts with change and want to change their description field to reflect it. a history of engagements and followers over new accounts with Not all attribute changes are genuine, however. Sporadically, no or few followers [8, 15]. Repurposing is also more economic, in journalists, bloggers, activists, and everyday Twitter users report some sense, as adversaries can repurpose an account while keeping instances of accounts that were formerly different in every way the associated email address and the phone number the same. changing overnight, like Gregor Samsa in the Kafka novel: the ac- While repurposing can occur on any platform, exact policies for count of a young man tweeting about French politics switches to an changing profile information vary, meaning that some platforms impersonal account promoting Pixar movies, or the account of an are more susceptible to account repurposing. Controversially [16, attractive young American women switches to a patriotic middle- 19, 21], limits accounts to be only personal profiles that aged man tweeting only in Chinese. In instances such as these, correspond to a real person [4] and disallows owning multiple accounts keep their followers, but transform all of their character- accounts, so and name changes are uncommon and istics overnight: name, screen name, description, location, website, Tuğrulcan Elmas, Rebekah Overdorf, Ömer Faruk Akgül, and Karl Aberer regulated. Users can create a page to post anonymously to some pro-opposition account, “Oy ve Ötesi” (English: Vote and Beyond). extent, but pages’ features are limited, e.g. they cannot befriend or He found that just prior to being called “Oy ve Hilesi,” the account interact with personal profiles. “had a sexy girl profile picture and was tweeting romantic quotes” as On the other hand, Twitter, the focus of this paper, allows for part of a scheme to artificially gain followers. Once it had acquired personal profiles, anonymous accounts using , and 40,000 followers, the account was sold on a webmaster forum for hobby accounts (e.g. parody, commentary, and fan accounts). These 200 Turkish Lira (∼$70 USD at that time). [31] different types of accounts are functionally the same; unlike Face- In this case, the account was initially created to grind followers book, there is no distinction between personal profiles and pages, and engagements — its only purpose was to become popular. Once so a personal profile can easily be transformed into a hobby orfan it gained 40,000 followers and appeared to be popular and therefore account. Twitter also allows users to manage multiple accounts someone with influence, the owner sold it to someone who wanted and even has an official how-to guide on the subject [2]. Twitter to influence social media users. The new owner purged the old does impose some restrictions: they do not allow impersonation or and irrelevant tweets, changed the name, the description and the platform manipulation (e.g. creating multiple accounts to inflate profile picture and, thus, shifted from a fake personal profile ofa popularity of a user or a narrative) [3]. woman to an anonymous account used to attack the opposition. Many repurposing accounts appear to change because they are Interestingly, the account kept its friends list (those accounts that it sold. Twitter explicitly prohibits account transfers or sales [35]. followed) at the time of the repurposing. However, it only followed However, there is no rule that explicitly regulates repurposing two accounts by the time of this analysis, but managed to retain of one’s own account. On Facebook however, misleading name 36,000 followers. This may be because the account needed to keep changes on pages are not allowed. Facebook states that "Name its followers that follow it back as part of a follow-back scheme. changes and merges must not result in a misleading or unintended In 2018 a popular security blogger and researcher @x0rz found connection and must not substantially change the Page’s subject that a number of fake accounts were attacking far-right French matter" [13]. It also notifies the followers or members of pages politicians in a coordinated manner. All of the accounts had the and groups when a name change occurs. Twitter has neither a same gmail address and stolen photos. Later, the accounts were misleading name change policy nor notification policy. repurposed: they purged all of their tweets, and claimed to be “some This contributions of this paper are as follows: sort of artificial neural network company or laboratory filled with fake content" [39]. It is not clear if the accounts were still employed • We study (§2) and define (§3) the phenomenon of account by the same adversary or changed hands. repurposing Twitter. We also stumbled on a similar instance of repurposing accounts • We establish a ground truth dataset of repurposed accounts “in the wild”, which we detected while following accounts that were using datasets published by Twitter Election Integrity (§4.2). posting content related to the gilet jaune (“yellow vest”) movement • We design and build a framework to detect misleading re- via the Twitter API. One account, which claimed to be a gilet jaune purposings of Twitter accounts in the wild (§4). supporter, was initially spreading polarizing content in support of • We detect and analyze the phenomenon of account repur- the protesters. It was then suddenly repurposed to be an account posing on two datasets: a set of suspended accounts which promoting Toy Story 4. The account has since been suspended. were involved in coordinated inauthentic behavior (§5.1) None of those accounts were popular, and they likely had little and popular accounts found via Internet Archive’s Twitter impact on their own. However, they can be deployed to manipulate Stream Grab dataset (§5.2). a specific discussion in a coordinated manner, which might make • We provide an analysis of the types of accounts that are sold them effective en masse. on an online Twitter account market (§5.3). Uren et al. [36] reported a group of low impact, coordinating • We provide a comprehensive dataset of 115,000 accounts accounts that were repurposed after being either bought or com- flagged by our framework as repurposed. Those accounts promised, likely by actors linked to the Chinese government. These were active in 2020 and repurposed after amassing 5,000 accounts were repurposed to promote government propaganda. followers (§4.1). The new account owners made a minimal effort to hide the fact that the accounts were repurposed, but the authors were able to 2 CASE STUDIES trace the previous owner of the account through unchanged profile To the best of our knowledge, this work is the first robust, aca- attributes, past tweets which were left undeleted, and a bot run demic study of repurposing behavior. However, sporadic cases of by the previous owner which was still reposting tweets by the re- malicious repurposings have been reported in the media, in purposed accounts. Some of the accounts were even tweeting in a posts, and in white papers. We draw on these cases to both define different language before the repurposing. misleading repurposing and to understand the characteristics of In yet another context, Grossman et al. [29] reported that sock- what a misleading repurposing looks like in order to learn which puppets that were posing as Qataris living in Saudi Arabia used a signals can be used to detect them. combination of “handle switching” (change of screen name) and The first report of repurposing that we found was in a 2015 Daily “wiping tweets” (purging). The authors suspected that the accounts Dot article by Efe Kerem Sözeri about social media manipulation had wiped their tweets because each account had a large gap be- related to the local Turkish elections in the Spring of 2014. In this tween the creation date and the date of the first tweet. The authors case, Sözeri studied an account by the name of “Oy ve Hilesi” (Eng- emphasized the high number of followers and claimed that these lish: Vote and Fraud). This account’s sole activity was attacking a accounts “increase their audiences with follow-back spam behavior,” Misleading Repurposing on Twitter which we refer to as grinding, and then are repurposed to mimic Table 1: The signals of repurposing based on our observa- public figures, similar to the case in [31]. tions from §2. These examples show that malicious users repurpose accounts in order to create popular or coordinated sock-puppets by repurposing Signal Attribute(s) Cases many accounts, which are either compromised, bought or were fake Profile Change account metadata All from the start but used for another purpose. In the remainder of this Unlinking screen name All Purging tweets & likes Sözeri, x0rz, Grossman paper, we use these examples as a basis for finding and studying Isolating friends & followers Sözeri repurposed accounts. Style Change tweets All Interest Change topics of interest All 3 DEFINING MISLEADING REPURPOSING Language Change tweets & settings. Uren et al. Source Change source Not Reported In order to capture cases like those in the previous section, we first Dormancy tweets Uren et al. must come up with a definition of the repurposing behavior. Each Grinding followers & engagements Sözeri, Grossman account changed its identity in order to fit its new purpose. The primary indicator of repurposing in these cases is the change of attributes from one snapshot in time to the next. Since accounts change often for legitimate reasons in ways that do not change the purpose of the account, defining which changes they tweet about different topics, and/or they change which ac- need to take place in order to call an account repurposed is challeng- counts they follow to different authorities on new topics. This may ing. Therefore, we narrowly define repurposing in this paper asa be so extreme that the language of the tweets is different between significant and misleading change to the account attributes where the original and repurposed account. the information about the former user is purged and new profile information is entered. We acknowledge that this definition may not encompass all instances of repurposing. For example, a nor- Pre-repurposing Activity. The final set of signals is related to the mal user’s dormant account becoming compromised and used for activity just before repurposing. Dormancy refers to a long period tweeting spam, but with unchanged attributes may be considered of inactivity before the spike in activity due to the repurposing. to be repurposed. Conversely, grinding entails maximizing followers and engagement. We study the cases in the previous section to understand how Grinding is a slang term used in video gaming to mean increasing the attributes change. We identify a series of signals, i.e., patterns a character’s level by performing small, repetitive actions in order that appear repeatedly in each of the use cases. These signals, sum- to progress through the game more easily [32]. In the context of marized in Table 1, can be used to detect repurposing. Twitter, grinding entails gaining followers and engagements (i.e. “leveling up” the account) to appear to be a popular account, which Profile Changes. The most prevalent signal is changes to the in tern garners more followers. account metadata and settings fields. These are changes on, among None of these signals alone can guarantee that accounts are others, name (i.e. Justin Bieber), description field, website, location, repurposed. An account can change its screen name due to a life and language. event (marriage or adapting a title) or just for personal taste. Users can purge their tweets, likes, friends, or even followers, making Erasing History. Another set of signals are related to the need a clean sweep if they are not satisfied with them anymore. The to erase historical records to hide past activity. We observe three tweeting style can change if the owner of the account changes. techniques that yield distinct signals: purging, unlinking, and iso- The reverse also holds, repurposing cannot guarantee any of these lating. Purging deletes all previous tweets and likes, removing any signals. An account might not change its screen name to unlink sign that an account has been repurposed. Unlinking, changes the itself if it is not linked to anyone (i.e. nobody replied them). An screen name (i.e. @justinbieber), so that replies and mentions of the account might not change all profile attributes and/or purge totally, original account made by other accounts cannot be found or associ- if they are not completely irrelevant to new identity it represents ated after repurposing: searching for the new screen name does not (e.g. retweets of cat videos). The style might stay the same because reveal any pre-repurposing activity (except the immutable user id, the account owner is the same even if the account is repurposed. which can only be accessed programmatically via the Twitter API). Finally, isolating removes friends (followees) to appear popular by reducing the ratio of followed to followee users. Isolating can also 4 IDENTIFYING REPURPOSED ACCOUNTS include removing unwanted followers such as fake accounts. With these signals in mind, we next develop a methodology to detect repurposed accounts. Our goal is not to develop a classifier Changes to Content and Structure of Tweets. A third set of signals that Twitter could use to detect malicious accounts but instead are artifacts of the account being used by another person or for to detect repurposed accounts in order to study their behavior. In another purpose. When another person is controlling the account, short, our methodology consists of i) collecting a historical dataset the source of the tweets (i.e. which device or platform the tweets are that contains past profile snapshots to reconstruct the users’ Twitter posted from, e.g. "Twitter for iPhone") often change. A new purpose history (§4.1) ii) hand-labelling this data to establish ground truth cases a shift in the subject or structure of the tweets themselves, (§4.2) and iii) building a classifier to detect suspect repurposed which results in a change of style. The accounts’ interests change; accounts (§4.3). This process is summarized in Figure 1. Tuğrulcan Elmas, Rebekah Overdorf, Ömer Faruk Akgül, and Karl Aberer

the abbreviated archive dataset to acquire all profile snapshots of the same user, if available. The Twitter datasets contained 83,481 unique user ids, 38,426 of which were found in the archive dataset. We found 17,220 screen name changes involving 8,370 accounts.

4.2 Ground Truth To build a ground truth for repurposed accounts, we use human annotation. We refrain from crowdsourcing the labeling task for a few reasons. First, the elections-integrity is only fully available to researchers given specific access by Twitter. Second, the archive Figure 1: Methodology Summary. We use past snapshots of dataset, although public, contains sensitive information, e.g. the an account and detect if a repurposing might have taken former screen names and names of accounts may be from legitimate place based on the ground truth we built. users whose accounts were compromised. Finally, expert annotation is more reliable than crowdsourcing for such a complex task. 4.1 Datasets For this labeling task, we use the screenname-changes dataset. For each instance, we select the last available snapshot of the pro- Archive Dataset. In order to study repurposing, we need a dataset file before the change and the first snapshot after the changeto of different snapshots of a user’s profile from different times (atthe compare. We examined the following semantically interpretable very least, one snapshot from before and one from after the repur- attributes: name, screen name, description, location, url, tweet lan- posing). To this end, we use public Twitter data that is archived by guage (most common), profile settings language, and tweet source the Internet Archive’s Twitter Stream [1]. This dataset contains a 1% (most common). We also examined the tweets themselves from sample of all tweets, including retweets and quotes, and includes a each snapshot. profile snapshot of the user who posted the tweet and, if applicable, To make the labeling decision, we asked ourselves the following the retweeted/quoted user. Because the sample includes retweets question: Did the account change in a way that makes it seem that and quotes, popular users with many retweets or quoted tweets the account is now owned by a different person or has the account will appear more often than accounts with fewer of these inter- rebranded itself dramatically? We allowed for the response: Yes, actions. As such, the dataset is biased towards active and popular No, and Can’t Decide. accounts, which is not necessarily a disadvantage in this study, as We considered the following cases to be “Yes": such accounts have the greatest potential to have an impact. Even though it contains only 1% of tweets, this dataset is massive. At the • user changing name, surname, , nationality, descrip- time of analysis (October 2020), the dataset dated from September tion, url, location (given they were not blank) in a way so 2011 to June 2020 and contained 446 million user ids. Of these, 294 that the user appears to be a different person/organization million (65.9%) occur more than once. We create an abbreviated • personal profile changing into a hobby or an organization version of this dataset by only considering accounts in which the account or vice-versa screen name changed, since we found this signal in every exam- • hobby/organization accounts rebranding itself to be a differ- ple of repurposing we studied. In general this is a rare action on ent hobby/organization altogether Twitter since screen name is a unique identifier and the way that • blank profile repurposed to be a personal profile later profiles are searched (i.e. “follow me on Twitter at @justinbieber") We considered the following cases to be “No": and shared (i.e. twitter.com/justinbieber). Additionally, prior work has shown that this attribute is fairly stable for most users [38]. We • user changing his name minimally (@john123 -> @john4ever) found that only 13.3% (59 million) of users in the archive dataset • changes due to a life event (i.e. @janedoe -> @janemare) change changed their scree name, confirming this finding. • due to a slight rebranding (@iShareMusic -> @iShareMetal- Elections-Integrity Datasets. To establish ground truth, we Music) hand-labeled a set of accounts. Because repurposing is a relatively • personal profiles repurposed to be politically engaged citi- rare event, finding positive samples in normal Twitter data isa zens without changing name or tweeting style (see the dis- challenge. Instead, we look to the datasets published by Twitter cussion) within the context of elections integrity [34]. By October 2020, We considered the following cases to be “Could not Decide": there were 35 datasets focused on 16 countries. Since repurposings often coincide with malicious behavior, this dataset contained many • user changes their name but no other identifying features, positive cases. Additionally, per our definition, these malicious cases e.g. the description field, so it is unclear whether the account are the focus of this study. Twitter provided us with the unhashed is repurposed or just changed the name / for versions of the datasets, which gave us access to the user ids of each some reason. (i.e. John Doe became Mohammad Lee, but his account. These datasets do not include historic screen names for any description field stayed as "Lawyer. Support all the rights and accounts, but do include a snapshot at the moment of suspension. freedom! Somalia is my Home." and his location remained Screenname-Changes Dataset. We combine these datasets to as Somalia.) establish a dataset of malicious accounts with screen name changes. • user rebranded dramatically, but the purpose of the account We search for the user ids from the elections-integrity dataset in remains the same (i.e. still a spam account) Misleading Repurposing on Twitter

• it’s not clear if the rebranding is related to a real world Table 2: The results of our experiments using tree based clas- change (e.g. a pub turning into a night club at the same sifiers. We see that a tree using only Rule 1 and Rule 2per- location) forms reliably while Random Forest slightly outperforms it. • users’ profile attributes are mostly blank • users’ profile is changed slightly, but the tweets change dra- Classifier C.V. Acc. Acc. Prec. Recall F1 matically and/or shows traces of compromisation. This case Rule 1 83.3 91.5 88.7 95 91.7 Rule 1 & Rule 2 88.3 91 94.5 87 90.6 may not be repurposing but qualifies with compromisation Decision Tree (Max Depth 2) 88.1 90 92.5 87 89.6 with minimal effort to hide as in [36]. Decision Tree (Max Depth 3) 88.4 89.5 89.9 89 89.8 Decision Tree (Max Depth 4) 88.3 90.5 91.7 89 90.3 We annotated all entries from the screenname-changes dataset Decision Tree (No Depth Limit) 84.3 89 93.3 84 88.4 that were in English or French. Even though the accounts were Random Forest 91.9 91.5 95.6 87 91 labeled as English/French, many had a lot of content in other lan- guages, e.g. Arabic, Persian, or Korean; we discarded such accounts. We annotated an additional 700 accounts in Turkish, but did not difficult in terms of textual analysis. Thus we also omit anytext expand this analysis further because many of the accounts had analysis. content had content that was technically in Turkish, but was un- To evaluate our classifier, we used a training set of 1,422 samples intelligible or the purpose was unclear, even to a native Turkish (910 negatives and 512 positives) and a test set of 200 samples speaker. that were annotated by both authors with balanced classes. We The first annotator labeled 1,876 profiles. They labeled 612as experimented with Decision Trees of different depth and leaf nodes positive, 1,010 as negative and 254 as unsure. The second annotator limits as well as more complex tree based classifiers. Our goal is not labeled 100 positive and 100 negative cases from the first author’s a classifier with the best recall and/or F1 score, but a classifier that annotation set. The inter annotator agreement was K = 0.80 which is interpretable and easy to deploy, as we’ll use it on the massive is substantial agreement. The agreement on negatives was higher archive dataset. Once we flag suspicious accounts in the wild using (93%) than positives (87%). The framework therefore will be very such a classifier, we can annotate more accounts and deploy amore effective in discarding negatives, which are much more frequent complex classification framework in the future. among all Twitter accounts. We found that regardless of their depth, all Decision Trees have these two rules at the first and second depth: Rule 1) 푁퐿퐷name > 4.3 Detection 0.721 and Rule 2) 푁퐿퐷description > 0.742. The Decision Tree using only these two rules achieves 88.3% cross validation accuracy, 91% With ground truth established, we next handcraft features to cap- accuracy, 94.5% precision, 87% recall and 90.6% F-score. As seen ture the signals we defined in Table 1. Using these features, we in Table 2, more complex trees only slightly increase the recall compare the last profile snapshot with old the screen name, pre- in exchange for precision. While Random Forest over-performs vious snaphot, and the first profile snapshot with the new screen this classifier’s cross validation accuracy, its contribution tothe name, next snapshot. For the string attributes, we compute Leven- precision is marginal. We opt not to use Random Forest but instead shtein distance between the string fields and normalize it by the use the Decision Tree of two rules for its simplicity. We later validate maximum of the two and name it NLD. The formula is as follows: the precision of our classifier in § 5.2. ( ) lev attr푈prev , attr푈next 푁퐿퐷 = (1) 5 ANALYSIS attr ( ( ) ( )) 푚푎푥 푙푒푛 푈prev , 푙푒푛 푈next We analyze the signals we defined in Table 1 on three sets of accounts: 1) suspended accounts published within the elections- where attr is the string attribute, 푈prev is the previous snapshot, integrity dataset 2) popular accounts that were still active between 푈next is the next snapshot and len is the length of the string attribute. For the integer attributes, we compute the difference and ratio January 2020 and June 2020 and 3) accounts sold on the web. Our between snapshots. For the rest (major tweeting language, source, analysis demonstrates that grinding is a common tactic among settings language, location) we use a boolean to represent if the the adversaries who repurpose accounts. We also share our case attribute has changed. We also account for soft changes, e.g. change specific findings and negative results. in proportion of usage of major tweeting language. Finally, we compute the time between the two snapshots to measure dormancy. 5.1 Elections Integrity Accounts We did not have access to the previous profile photos of the First, we analyze the accounts in the screenname-changes dataset, users and thus do not do any image analysis. Additionally, the which contains the user ids found in Twitter’s elections-integrity number of unique tweets per screen name is low, much lower dataset, which were reported to have participated in information than the number of profile snapshots per user as those snapshots operations that undermine public discourse. Using the decision are contained within the retweets of the same tweet, and tweets tree classifier from §4.3, we classify all accounts in the screenname- are short by nature. For only 1,069 accounts the major tweeting changes dataset. Each dataset is named after the suspended network language of both snapshots was English. Of those accounts, only tied to the country, the date they are suspended, and id in paren- 170 of them had at least 10 unique tweets per snapshot. The most theses to distinguish datasets with same country and date. Figure 2 frequent languages that did not change between snapshots were shows the results of the classifier for each dataset. We found 5983 Turkish (4,173), Arabic (3,856) and Indonesian (1,903) which are all repurposings involving 3573 accounts in total. Using the results of Tuğrulcan Elmas, Rebekah Overdorf, Ömer Faruk Akgül, and Karl Aberer this classification, we analyze different features of the repurposed accounts.

Figure 3: Histogram of dormancy in terms of years. The steeper decay of the left graph shows that repurposed ac- counts are more likely to lie dormant before they are re- purposed than those accounts that only have a screen name change but are not repurposed.

as repurposing. We find that repurposed accounts are more likely to purge tweets alongside a screen name change, although most keep the historical tweets. This may be to retain the engagements or the tweets in the old snapshot because they are not completely Figure 2: Number of repurposings (blue) and number of irrelevant to the new account and give the repurposed account repurposed accounts (orange) per dataset. Accounts from more legitimacy. Additionally, overactive accounts may not need China and Turkey have the most repurposings due to com- to purge tweets, as Twitter restricts collecting the timeline of any promisation and commercial activity. user to 3,200 tweets and end users are unlikely to scroll through many pages of tweets to reach the older tweets. Dormancy. First, we find that accounts that have long periods of inactivity are more likely to be repurposed. We suspect that an account that are dormant for a long period is more likely to be sold than an active account, either because it was a fake account that is no longer needed by its owner, or because it is more likely to be successfully compromised. We define dormancy as the amount of time between the last activity of an account and the moment of repurposing. Recall that we are only able acquire snapshots when there is activity on the account, and even then we only have snapshots for 1% of tweeting activity. So while we cannot compute the exact value of dormancy for each account, we can compute a proxy which show a trend. To this end, we compute dormancy’ as the amount of time between the last snapshot before and the first Figure 4: Histogram of purging in terms of logarithm of sta- snapshot after the screen name change. We compute dormancy’ tuses count after divided by statuses before. Repurposed ac- for both the repurposed and non-repurposed accounts and find counts are more likely to purge the tweets but they prefer to that dormant accounts tend to be more likely to be repurposed, as keep them in many cases. shown in Figure 3. Purging. We next evaluate the prevalence of purging. Recall that Repeated Repurposings. Although a rare behavior, some accounts purging is the action of deleting tweets from the profile while re- are repurposed multiple times. Figure 5 shows the distribution of purposing. We consider an account to have been purged if there the number of repurposings per account for each dataset. This be- are fewer tweets after the repurposing than before. Again, we can- havior is most common in the datasets originating from China and not compare the exact moment of repurposing, but we can get the Indonesia. We manually examined the accounts with the highest number of statuses (tweets and retweets) from the last snapshot number of repurposings from these two datasets and found many before and the first snapshot after the screen name change. We take of them to be fan or roleplaying accounts of various Korean singers. logarithm of the ratio of the number of in each snapshot, which The repurposings seem to be changing the singer that the account yields a negative value if the account has deleted any tweets. Fig- promotes. While not necessarily malicious, this change definitely ure 4 presents a histogram of the resulting values for repurposed fits our definition of repurposing. Due to our lack of knowledge on accounts and those with screen name changes but not classified the Korean music industry, we leave investigating those accounts Misleading Repurposing on Twitter to future work. We have not observed a pattern within Turkish Additional Findings. Briefly, we also find that repurposed ac- accounts that are repurposed multiple times. counts’ other profile attributes, e.g. profile image, location and homepage also change and that repurposed accounts are more likely than others to lose followers and friends. We do not generally observe a high number of repurposings that change the tweet lan- guage nor that change the language setting. Exceptionally, we find English speaking accounts changing to Chinese speaking accounts, apparently due to the account being compromised. We also find accounts that switch between Indonesian and English or Tagalog. The latter group also seem to be Korean music fan accounts.

Evidence of Commercial Activity. We also found three patterns of repurposing that suggests commercial activity. First, the Turkish dataset had the most repurposings, which is inline with Twitter’s de- scription of the dataset: the network was a mixture of compromised and fake accounts. From analysis of these accounts, we suspect that some accounts were sold, rather than compromised. We found 24 accounts that included satılık (“for sale”) explicitly in their descrip- tion field. That is, the accounts were repurposed after they were sold to fit the desired purpose of the buyer. In [36], researchers suggests that some Facebook accounts are Figure 5: The number of times same account is repurposed bulk-purchased from resellers and repurposed for Chinese propa- per dataset. Such behavior is more common among the ac- ganda. We build onto their work and suggest that many Twitter counts from China and Indonesia. accounts were in fact purchased as well. We found 225 accounts within China 08/2019 dataset (3) and 52 Twitter accounts in Turkey 05/2020 dataset with a homepage ending with .com that are later Grinding. Lastly, we observe that many repurposed accounts repurposed. We found that all the websites are down, but we could had a high number of followers relative to friends (“followees”). find that some pointed to legitimate websites by checking Internet In Figure 6 we present a heatscatter of the ratio of followers to Archive’s Time Machine. The websites were about various topics friends against the total number of friends (pre-repurposing). Note did not appear to follow a specific agenda. We suggest that those ac- that most accounts are clustered around 10,000 followers with a counts were bulk-purchased from sellers since the owners of those ratio close to or more than 1. This suggests that the accounts were websites closed them and did not need their accounts anymore. grinding in order to maximize followers. The high number of fol- Finally, we observed that 30 accounts from the Russia 08/2018 lowers alone while repurposing might also suggest that accounts dataset (named as "ira" by Twitter) were using a (likely malicious) with high number of followers are more likely to be target of those source to post tweets from called “masss post푥”(푥 ∈ {2, 3, 4, 5}). looking to compromise or purchase accounts. The accounts had names and screen names that were random strings. They originally reported their locations as cities in Eng- land (e.g. Coventry, Liverpool, London), but after repurposing, all changed their locations to “USA” or a specific US state. Similarly, all changed their names to names that are more prevalent in the US, (exceptionally, one adopted the name of a local news outlet). The accounts were created in various dates in 2014. They might be bulk created and/or purchased. Their location fields suggest that they might first aimed at influencing English Twitter users, then shift to US Twitter users.

5.2 Popular Accounts In the previous subsection, we find that repurposed accounts can be found in datasets of malicious accounts, but in this section we aim to find repurposed accounts in the wild. We analyze the prevalence of such accounts, determine which countries they originate from, Figure 6: Repurposing targets account with high number and study their topical interests to uncover whether or not they of followers. Such accounts also have friend follower ratio engage in critical discussions. We further corroborate our findings close 1, which might indicate that they were grinding before from the previous section regarding the strategies of repurposing. the repurposing. Detection. To detect repurposings in the wild, we turn to ac- counts in the archive dataset. We focus on popular Tuğrulcan Elmas, Rebekah Overdorf, Ömer Faruk Akgül, and Karl Aberer

Table 3: Validation of our framework using datasets from scheme as described in §4.2, except that we did not examine the English and Turkish tweeting users. Most followers are tweets themselves from each snapshot. Two authors independently those who had the highest number of followers during the annotated all the repurposing instances and discussed the cases repurposing, while the rest are random samples from users in which they did not agree and either came to a consensus or who repurposed after reaching 5000 followers and repur- assigned a final label of "Disagree." The results of this validation posed before reaching 5000 followers. While users with high are listed in Table 3. number of followers tend to have complete profiles that self- We found that the accounts that had more complete profiles (i.e. state their purpose, other users either do not have purpose the name and description fields with information that states what or do not self-state them, making it hard to infer if there is the account is about) generally have higher follower counts (thus, repurposing (Unkn.). On few users, we cannot agree on the an audience), and were therefore easier to annotate. However, for annotation (Dis.), or if the rebranding is dramatical (Rebr.) the other lists, ascertaining the purpose of the account both before Precision is computed as true positives (TP) divided by true and after the repurposing was a difficult task. This may be because positives plus false positives (FP), discarding the rest. unlike accounts involved in information operations or those with a large audience, such low profile accounts do not follow a cer- Sample TP FP Dis. Rebr. Unkn. Prec. tain agenda and do not self-state their purpose. Many use obvious Most Followers (en) 65 6 0 12 17 91.5% pseudonyms and often change between pseudonyms, which en- Most Followers (tr) 81 3 3 2 11 96.4% courages changes to the screen name, name, and description fields Followers Before >5000 (en) 43 15 10 0 31 74.1% Followers Before >5000 (tr) 62 12 3 0 23 83.7% even if the purpose does not change. Followers Before <5000 (en) 25 24 8 3 40 51% Additionally, the annotators found it difficult to label accounts Followers Before <5000 (tr) 48 23 5 0 24 67.6% in which one of the snapshots seemed to be a personal profile, but the other snapshot’s purpose was difficult to intuit, i.e. if it wasa pseudonym of the old personal profile or if it had morphed into both because the entire dataset is very large and because we aim a hobby account. The annotators chose to leave this case as “un- to find users with the potential to impact to public discussion. As known” (unkn). The annotators also could not assess if a rebranding such, we limit our study to accounts which were still active between was slight or dramatical in certain cases, so these are also reported January 2020 and June 2020 and had more than 5,000 followers. We in their own category (rebr.). choose 5,000 followers to be consistent with the threshold used The precision on accounts that were repurposed before reaching by Twitter use to reveal unhashed versions of the accounts within 5,000 is quite low, so we opt out from analyzing those accounts. elections-integrity dataset. However, the precision for the other lists is sufficiently high to We found 1.57 million active users with at least 5,000 followers provide reliable results, so for the remainder of this subsection, we between January 2020 and June 2020. Using the classifier we in- analyze the repurposings found by the classifier for accounts with troduced in §4.3, we found 396,863 repurposings involving 237,004 more than 5,000 followers before the repurposing. We refer to these accounts (15%) (note that some accounts were repurposed multiple accounts as “popular.” times). Only 115,980 accounts of those accounts were repurposed after reaching 5,000 followers, which makes up 7.3% of the 1.57 mil- Location. The most frequent location fields of repurposed popu- lion active popular accounts. Of these, 78.9% were still active on the lar accounts is listed in 4. The table is sorted by the reported loca- platform as of October 2020, while the rest were either suspended tions of the accounts after they got repurposed. However, sorting or deleted. In 98.7% of these repurposings, the new profile appears by the frequency before accounts are repurposed does not signifi- to have retained many of the followers of the old profile, since the cantly change the results. Repurposing appears to be more popular accounts still have more than 5,000 followers. in Brazil, Turkey, The United States, Mexico, Indonesia, India and Nigeria and/or target the population in these countries more often. Validation of the results. Due to the large difference between the base rate of the data that the classifier was trained on (malicious Grinding. Next, we look at how many followers these repurposed accounts) and the base rate of data in the wild (which likely has accounts manage to gain and maintain. First, we find that the most fewer cases of repurposing), we validate a subset of the results popular in the description fields of the popular repurposed manually to confirm that the classifier is not biased by this fact. accounts were related to “follow back” (i.e. stating that if you follow Evaluating the recall of the classifier on real world data is chal- their account they will follow yours), before as can be seen in lenging due to the vast number of negatives, since the event of Table 5. Although still prevalent, the number of such hashtags repurposing is relatively rare. However, we can validate the preci- drops significantly after the accounts are repurposed, suggesting sion of the classifier using the positive labeled samples. We chose that some of the accounts fulfilled its mission of grinding. This alone 100 users with English tweets and settings and 100 users with Turk- suggests that many accounts are aiming to quickly gain followers. ish tweets and settings from each of the following lists: 1) the list Additionally, Figure 7 shows that most of the accounts that were of users with most followers before the repurposing, 2) a random repurposed have a follower to friend ratio close to 0. Recall that all sample of users who had more than 5,000 followers at the time of accounts had at least 5,000 followers at that time, which means that the repurposing, and 3) a random sample of users who had less the accounts have a large number of friends. Similar to the case of than 5,000 followers at the time of the repurposing but managed accounts in the elections-integrity dataset, this suggests that that to amass 5,000 followers afterwards. We used the same annotation they became popular through grinding. Misleading Repurposing on Twitter

Table 4: The most frequent reported locations among the re- Table 5: The most frequent hashtags found in the descrip- purposed fake public figures are from Saudi Arabia, Brazil, tion field of the accounts before they got repurposed are Turkey, USA, Mexico, Indonesia and India. The table is those related to grinding with a few exceptions. The fre- sorted by frequency after the repurposing. quency of such hashtags drops significantly after the repur- posing. Location Freq. After Freq. Before Saudi Arabia (in Arabic) 1402 983 Freq. Freq. Meaning Brasil 1361 1289 Before After İstanbul, Türkiye 1303 1111 #1 447 410 — Türkiye 1216 998 #followback 250 105 — United States 960 792 #teamfollowback 226 79 — Rio de Janeiro, Brasil 645 596 #follow 107 63 — Worldwide 624 623 #mgmw 107 49 More Gain With Volume México 613 538 #rt 104 84 Retweet #maga 103 84 Make America Great Again Indonesia 605 524 #takipedenitakipederim 100 41 I follow back (Turkish) São Paulo, Brasil 556 495 #相互フォロー 81 6 I follow back (Japanese) India 554 500 #siguemeytesigo 71 44 I follow back (Spanish) Los Angeles, CA 546 451 #f4f 70 22 Follow for follow Lagos, Nigeria 469 349 Argentina 454 439 Nigeria 441 342

Figure 8: The histogram of repurposings by the ratio of friend count after to the friend count before. The repur- posed accounts are more likely to unfollow their friends al- though it’s not the most common behavior among them.

Figure 7: The histogram of repurposings by the follower rare. Only 296 accounts lost 90% of their followers. Thus, we omit friend ratio (log). Most of the users have follower friend ra- analysis of the isolation by losing followers. tio close 1 even if they have more than 5000 followers, sug- gesting that they were grinding for followers. Topics of Interest: As a first step to assess risk these accounts im- pose, we analyze their topics of interest. For simplicity, we consider a hashtag to be a topic. We collected most recent 200 tweets from all popular repurposed accounts and tallied the number of accounts Isolation. Some accounts unfollow their friends to increase fol- that use each hashtag in these tweets. Table 6 shows these results. lower / friend ratio so that they will appear more popular. To eval- We observe many political and health related hashtags, which may uate this, we compute the logarithm of number of friends after the indicate that these popular accounts are using these hashtags with repurposing divided by the number of friends before repurposing. malicious intentions (i.e. to artificially sway public opinion). Due Figure 8 shows the results. 13,737 repurposing involved purging to their size, they may have impact and thus need to be addressed the friends isolating the account, which makes it also a rare behav- immediately. We leave a more robust study of fake public figures’ ior. This might be because the accounts still grind followers even topical interests and impact to the future work. after the repurposing. Alternatively, repurposing could be another strategy to grind for more followers. Another reason could be, as Negative results: Only 11.8% (22,393) of repurposings appear to in the case of Oy ve Hilesi that we introduced in § 5.2, the accounts have taken place on accounts which were dormant for a year or might isolate later or not isolate at all to keep the followers that more, which is rare but not negligible. While this result does appear follow the account in exchange for follow-backs. Meanwhile, isola- to contradict from the results in the previous subsection, note that in tion by removing followers or getting unfollowed seem to be very this subsection we are only studying accounts with 5,000 followers Tuğrulcan Elmas, Rebekah Overdorf, Ömer Faruk Akgül, and Karl Aberer

Table 6: The most popular hashtags among the repurposed (in 2013). Our analysis focuses on popular accounts that are sold in popular accounts. Besides hashtags that are popular in gen- order to be used as is or to be repurposed but retain its followers. eral (e.g. #NewProfilePic) there are many politics and health We analyzed all accounts on sale on three Turkish websites related hashtags might have serious impact if those repur- which contain public postings of Twitter accounts among other posed popular accounts have malicious intentions (i.e. arti- platforms: ilansat.com (465 accounts), hesapsatis.com (206 accounts) ficially sway public opinion). and hesapsat.net (26 accounts) in August 2020. Each account that was posted for sale consisted of the following relevant fields: seller’s Hashtag Explanation # of Authors Freq. name, screen name, number of followers, number of friends, and sell- NewProfilePic 3994 6215 ers’ remarks. We could only find 149 accounts on Twitter through COVID19 Pandemic 3824 11097 screen names mentioned in the posting. We suspect that the rest _ of the accounts were already sold and their screen names have al- Q mÌ'@ hAJ.“ "Good morning" 2496 7076 "Corona" 2341 9576 ready changed. This dataset is too small to perform any meaningful AKðPñ» quantitative analysis, so we present observations and statistics, but coronavirus Pandemic 2064 5093 leave a more large-scale study with more samples to future work.  " Saudi" 2026 6688 éK Xñª‚Ë@ Account Promotion. We observe that many sellers promote their BTS K-pop Band 1949 11199 accounts as “organic,” communicating that the the accounts’ fol- BlackLivesMatter Protests 1927 3138 lowers are not bots, while a few indicate that the followers are bots IK ñºË@ "Kuwait" 1646 9156 or that the account itself is a bot. Some claim that their accounts do JIMIN BTS vocal 1631 5499 not get their engagements through retweet groups (i.e. groups of ac- count retweeting each other constantly to maximize engagements) while others claim that they do utilize such groups and further claim that even though other sellers advertise otherwise, all accounts on or more, so it is unlikely that an account owner would leave fallow the site use such groups. Email and phone verification also seem such accounts. Similarly the purging of old tweets also seems to be to be important, as sellers advertise these features. Although the not very prevalent among these popular accounts, with only 12.3% number of followers appears to make the account more valuable, (23,355) of accounts appearing to purge at least 90% of their tweets. it is not the only factor; the number of friends, the quality of the We also found that the change of the language user setting is followers, the number of engagements, and the content are used in quite rare among these popular accounts, and when it does occur combination to justify the quality and the price of the account. it is generally a change between English and one of Indonesian, Surprisingly, we found 3 accounts that were promoted as being Arabic, Portuguese, and Spanish. The most prevalent language user verified (i.e. had blue check marks). One of these accounts was setting change is from English to Arabic which occurred just 576 a hobby account sharing Islamic quotes, and it is unclear how it times. We also did not observe a significant number of changes became verified. The seller admits that he grinded the account, in tweeting language. In the previous subsection, we found that which had 2.1 million followers and 1 million friends. Another ac- language changes were due to bulk purchasing or due to many count appears to be a person who grinded his own account and accounts becoming compromised together. Since popular accounts now has 1.6 million followers and 1.4 million friends. This user are more valuable accounts, it might be difficult for adversaries from does not appear to be a celebrity, as he has no presence in the foreign countries to compromise them. Additionally, adversaries news except for a few embedding his tweets as examples may be only purchasing popular accounts from local markets. of social media reactions to an event. The user also has a weak We found that repeated repurposing is rare among popular ac- presence in (51k followers), so the person appears to counts. 69.3% (80,417) of the accounts are repurposed only once have managed to convince Twitter that he deserves to be verified and 17.3% (20,149) are repurposed twice. The rest, 13.2% of the and is now selling his valuable account. The third account appears accounts are repurposed three times or more. While this finding to be the compromised account of a singer, Luis Mas, who had a is not negligible, we believe that multiple repurposing might be verified profile. His official website still pointed to the nowcompro- more prevalent among non-popular accounts that cannot be easily mised account (https://luismasmusic.com, which recently became disposable. We leave analysis of such accounts to future work. inaccessible) and mentions to his screen by his fans can still be observed. The account now belongs to a Turkish person claiming 5.3 A Market for Accounts to be a doctor. Twitter does not allow verified users to change their screen names. Hence, our framework does not detect repurposings The cases we found in which popular accounts were later repur- of verified accounts when we only consider users who changed posed raise a number of questions that we cannot answer by looking their screen name. We leave analysis of abuse of verified profiles to only at the accounts. Here, we look to three online markets from future work. Turkey that sell account credentials to learn how these accounts are promoted and to find evidence of people grinding accounts to later Grinding. Of the 149 accounts that were still not sold at the sell. A previous study covered a detailed analysis of similar account time of analysis, we assessed 71 of them as grinding/grinded for markets [33], however, the accounts that they inspected differ from follows, either 1) because the seller openly admits it in the posting, those that we are interested in as they were generally low quality 2) because the profile openly asks for follow backs in the account’s accounts that were created and sold in bulk for 2-10¢per account tweets or description fields, or 3) because the friend:follower ratio Misleading Repurposing on Twitter was very close to 1:1. This further illustrates that grinding is a was followed by many famous journalists and economists and had common behavior among Twitter account sellers. We found 21 obtained 62,000 followers. postings (14%) in which the seller had a masculine name, but the Because of the success of accounts like the one just described, account being sold was a (likely fake) profile of a woman with a repurposing incentivizes attackers to compromise accounts by creat- different name. This would appear to be grinding via catphishing. ing a market demand for popular accounts that can be repurposed. We found 14 sellers that sold at least 5 accounts accounting for Repurposed accounts that are malicious are clearly a problem 18% of all postings (128 accounts). These sellers might be grinding on Twitter, and as they continue to successfully interfere with po- many accounts and selling them as a way to earn income. Some litical discourse the problem will likely only grow. Because there sellers specifically indicate in their postings that they have many are legitimate reasons to change profile attributes, this problem is accounts and that clients can message them if they are interested inherently difficult to address, and each solution to this problem in buying in bulk, where prices go as low as 4TL (0.5$) per account has drawbacks. One approach is to adopt an approach used by Face- (similar to the case found in [33]). book: notifying the followers of any screen name change. However, this approach has a disproportionate impact on users who change their names in real life, primarily married/divorced women and 6 DISCUSSION trans* users, but others as well. Twitter can also adopt a policy Security Implications and Challenges. Repurposing has security that disallows unauthorized screen names changes for accounts implications for Twitter and its users. First, in practice, repurposed with many followers (as is already the rule with verified accounts). accounts are new accounts that steal the followers of the accounts Additionally, Twitter can update its policy on misleading name that they repurpose. These followers lend credibility to the account changes. in the eyes of users and Twitter itself. With respect to the former, an account with few followers and an old account creation date is Not Quite Repurposed Accounts. Repurposing is easy to detect likely to appear more credible to users than a newly created account and therefore likely to be avoided by adversaries who want to with few followers [23]. With respect to the later, Twitter imposes a avoid detection. However, we observed a stealthy strategy used quality filter to filter out low-quality content and improve theuser among accounts suspended in October 2018 originating from Rus- experience. Although this filter is black-box, we can presume that sia. We identified 43 accounts that had initially blank or politically the filter considers account age and the number of engagements, neutral description fields but then later adopted description fields because these features have proven useful for bot and spammer de- that explicate their political stance (e.g. pro-Israeli, patriot, con- tection [8, 15]. Repurposing evades both by stealing history from an servative, #Blacklivesmatter) alongside a corresponding change existing account. Twitter also works to detect and suspend accounts in demographic attributes (e.g. Christian, Black). All 43 accounts that participate in coordinated activity. A malicious user looking changed their screen names, but they appear to keep the same to engage many accounts in a coordinated manner may opt to pur- identity (e.g. Jenna the Traveler became Jenna Abrams) and so we chase accounts from different sources and repurpose them together, did not consider them to have been repurposed per our definition. since this decreases the likelihood that the set of accounts were Instead of repurposing, the account appears to add a new purpose. used in a coordinated manner in the past and therefore increasing Their current purpose of showing off their personality does not the time before Twitter suspends the accounts. change and coexists with the new purpose. These accounts might While inflated follower counts may give some credibility to fake have grinded (non-aggressively) politically neutral personal ac- accounts, the highest credibility impact comes from the tweets counts, then repurposed them by overloading them with a political that are retweeted by someone that the user knows, making it agenda without purging totally and/or repurposing. Such accounts very important to build a genuine follower base [23]. For example, can be repurposed over and over for any topic within the same @TEN_GOP was a fake account with 130,000 followers employed by context, i.e. a regular American citizen account can be repurposed Russians which claimed to be the “Unofficial page of the Tennessee by overloading it with pro-republican content while still tweeting Republicans" while spreading anti-Clinton messaging during the regular non-political tweets. Then the pro-republican tweets can 2016 election. Influential users interacted with the account, includ- be deleted and the account can be partially repurposed. We leave a ing Donald Trump who thanked them [26]. deeper analysis of such accounts to the future work. Repurposing makes it easier for a malicious account to obtain followers and engagements from influential people and/or organiza- 7 RELATED WORK tions, because they can be obtained before the repurposing through Twitter Attribute Changes. Previous work has analyzed how a non-malicious account. For example, during the 2014 elections in users change their profile attributes, primarily to uncover when Turkey, a “fan account” posted comical tweets about president Er- and under which circumstances these changes are made [25, 30]. dogan in support for his rival and attracted 2,500 followers, mostly These studies find that screen name and location are the most sta- those with a pro-opposition stance. It was then repurposed twice ble profile attributes [38] and that users change their attributes for to be the account of two different new, short-lived political parties many reasons, including maintaining multiple accounts, changing in Turkey. The account was repurposed a third time in 2015, claim- user identifiability (anonymizing or deanonymizing oneself) and ing to be an economist living in Canada having degrees from a username squatting [17]. Regarding screen names in particular, prestigious university with a profile photo stolen from the web[5]. Mariconti et al. found that adversaries hijack the screen names This account, now with 7,500 followers, was critical of the Turkish of popular users who recently changed their screen name in or- government and reached the news multiple times [6, 7]. By 2020, it der to gain visibility, often with malicious intent. Zannettou et al. Tuğrulcan Elmas, Rebekah Overdorf, Ömer Faruk Akgül, and Karl Aberer found that 9% of 2,700 accounts operated by Russian trolls changed [4] [n.d.]. What names are allowed on Facebook? ([n. d.]). https://www.facebook. their screen name [42], but do not go into detail about if these ac- com/help/112146705538576 [5] animus. 2015. sinan erdil. Accessed on 2020-10-16. Ekşi Sözlük. counts are repurposed. In this paper, we do not look to study profile [6] Anonymous. 2015. Başkanlık Tweeti Rekor Kırdı. (2015). https://odatv4.com/ changes themselves, as in prior work, but study profile changes as baskanlik-tweeti-rekor-kirdi-0405151200.html [7] Anonymous. 2020. Karantinada kanal ihalesi. (2020). http://platform24.org/ signals of a phenomenon on Twitter. medya-izleme/4282/-karantinada-kanal-ihalesi [8] Fabricio Benevenuto, Gabriel Magno, Tiago Rodrigues, and Virgilio Almeida. Accounts Changing Ownership. We find that many accounts may 2010. Detecting spammers on twitter. In Collaboration, electronic messaging, become repurposed after they change hands and account owner- anti-abuse and spam conference (CEAS), Vol. 6. 12. ship has shifted. Accounts can change ownership either from being [9] Nikan Chavoshi, Hossein Hamooni, and Abdullah Mueen. 2016. DeBot: Twitter Bot Detection via Warped Correlation.. In ICDM. 817–822. compromised or on mutual agreement between the previous and [10] Zi Chu, Indra Widjaja, and Haining Wang. 2012. Detecting social spam campaigns current account owners, often as a result of commerce. Compro- on twitter. In International Conference on Applied Cryptography and Network Security. mised accounts are well studied in the literature, including what [11] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and the compromised accounts are used for [37] and user reactions to Maurizio Tesconi. 2015. Fame for sale: Efficient detection of fake Twitter followers. their accounts becoming compromised [28, 41]. Other work ana- Decision Support Systems (2015). [12] Tuğrulcan Elmas, Rebekah Overdorf, Ahmed Furkan Özkalay, and Karl Aberer. lyzes how accounts change hands by being bought and sold on 2020. The Power of Deletions: Ephemeral Astroturfing Attacks on . underground markets, as we did in §5.3. Thomas et al. studied how arXiv preprint arXiv:1910.07783 (2020). Twitter accounts were bought and sold on illicit forums. Interest- [13] Facebook. [n.d.]. Pages, Groups and Events Policies. Accessed on 2020-10-14. ([n. d.]). ingly, albeit with few data points to corroborate, they found that in [14] Jennifer Golbeck. 2019. Benford’s Law can detect malicious social bots. First 2013 bulk accounts could be bought for between 2¢and 10¢, which is Monday (2019). [15] Wafa Herzallah, Hossam Faris, and Omar Adwan. 2018. Feature engineering for much cheaper than the Turkish accounts that we studied being sold detecting spammers on Twitter: Modelling and analysis. Journal of Information for 50¢per account. Other stolen and sold credentials have also been Science 44, 2 (2018), 230–247. looked at, including Steam accounts [24] and Gmail accounts [27]. [16] Amanda Holpuch. 2020. Facebook still suspending Native Ameri- cans over ’real name’ policy | Technology | The Guardian. (2020). Platform Manipulation. Our analysis in this paper includes ac- https://www.theguardian.com/technology/2015/feb/16/facebook-real-name- policy-suspends-native-americans counts that were suspended due to platform manipulation and un- [17] Paridhi Jain and Ponnurangam Kumaraguru. 2016. On the dynamics of username dermining democracy [34]. Although past research indicates that changing behavior on twitter. In Proceedings of the 3rd IKDD Conference on Data Science, 2016. 1–6. these accounts may have been purchased or compromised and then [18] Franz Kafka. 2006. Die verwandlung. Routledge. repurposed, they fall short of a robust analysis of the prevalence [19] Jacob Kastrenakes. 2018. German court says Facebook’s real name policy is due to lack of ground truth [29, 36]. Past research primarily focused illegal. The Verge (2018). [20] Franziska B Keller, David Schoch, Sebastian Stier, and JungHwan Yang. 2019. on accounts with automated behaviors, e.g. spammers [10, 40, 43], Political Astroturfing on Twitter: How to Coordinate a Disinformation Campaign. retweet bots [14], fake followers [11], and astroturfing bots [12]. Political Communication (2019), 1–25. Research on identifying non-automated sock-puppets is currently [21] Karyne Levy. 2014. Facebook Apologizes For Real Name Policy - Business Insider. (2014). https://www.businessinsider.com/facebook-apologizes-for-real-name- inchoate due to a lack of ground truth. Much of current research policy-2014-10?r=US&IR=T relies either on datasets annotated by Twitter (e.g. [34]) or on hu- [22] Suman Kalyan Maity, Aishik Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2017. Detection of sockpuppets in social media. In Companion of the 2017 ACM man annotation [22]. Our work tries to break out of this pattern. Conference on Computer Supported Cooperative Work and Social Computing. 243– Timely detection of fake accounts through classification of repur- 246. posing behavior can also lead to early detection of accounts that [23] Meredith Ringel Morris, Scott Counts, Asta Roseway, Aaron Hoff, and Julia Schwarz. 2012. Tweeting is believing? Understanding microblog credibility participate in social media manipulation and disinformation, since perceptions. In Proceedings of the ACM 2012 conference on computer supported fake accounts often participate in coordinated attacks [9, 20]. cooperative work. 441–450. [24] Marti Motoyama, Damon McCoy, Kirill Levchenko, Stefan Savage, and Geoffrey M Voelker. 2011. An analysis of underground forums. In Proceedings of the 2011 8 ETHICS AND REPRODUCIBILITY ACM SIGCOMM conference on Internet measurement conference. 71–80. [25] Kumari Neha, Shashank Srikanth, Sonali Singhal, Shwetanshu Singh, Arun Bal- In this study, we only use data provided by Twitter and the public aji Buduru, and Ponnurangam Kumaraguru. 2019. Is change the only con- data provided by Internet Archive, both of which have been ana- stant? Profile change perspective on# LokSabhaElections2019. arXiv preprint lyzed extensively by previous work. We do not use nor store any arXiv:1909.10012 (2019). [26] Luke O’Brien. 2017. Twitter Ignored This Russia-Controlled Account During other data. Due to the Twitter TOS, we are not able to share the The Election. Team Trump Did Not. Huffpost (1 2017). repurposed accounts’ past data. However, we will make available [27] Jeremiah Onaolapo, Enrico Mariconti, and Gianluca Stringhini. 2016. What the list of user ids, with no , of the popular accounts happens after you are pwnd: Understanding the use of leaked webmail credentials in the wild. In Proceedings of the 2016 Internet Measurement Conference. 65–79. that we identify for the sake of reproducibility. Recall that our clas- [28] Richard Shay, Iulia Ion, Robert W Reeder, and Sunny Consolvo. 2014. " My sifier does not achieve 100% precision, so not all of the userids religious aunt asked why i was trying to sell her viagra" experiences with account hijacking. In Proceedings of the SIGCHI Conference on Human Factors in Computing correspond to repurposed or malicious accounts. The study can be Systems. 2657–2666. easily reproduced by the methodology we define here. [29] Khadija H. Shelby Grossman and Emily Ross. 2020. Royal Sockpuppets and Handle Switching: How a Saudi Arabia-Linked Twitter Network Stoked Rumors of a Coup in Qatar. Stanford Internet Observatory (10 2020). REFERENCES [30] Jinsei Shima, Mitsuo Yoshida, and Kyoji Umemura. 2017. When do users change [1] [n.d.]. Archive Team: The Twitter Stream Grab. https://archive.org/details/ their profile information on twitter?. In 2017 IEEE International Conference on Big twitterstream Data (Big Data). IEEE, 3119–3122. [2] [n.d.]. How to manage multiple accounts. ([n. d.]). https://help.twitter.com/en/ [31] Efe Kerem Sozeri. [n.d.]. How pro-government trolls are using a sexy Twitter bot managing-your-account/managing-multiple-twitter-accounts to sway Turkey’s election. The Daily Dot ([n. d.]). https://www.dailydot.com/ [3] [n.d.]. Platform manipulation and spam policy. ([n. d.]). https://help.twitter. layer8/turkey-election-twitter-troll-vote-and-beyond-vote-and-fraud/ com/en/rules-and-policies/platform-manipulation [32] Techopedia. [n.d.]. Grinding. Accessed on 2020-10-14. ([n. d.]). Misleading Repurposing on Twitter

[33] Kurt Thomas, Damon McCoy, Chris Grier, Alek Kolcz, and Vern Paxson. 2013. AAAI Conference on Web and Social Media. Trafficking fraudulent accounts: The role of the underground market in Twitter [39] x0rz. [n.d.]. Uncovering Foreign Trolls (Trying) To Influence French Elections spam and abuse. In 22nd {USENIX} Security Symposium ({USENIX} Security 13). on Twitter. Medium ([n. d.]). https://blog.0day.rocks/uncovering-foreign-trolls- 195–210. trying-to-influence-french-elections-on-twitter-a78a8c12953 [34] Twitter Inc. [n.d.]. Elections Integrity. Accessed on 2020-10-14. ([n. d.]). [40] Sarita Yardi, Daniel Romero, Grant Schoenebeck, et al. 2010. Detecting spam in a [35] Twitter Inc. 2020. Platform manipulation and spam policy. Accessed on 2020-10- twitter network. First Monday (2010). 14. (2020). Twitter Help Center. [41] Eva Zangerle and Günther Specht. 2014. " Sorry, I was hacked" a classification of [36] Tom Uren, Elise Thomas, and Jacob Wallis. 2019. Tweeting Through the Great compromised twitter accounts. In Proceedings of the 29th annual acm symposium Firewall: Preliminary Analysis of PRC-linked Information Operations on the on applied computing. 587–593. Hong Kong Protest. Australia Strategic Policy Institute: International Cyber Policy [42] Savvas Zannettou, Tristan Caulfield, Emiliano De Cristofaro, Michael Sirivianos, Center, Barton (2019). Gianluca Stringhini, and Jeremy Blackburn. 2019. Disinformation warfare: Un- [37] Courtland VanDam, Jiliang Tang, and Pang- Tan. 2017. Understanding derstanding state-sponsored trolls on Twitter and their influence on the web. In compromised accounts on twitter. In Proceedings of the International Conference Companion Proceedings of The 2019 World Wide Web Conference. 218–226. on Web Intelligence. 737–744. [43] Jinxue Zhang, Rui Zhang, Yanchao Zhang, and Guanhua Yan. 2018. The Rise [38] Ryan Wesslen, Sagar Nandu, Omar Eltayeby, Tiffany Gallicano, Sara Levens, of Social Botnets: Attacks and Countermeasures. IEEE Trans. Dependable Sec. Min Jiang, and Samira Shaikh. 2018. Bumper Stickers on the Twitter Highway: Comput. (2018). Analyzing the Speed and Substance of Profile Changes. In Twelfth International