Internet Research Agency Built Its Disinformation Campaign

The Political Quarterly

How Russia’s Internet Research Agency Built its Disinformation Campaign

ANDREW DAWSON AND MARTIN INNES

Abstract In this article we analyse features of the information influence operations run by the St. Petersburg based Internet Research Agency, targeted at Europe. Informed by publicly available ‘open source’ data, the analysis delineates three key tactics that underpinned their disinformation campaign: account buying; ‘follower fishing’; and narrative switching. Both individually and collectively these were designed to build the reach, impact and influence of the ideologically loaded messages that social media account operators authored and ampli- fied. The particular value of the analysis is that whilst a lot of recent public and political attention has focussed upon Kremlin backed disinformation in respect of the 2016 United States presidential election, far less work has addressed their European activities. Keywords: disinformation, Russia, Kremlin influence, Internet Research Agency

WITH HINDSIGHT, the sheer scale and scope of influence European politics. For understand- activities performed by the St. Petersburg able reasons, much of the public conversa- based Internet Research Agency (IRA) and tion to-date has focussed upon the American allied Kremlin units attempting to influence situation. But there is significant evidence of and interfere with the 2016 US presidential similar influence and interference strategies election process, suggests much more could being operationalised in Europe. Under- have been done to interdict their efforts. For standing this arena is especially timely, as described by two recent studies commis- given that the European parliamentary elec- sioned by the US Senate Permanent Select tions are scheduled for May 2019. Committee on Intelligence—from Oxford Second, although the Senate commissioned University and New Knowledge—and the studies did a commendable job in document- indictments filed by Special Counsel Mueller, ing and describing the volume and variety the Russian backed activities were extensive, of Kremlin backed influence campaigning, diverse and pervaded all major social media the amount of material involved means there platforms. That said, doubts remain about is more to do in terms of distilling the IRA’s precisely what impact they accomplished. In disinformation playbook. Detailed ‘digital their recent book Network Propaganda,theHar- forensic’ investigative methods are needed to vard academics Yochai Benkler, Robert Faris craft an evidence-based understanding of and Hal Roberts have cautioned that digital how IRA operators built their audience and influencing activity does not easily translate influence. This is on the grounds that identi- into measurable behaviour change.1 Thus we fying their key tactics and techniques may need to be wary of over-attributing any causal enable similar disinformation campaigns to effects to even the most sophisticated disinfor- be detected in other contexts. mation campaign. Finally, we discuss some of the challenges Set against this backdrop, given that digi- with attributing authorship and impact to dis- tal influence engineering is now being information communications. This reflects undertaken regularly and routinely, this arti- how, not only have the US and its allies cle highlights three issues. First, the impor- learned about some of the methods used to tance of developing a better understanding seed and amplify false and misleading infor- of how Russian state assets have sought to mation online, so too have those authoring use disinformation and misinformation to such messages. Contemporary efforts at

© 2019 The Authors. The Political Quarterly published by John Wiley & Sons Ltd on behalf of Political Quarterly Publishing Co (PQPC). 1 This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. communicating disinforming narratives are groups on Facebook twice daily, with a target increasingly sophisticated, as those involved of at least 500 subscribers by the end of the are learning ‘what works’ in making messages first month. On Twitter, operators were gener- more persuasive and in masking their origins. ally responsible for around ten accounts with Two main data sources underpin the evi- up to 2,000 followers each, tweeting at least dence and insights reported. First, there are fifty times daily.3 One source suggests they a small number of published accounts and were required to make 135 comments per stories from former workers at the Internet twelve-hour shift, working in internet forums Research Agency describing its organisation and that they would be provided with five and routines. This is supplemented by analy- keywords to feature in all posts to encourage sis of the ‘FiveThirtyEight Internet Research search engine pickup.4 The latter is consistent Agency Twitter dataset’—an extensive non- with workers describing receiving regular anonymised corpus of tweets posted by IRA ‘taskings’ from their managers in terms of a accounts. As detailed below, this includes a list of subjects/topics to focus upon.5 Similar large number of Russian language and tasks were defined in relation to targeting American facing accounts. But there were comments towards outlets such as CNN, BBC also accounts messaging in a number of and the New York Times.6 other languages. Herein we focus in particu- One interviewee described working in lar on accounts that were oriented towards teams of three: operator one would function Germany, as much less attention has as ‘the villain’ criticising the authorities; then focussed upon what these were doing. the others would enter a debate with him/ her. One would post an image/meme in support of their argument, the other posting Work at the Internet Research a link to a supportive source.7 Other intrigu- Agency ing comments include a suggested pattern for developing accounts in that they are There are at least seven published journalis- started with a politically neutral stance, tic accounts, based upon interviews with for- which is amended later.8 Operators wrote mer employees at the Internet Research Twitter bots to amplify visibility and because Agency. Collectively, these provide insights the costs of doing so were low.9 A few oper- into the nature of the work it performed, ators were ideologically committed to their including the organisation of decision mak- work, but most were not. Workforce turn- ing and delivery, main roles and responsibil- over was high and featured a lot of young ities, and the performance indicators workers people and students.10 were subject to. By collating and analysing These organisational arrangements certainly these materials, it is possible to construct an help make sense of some of the patterns that outline picture of the organisational rhythms can be observed in the messaging data. For and routines that shaped the kinds of digital example, sixteen confirmed Internet Research behaviours presenting in the social media Agency accounts all used quotations from accounts they were operating. Orwell, Shakespeare and other famous liter- The picture painted is of an organisation ary figures as part of the account biography. based around an orthodox division of labour Similarly, a second group of accounts all with different departments focussing upon shared fragments of the same base image as specific geographic regions/countries, accom- the account profile picture. There is a strong panied by some platform specialisation.2 For probability that these were individuals engag- example, one unit focussed upon producing ing cognitive shortcuts to get the job done memes, whilst another was tasked with com- quickly. Given that the staff were under con- menting on posts by other users. Individual siderable pressure to meet their performance operators ran multiple fake accounts: trolls metrics and were not necessarily deeply were expected to make around fifty com- invested in their work, these are precisely the ments on news articles every day. Or, they kinds of ‘easing behaviours’ found in many were tasked with maintaining six Facebook organisational settings. Analogous to what pages, posting three times daily about the happens in police detective work and psycho- news, and discussing new developments in logical profiling, these little ‘tells’ and

2ANDREW D AWSON AND M ARTIN I NNES

The Political Quarterly © 2019 The Authors. The Political Quarterly published by John Wiley & Sons Ltd on behalf of Political Quarterly Publishing Co (PQPC). behavioural signatures can be used as clues identities, was critical to how IRA accounts about where suspicions should be directed. built a following to enhance their persuasive One especially salient point though is that capacity and capability. Many did so over these social media accounts that were used several years, but not all did. Some tried to for spreading disinformation were not trans- shortcut the process of building audience mitting such materials all the time. Although and influence. there was quite considerable variation, broadly speaking they seemed to operate around an 80:20 ratio. That is, most of the Buying followers time these accounts were engaged in mim- An alternative to the normal organic strategy icking the kinds of interests and values and long-term investment required to grow coherent with the social identities that they follower numbers, is to ‘buy’ a following. were ‘spoofing’, and then occasionally they Websites such as ‘buycheapfollowerslikes.org’ would message avowedly political content. offer to increase a client’s Twitter following This pattern of behaviour was aided by the by 1,000 accounts for less than $20. According fact that many accounts would expend a lot to reviews, the followers have profile pictures, of time ‘amplifying’ messages from other unique bios and are active tweeters; however users, and then only rarely ‘authoring’ new they will not interact with posts, as they are material themselves. Such patterns are bots. This is what is depicted in Figure 1, important as they render the task of defini- which is an account run by the IRA that sim- tively attributing accounts to Kremlin direc- ply purchased follower accounts when set up. tion and control challenging. It is not clear whether such an action was performed as part of the overarching organisa- How IRA accounts built audience tional strategy, or because a worker was fl ‘gaming’ the performance measures they were and in uence subject to. In the graph, a running total of Spoofing personas in terms of constructing a tweets are plotted on the X axis and follower false account profile, and then messaging count on the Y axis. The dotted line represents around issues of interest to the thought com- its followers and the solid line those accounts munities associated with such digital social it is following.

TodayInSyria 35000

30000

25000

20000

15000

10000

5000

0 1 387 773 8493 3861 4633 5019 5405 7335 8879 9651 1159 1545 1931 2317 2703 4247 7721 9265 3089 6177 6563 8107 3475 5791 6949 11967 12353 13511 14669 10423 10809 11195 11581 13897 14283 15055 10037 12739 13125 Running total of tweets following followers

Figure 1: Example of an account purchasing followers

H OW R USSIA’ S I NTERNET R ESEARCH A GENCY B UILT ITS D ISINFORMATION C AM- PAIGN 3

© 2019 The Authors. The Political Quarterly published by John Wiley & Sons Ltd on behalf of The Political Quarterly Political Quarterly Publishing Co (PQPC). In this case, the English language account following the IRA account (follow-back). was initiated in 2015 by it acquiring 10,000 After a few days, the IRA operator then followers in a very short period of time. We unfollows the accounts, increasing their ‘fol- have identified similar patterns of behaviour lowers per followed’ ratio and with it boost- in several of the German language IRA ing the account’simplied‘authority’,atleast accounts, as exemplified in Figure 2. in terms of how this is assessed by platform On 4 June 2016, this account very rapidly algorithms. This tactic is commonly used by received 5,000 new followers. As can be seen ‘social media influencers’,suchascelebrities at two later points on the graph, the follower or marketers. Figure 4 tracks this pattern in count drops by around 1,000 accounts; this relation to a German language account. is likely due to the fake accounts being dis- The solid line documents how this account covered and banned by Twitter. The number followed other accounts over time. They added of followers is also unusually high: research thousands of accounts in a very compressed suggests the average Twitter account has 453 time frame hoping to get ‘follow-backs’.How- followers.11 ever, a high following to follower ratio indicates a low Twitter authority score, so they Follower fishing need frequently to stop following thousands of accounts as well. This creates a step like pat- A second pattern of behaviour designed to tern of steep rises and sharp drops. build audience and influence is ‘follower The longitudinal analysis to detect these fishing’. The logic of which is outlined by kinds of behavioural patterns is time- the metrics in Figure 3 below, based upon consuming. But in an attempt to test how an English language account. widespread it was, we analysed sixty-nine Around half of the most active IRA ‘Ger- randomly selected IRA Twitter accounts, with man’ Twitter accounts engaged in prolific different country focusses. Table 1 shows dra- ‘follower fishing’, with thousands of accounts matic differences between the behaviour of being followed and unfollowed regularly. The accounts ‘representing’ different countries. For tactic works by the IRA accounts following example, no Italian, but 86 per cent of the US hundreds or thousands of new accounts in a accounts reviewed employed follower fishing very short time frame. The aim is to get the to increase their followers and 21 per cent pur- newly added accounts to reciprocate by chased followers.

Erdollum 8000

7000

6000

5000

4000

3000

2000

1000

0 1 72 427 498 569 853 924 995 143 214 285 356 640 711 782 1350 2060 1066 1137 1208 1279 1421 1563 1634 1705 1776 1847 1918 1989 2202 2273 2415 2486 2557 1492 2131 2344 Running total of tweets following followers

Figure 2: Example of a German account purchasing followers

4ANDREW D AWSON AND M ARTIN I NNES

5000

4000

3000

2000

1000

0 1 89 177 265 353 441 529 705 793 881 969 617 2553 2641 1057 1145 1233 1321 1409 1497 1585 1673 1761 1849 1937 2025 2113 2201 2289 2377 2465 2729 2817 2905 3169 3257 3345 2993 3081 Running total of tweets following followers

Figure 3: Basic pattern of ﬁshing for followers

Berkhoﬀ85 6000

5000

4000

3000

2000

1000

0 1 78 155 232 386 540 694 771 848 925 309 463 617 1618 1772 2465 2773 1695 1849 1926 2003 2157 2234 2311 2388 2542 2696 2850 3004 1310 2080 2619 2927 1079 1156 1233 1387 1464 1541 1002

Running total of tweets following followers

Figure 4: German example of ‘follower ﬁshing’

Within the large amount of mainstream Narrative switching media commentary that has attended to the A third pattern of IRA account behaviour activities of the IRA over the past couple of ‘ ’ years, seventeen spoofed accounts have was narrative switching . The operators 12 would start by talking about fairly mundane acquired a certain public infamy, either ’ owing to the contents of their messaging or issues, consistent with the spoofed owner s the relatively sizeable numbers of followers persona. However, at some point, often after they had. All of them engaged in follower an extended time period, messages became ﬁshing to build audience and inﬂuence. overtly political and frequently aligned with

H OW R USSIA’ S I NTERNET R ESEARCH A GENCY B UILT ITS D ISINFORMATION C AM- PAIGN 5

Region Total Follower FF % Purchased Purchased % Accounts Fishing Followers Ukraine 3 1 33% 2 67% Iraq 7 1 14% 1 14% Italy 8 0 0% 0 0% United Kingdom 9 6 67% 2 22% Germany 11 4 36% 1 9% United States 14 12 86% 3 21% Famous Accounts 17 17 100% 4 24% (Unknown/US) Totals 69 41 59% 13 19%

established pro-Russian interest narratives. instead of the eight it had previously. The The IRA not only switched from banal to tweets appear to be just news headlines from pro-Russian views, but also switched Germany and around the world, most of abruptly between different political positions which are retweets. according to current Russian operational Previously, this account had posted anti- priorities, or even just to create confusion. Alternative fur€ Deutschland (AfD) statements, Narrative switching can also be used to col- such as: ‘at least she is not populist stupid as lect certain followers, for example people the afd in #merkelmuststay’: ‘the people who interested in French yellow jacket protests, choose the #afd are just sick # MerkelMustS- so that they can be targeted with messages tay’; and, ‘#AfD is shit, AFD is shit at a later time. In some instances, these #MerkelMustStay’. On 24 September, how- switches happened after the account was ever, it suddenly started posting original pro- dormant for a time. However, this was not AfD tweets including: ‘I #chooseAfD, because always the case. I remember # asylum crisis and no longer A collection of IRA accounts were identi- trust #Merkel!’; ‘I #chooseAfD, because I want fied where there was a significant ‘pause’ in to live in the Federal Republic instead of Cali- their messaging, lasting several months. phate #Germany!’; and retweeting ‘Every Potentially, this might mean that they were German #Patriot will vote #AfD tomorrow! dormant accounts that had been purchased We need a political earthquake to save #Ger- by IRA operators. Alternatively, this break many! # Btw17 #NoAntifa #NoIslam’. could have been used to suppress past politi- Possible (unsubstantiated) explanations for cal affiliations so the account could be ‘re- this switch in position are that it was a direct purposed’ by the operator. Another theory response to Chancellor Merkel’s public state- suggests dormancy because IRA operational ment on 14 September that the EU would priorities had changed. For example, German not consider lifting sanctions on Russia.13 In accounts can only really be used to influence the September elections AfD achieved what German opinion. As such, if IRA manage- The Guardian called a ‘stunning success’; ment or their political overseers had deter- meanwhile Merkel’s party had their worst mined an alternative priority for their staff, result since World War II.14 It should be then these accounts may have been less rele- noted that although each of the eleven vant, and the operators’ attention directed accounts posted pro-AfD tweets, their num- elsewhere. bers were small compared to the pro-Merkel Figure 5 shows the graph for one German tweets they had previously shared. One rea- account displaying this pattern of behaviour. son for this could be that the German At some point during the break, 80 per cent authorities were said to be on the lookout of this account’s tweets were deleted by its for Russian interference and operators were operator. On 14 June 2016 it began tweeting directed by their managers to be subtler than again, but with seventy-five followers in the past.

6ANDREW D AWSON AND M ARTIN I NNES

The Political Quarterly © 2019 The Authors. The Political Quarterly published by John Wiley & Sons Ltd on behalf of Political Quarterly Publishing Co (PQPC). THOMAS_GERSTER 100 600 90 80 500 70 400 60 50 300 40 30 200 20 100

10 Running total of tweets 0 0 21/07/2016 12/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 30/07/2016 12/08/2016 12/08/2016 12/08/2016 14/06/2017 22/06/2017 30/07/2017 21/07/2016 21/07/2016 21/07/2016 21/07/2016 21/07/2016 12/08/2016 12/08/2016 11/07/2017 13/08/2017 06/09/2017 24/09/2017

following followers tweets

Figure 5: Example of ‘narrative switching’

patterns on a time-line. The larger the bubble, Synchronicity analysis the greater the volume of messages sent at As noted previously, IRA operators were that point in time by that account. required to run multiple accounts and to gen- It can be observed that the key ‘pulses’ of erate certain volumes of activity. This pro- messaging activity are very similar, but not vides an opportunity for attributing accounts identical. All of these accounts stopped post- to Kremlin direction and control by analysing ing tweets on 12 August and then were inac- temporal sequences of messaging behaviour. tive for a minimum of six months. The The method for doing this involves ‘syn- inference drawn is that these accounts were chronicity analysis’ as it looks for simultane- probably being controlled by one author, ous patterns of messaging as a way of possibly using a system such as Tweetdeck. identifying where multiple accounts are This would be consistent with the working being controlled by the same author. arrangements at the Internet Research The potential power and value of syn- Agency described above. chronicity analysis is demonstrated by exam- In July and August 2016, these accounts ining the posting activities of eleven prolific sent hundreds of pro-Merkel tweets, often IRA accounts adopting a pro-Merkel stance. using the hashtag #MerkelMustStay. As with On 21 July 2016, @Martin_S32 posted its first other IRA tweets, they frequently integrated tweet of the day ‘Mommy is the best an unrelated trending hashtag in order to #MerkelMustStay’. In total, it posted 265 push their own hashtag, for example tweets that day, most displaying very strong #WorldElephantDay. These posts came dur- support for Angela Merkel. Initially it was ing a period when Angela Merkel was under assumed that this might be one of a small intense political pressure to step down as number of known Russian accounts adopting Chancellor, with Reuters noting that in Jan- this stance, with others communicating more uary, 40 per cent of Germans thought she extreme right-wing messages. Further investi- should resign over her refugee policy.15 This gation revealed, however, that a series of IRA would be consistent with the Russian state’s accounts were communicating messages sup- known modus operandi for seeking to lever- portive of the German Chancellor. Even more age political weakness to amplify social and significantly, when the timings of these mes- political tensions. sages were plotted, they displayed a very sim- Germany was subject to a series of terror- ilar pattern. Figure 6 plots these activity ist attacks between 18 July and 26 July 2016,

H OW R USSIA’ S I NTERNET R ESEARCH A GENCY B UILT ITS D ISINFORMATION C AM- PAIGN 7

Account ID 8 9 10 11 12

Figure 6: Pro-Merkel accounts schedule

in which fifteen people died. The so-called with the timings of the tweets, this points to Islamic State claimed responsibility for two the likelihood of a single person or team of the attacks and three of the attackers were operating all of these accounts in a co-ordi- asylum seekers.16 The Nice terror attack also nated strategic fashion. occurred in July, sending shockwaves Synchronicity analysis revealed this group around Europe. Despite this sequence of comprised forty-five accounts, thirty-four events, these accounts were very supportive more than we had previously discovered. It of the Merkel government’s refugee policy: also revealed another German team of ‘Lady #Merkel, Stick to your line! “The twenty-five accounts operating in a very dif- #refugees” are not dangerous mass but peo- ferent manner, obsessed with Brexit, with ple! # MerkelMustStay’; and ‘The refugee three of their top five hashtags directly policy of Mrs. Merkel shows that she is com- related to the Brexit vote. These posted a passionate! #MerkelMustStay’. variety of messages that promoted the idea Consistent with this line, all of the of leaving the EU such as: ‘#Brexit will only accounts disparaged the anti-immigration have small consequences! You cannot act in AfD, with multiple variations of ‘#AfD is the EU #BritainInOut #GoodbyeUK #Remai- shit, AfD is shit #MerkelMustStay’ being nINEU #EUref’ and ‘Do not let yourself be posted during a time when polls were sug- manipulated with fear. #Brexit #BrexitOrNot gesting AfD was enjoying some mainstream #GoodbyeUK #BritainInOut’. Confusingly, support (35 per cent CDU to 12 per cent these same accounts also posted anti-UK AfD).17 The media has typically blamed Rus- messages in apparent offence that the British sia for supporting populist/anti-immigrant public had chosen to leave, tweeting ‘#bri- parties, but at a point where Merkel (and EU taininout #goodbyeUK Let them drown on unity) was politically weakened, Russian their island now’ and ‘#britaininout #good- controlled accounts were messaging support byeUK We do not need snobs’. for her domestically. There were significant In direct opposition to the pro-Merkel similarities between the content of tweets accounts, the latter group invoked anti-refu- being posted by these accounts. Some were gee sentiments: ‘#britaininout #goodbyeUK copied verbatim, and others had single word You flee like #Refugees’ and ‘#britaininout differences, usually a hashtag. Combined #goodbyeUK Take refugees with you!’. They

8ANDREW D AWSON AND M ARTIN I NNES

The Political Quarterly © 2019 The Authors. The Political Quarterly published by John Wiley & Sons Ltd on behalf of Political Quarterly Publishing Co (PQPC). also blamed Merkel’s migration policy for provides an important corrective to much of increasing terror in Europe ‘Thank you, Mrs the public debate that has taken place in the Merkel, that citizens now have to be afraid West about Kremlin disinformation cam- of terror in Europe. #stopptTerror’ and paigns, which has worried predominantly spouted populist rhetoric ‘Why can not our about impacts in these contexts. But this warships send these #migrants back immedi- focus neglects just how much of the IRA’s ately? #stopptTerror’. The Russians were effort was directed towards influencing the amplifying both sides of the political argu- views and perceptions of Russian speakers. ment simultaneously, trying to increase the In addition to the main cluster, the linked social fissures associated with them. accounts in the top right-hand corner and When the high profile (infamous) Internet constituting the second largest group (box Research Agency accounts, were tested in B), look to be those associated with the IRA’s this way only two (@southlonestar and American department. Intriguingly, in terms @southlonestar2) could be linked through of their temporal activity profiles and mes- synchronicity analysis (the latter appears to sage contents, these could be loosely con- have been set up as a ‘backup’ in case Twit- nected to a much smaller group of Russian ter suspended @southlonestar). One possible facing accounts. The latter were probably implication of this is that if an account engaged in covering similar topics, but for a achieved a certain scale of influence, then different audience. operators focussed on running that one per- Around the edges of the diagram, are a sona. Alternatively, it may simply be an arte- large number of smaller ‘satellite’ clusters. The fact of higher skill operators being in charge accounts here can often be distinguished on of these accounts. the basis of them using different languages, or Rather than just retrospective linkage, syn- focussing upon particular social identity poli- chronicity analysis has two potential uses tics (such as Black Lives Matter). Box C shows going forward. First, if there is an identified a cluster of forty-five German accounts, which Kremlin controlled account, then treating include the pro-Merkel accounts we talked temporal pulses of messaging activity as a about earlier. In total, 2,031 or 71 per cent of behavioural signature might enable identifi- the IRA Twitter accounts in the dataset could cation of other ‘accounts of interest’. Equally be linked to at least one other, comprising 119 possible is that, with a number of accounts, separate clusters. This affords a clear sense of the presence of similar pulsing sequences how multiple accounts were being employed can be interpreted as a potential indicator of by IRA operators to push key messages in a a common controller. co-ordinated fashion. To test the potential of these methods we applied them to all 2,848 Twitter accounts The attribution challenge featuring in the full Twitter dataset. The results are represented graphically in Fig- Detailed forensic analysis methodologies of ure 7 where the nodes represent IRA Twitter the kind outlined above are necessary for accounts, shaded by clusters. Some clusters establishing an evidence base for detecting have had their shapes changed to make current and future activities of a similar nat- them easier to differentiate. ure. This is important given the increasing Representing the data in this way, the difficulties associated with confidently closeness in proximity between two con- attributing accounts involved in communi- nected accounts indicates how similar their cating disinformation to Kremlin direction tweeting activity is. Thus, the large cluster in and control, as they have become increas- the centre of the graphic (box A) represents ingly sophisticated. Not all disinformation 449 accounts that have their region predomi- comes from overseas; much of it is authored nantly set to Russia and tweet primarily in by citizens resident in Western countries. Russian. This is significant in that it reflects When such sources are mis-attributed, it is the pre-eminent strategic objective of the especially problematic, as it negatively Russian state, in terms of the perceived impacts public confidence in the authorities importance of managing domestic public involved and can be used as propaganda opinion and in the ‘near abroad’. This against them.

H OW R USSIA’ S I NTERNET R ESEARCH A GENCY B UILT ITS D ISINFORMATION C AM- PAIGN 9

Figure 7: Network graph of synchronicity analysis on all IRA accounts

Such challenges have been exacerbated by their methods accordingly, the consequence a general recalcitrance on the part of the prin- being that some of their approaches have cipal social media platforms to admit proac- become more sophisticated and harder to tively that malign actors are operating on detect. Far more impactive, therefore, in their systems. Whilst they have responded to terms of leveraging disruption on the capac- these issues when pushed by governments, ity and capability to spread disinformation, their responses in terms of increasing restric- is a strategy that seeks to build an under- tions to application programming interface standing of a disinformation network over (API) access and related interventions have, time, and implements interventions against somewhat ironically, also made it harder for multiple nodes simultaneously. This is how independent researchers to assist in detecting police investigators have learned to do dis- and attributing disinforming communications ruptions of criminal networks in ofﬂine to Kremlin backed accounts. There is a dan- spaces to maximise impact. ger that the platform providers are manoeu- vring into an arrangement where only they are positioned routinely to detect bad actors. Conclusion Further challenges to the work of attribut- In his coruscating account of life in Russia ing disinformation has resulted from the and the state’s normalised use of ‘soft facts’ strategy pursued when a suspected Kremlin to convey multiple and shifting ‘truths’ to its backed account (or accounts) are detected. citizens, Peter Pomerantsev, articulates how Typically, steps are taken either to ‘take the aggregating effect is a profound suspen- down’ the account from the platform, or to sion of belief.18 Unlike classic propaganda, ‘expose’ its presence through publicising its the design is not intended to seduce people suspicious activities. The issue is that such to invest in a particular ‘truth’, but rather to approaches are of limited effectiveness in render them in a state of profound and radi- altering the dynamics of the disinformation cal doubt about what to believe—a state of ecosystem. Moreover, this has enabled the epistemic anarchy. IRA and other Kremlin backed units to learn One of the most striking aspects of the how they are being detected and to adapt growing number of analyses of the IRA’s

10 A NDREW D AWSON AND M ARTIN I NNES

The Political Quarterly © 2019 The Authors. The Political Quarterly published by John Wiley & Sons Ltd on behalf of Political Quarterly Publishing Co (PQPC). activity, especially around the 2016 US presi- Notes dential election, is just how extensive and 1 H. Benkler et al., Network Propaganda: Manipu- varied it was. This, notwithstanding, it is lation, Disinformation and Radicalization in Ameri- equally vital to recognise that similar efforts can Politics, New York, Oxford University have been deployed across a number of other Press, 2018. situations and settings. With this in mind, 2 RBC Magazine, ‘Investigation of RBC: how the what is required now is a series of detailed “factory of trolls” worked in the elections in and forensic analyses, of the kind outlined the United States’, 17 October 2017; https:// herein, to distil the key operational tactics www.rbc.ru/magazine/2017/11/59e0c17d9a and techniques that were being used. If this 79470e05a9e6c1; Radio Free Europe, ‘One pro- ’ can be accomplished, then it enhances capac- fessional Russian troll tells all , 25 March 2015; ity and capability to detect similar attempts https://www.rferl.org/a/how-to-guide-russia n-trolling-trolls/26919999.html (both accessed going forward, and to constrain their efficacy. 9 April 2019). The particular value of the behavioural 3 The Higher Learning, ‘Russia has a troll models discussed is that they document army that is trying to mold public opinion fairly unusual patterns of activity that func- on internet news sites’, 4 June 2014; http:// tion as ‘signatures’ or ‘tells’ that an account thehigherlearning.com/2014/06/04/russia-has- is possibly being run by an operator with a-troll-army-that-is-trying-to-mold-public-opi specific intents. The potential is one that is nion-on-internet-news-sites/ (accessed 9 April similar to how police detectives use beha- 2019). fi 4 Radio Free Europe, ‘One professional Russian’. vioural signatures to pro le repeat offenders: ! ‘ ’ their digital equivalents can be used to 5 Yahoo , Trolling for Putin: Russia s information war explained’, 5 April 2015; https:// detect malign influencers online. A benefitof www.yahoo.com/news/trolling-putin-russias- the kinds of diagnostic data used to ascertain information-war-explained-063716887.html (ac- these behaviours is that they are largely lan- cessed 9 April 2019). guage agnostic and do not rely upon an abil- 6 Radio Free Europe, ‘One professional Russian’. ity to read and interpret the contents of 7 Ibid. what is being communicated. 8 BuzzFeed, ‘Documents show how Russia’s troll At this precise moment, it is difficult to army hit America’, 2 June 2014; https://www. know how worried we should collectively be buzzfeednews.com/article/maxseddon/docume nts-show-how-russias-troll-army-hit-america (ac- about disinformation communication. There ‘ are indications that despite the efforts of cessed 9 April 2019); Radio Free Europe, One professional Russian’. governments, intelligence agencies and plat- 9 RBC Magazine, ‘Investigation of RBC’. form providers, misinformation and disinfor- 10 Yahoo!, ‘Trolling for Putin’. mation is becoming an endemic feature of 11 KickFactory, ‘The average Twitter user now the modern media ecosystem. And whilst has 707 followers’, 23 June 2016; https://kick there is certainly something objectionable factory.com/blog/average-twitter-followers-upda about attempted behaviour modification in ted-2016/ (accessed 9 April 2019). respect of democratic processes and out- 12 IRA accounts tested: TEN_GOP, Blackmatterus, comes, there is actually remarkably little Blacknewsoutlet, Blacktolive, Bleepthepolice, robust evidence that such disinforming com- Crystal1johnson, Jeblary2016, Jenn_abrams, Lgb- munications have a discernible measurable tunitedcom, Muslims_in_usa, Pamela_moore13, Southlonestar, Thefoundingson, Tpartynews, impact upon how the majority of people Trayneshacole, Usa_gunslinger, Wokeluisa. think, feel or act. Messaging of this kind ! ‘ fi ‘ ’ ’ 13 Yahoo , EU rm on Russia sanctions may be better at channelling peoples pre- over Ukraine: Merkel’, 14 September 2018; existing values and opinions than it is in https://sg.news.yahoo.com/eu-firm-russia-san changing them. Perhaps then it is more ctions-over-ukraine-merkel-153248229.html (ac- appropriate to argue that disinformation has cessed 9 April 2019). more impact in shaping the issues we collec- 14 C. Mudde, ‘What the stunning success of AfD tively think about, than what we individu- means for Germany and Europe’, The Guardian, ally think. That is, its pernicious influence 24 September 2014; https://www.theguardia resides in framing and agenda setting what n.com/commentisfree/2017/sep/24/germany- elections-afd-europe-immigration-merkel-radica troubles come to be defined as key public l-right (accessed 9 April 2019). policy problems.

H OW R USSIA’ S I NTERNET R ESEARCH A GENCY B UILT ITS D ISINFORMATION C AM- PAIGN 11

© 2019 The Authors. The Political Quarterly published by John Wiley & Sons Ltd on behalf of The Political Quarterly Political Quarterly Publishing Co (PQPC). 15 Reuters, ‘Forty percent of Germans say Merkel https://www.reuters.com/article/us-germany- should resign over refugee policy: poll’,29 afd-idUSKCN10B0FR?feedType=RSS&feedNa January 2016; https://uk.reuters.com/article/ me=worldNews (accessed 9 April 2019). us-europe-migrants-germany-merkel-idUKKC 17 Ibid. N0V70KM (accessed 9 April 2019). 18 P. Pomerantsev, Nothing is True and Everything 16 Reuters, ‘Germany’s far-right AfD claws back is Possible: The Surreal Heart of the New Russia, some support after attacks’, 31 July 2016; London, Faber & Faber, 2014.

12 A NDREW D AWSON AND M ARTIN I NNES