Characterizing the Social Media News Sphere Through User Co-Sharing Practices

Proceedings of the Fourteenth International AAAI Conference on Web and Social Media (ICWSM 2020)

Characterizing the Social Media News Sphere through User Co-Sharing Practices

Mattia Samory∗ Vartan Kesiz Abnousi Tanushree Mitra Computational Social Science Department of Computer Science Department of Computer Science GESIS, Cologne, Germany Virginia Tech, VA, USA Virginia Tech, VA, USA [email protected] [email protected] [email protected]

Abstract which characteristics of the news outlets drive the actual audience engagement on social media. What do users share We describe the landscape of news sources which share so- and reply to most? This poses challenges for identifying cial media audience. We focus on 639 news sources, both which of the many assessments to prioritize when evaluat- credible and questionable, and characterize them according ing news. In particular, the assumption that users are con- to the audience that shares their articles on Twitter. Based on user co-sharing practices, what communities of news sources suming an exclusive diet of partisan-aligned news appears emerge? We find four groups: one is home to mainstream, increasingly less relevant in the social news sphere, which high-circulation sources from all sides of the political spec- calls for a more nuanced characterization (Guess et al. 2018; trum; one to satirical, left-leaning sources; one to bipartisan Starbird 2017). conspiratorial, pseudo-scientific sources; and one to right- This paper provides just such a characterization. We fo- leaning, deliberate misinformation sources. Next, we measure cus on 639 news sources, both credible and questionable, which assessments of credibility, impartiality, and journalistic through a quantitative analysis of over 31 million Tweets and integrity correspond to social media readers’ choices of news 1 million news articles. We adopt multiple measures of news sources, and uncover the multifaceted structure of the social source quality, including external assessments of sources’ news sphere. We show how news articles shared on Twitter differ across the four groups along linguistic and psycholin- factuality, impartiality, and journalistic integrity. Then, we guistics measures. Further, we find that with a high degree compare and contrast the external assessments of the news of accuracy (˜80%), we can classify in what news commu- sources, the characteristics of their articles, and the audience nity an article belongs to. Our data-driven categorization of that shares them on Twitter. This brings novel insight into news sources will help to navigate the complex landscape of which external assessments of news sources correspond to online news and has implications for social media platform common audience, as well as into what types of news arti- maintainers to reliably triage questionable outlets. cles do different audience pools engage with. The paper is structured around three research questions: Introduction RQ1: What communities of news sources emerge, when considering the sharing practices of their social Two-thirds of American adults read news on social media, media audience? even though a majority of them expect the reported information to be mostly inaccurate (Matsa and Shearer 2018). First, we connect sources to reflect how many users share It is a pressing concern to inform users about the quality of links to both—a costly signal that sources cannot easily fal- the news outlets in their social media feeds. In response, so- sify (Donath 2011). We find four communities of connected cial media platforms, advocacy groups, and the social com- sources: highly circulated news sources spanning the entire puting research community developed multiple assessments political spectrum; engaging misinformation, such as click- of the quality of news sources—for example, political slant bait and satirical news sources; factoid misinformation, such (Elejalde, Ferres, and Herder 2017) or credibility (Soni et al. as conspiratorial and junk science sources; and misinforma- 2014). These assessments of news outlets allow researchers tion with an ulterior motive, comprised of far-right propa- to study the social news sphere and to characterize its so- ganda and fake news sources. cial media audience. However, on the one hand, there is no single reliable assessment of the quality of a news out- RQ2: How well external assessments of news sources let (Zhang et al. 2018). On the other, we know little about distinguish the audience-based news communities?

∗A portion of this work was conducted while the author was at We ﬁnd that the social news sphere does not simply follow Virginia Tech partisan polarization. We show that it is necessary to com- Copyright c 2020, Association for the Advancement of Artiﬁcial bine multiple assessments of political slant and journalistic Intelligence (www.aaai.org). All rights reserved. norms to explain similar audiences on Twitter.

602 Figure 1: Cluster composition in terms of expert assessments of credibility. We map the primary category of opensources.co for each source to the corresponding cluster. Cluster 1 is the destination for all of the credible sources, as well as for political misinformation sources; cluster 2 is home to satirical and clickbait sources; cluster 3 hosts most conspiratorial, unreliable, and junk science sources; cluster 4 is largely comprised of fake news and biased sources.

Finally we look for differences in the news articles that Webster and Ksiazek 2012). These studies contribute funda- the communities of news sources shared on Twitter. mental methods to assess the levels of fragmentation of the RQ3: What kind of language and engagement attributes social news sphere. However, results from this line of work characterize communities of news sources? are ambiguous, thus calling for further study. One study hypothesized that audience behavior is the in- We analyze how the articles differ in content, style, senti- clination of users to engage with like-minded media, result- ment, psycholinguistic categories, as well as in how much ing in a highly fragmented news media sphere (Webster and engagement they received on Twitter. Ksiazek 2012). Indeed, most social media users are often not To summarize, our data-driven study highlights a discon- careful about shaping a balanced information diet for them- nect between how experts categorize news sources, and how selves, and consume content on a limited number of topics their audience selects them on Twitter. We shed light on (Kulshrestha et al. 2015). Furthermore, certain topics, such which expert assessments do help in differentiating between as conspiratorial (Bessi et al. 2015) and political (Barbera´ audience-based clusters of news sources—although in pre- et al. 2015) content, seem especially polarizing, and result viously unidentified combinations. Our analysis of the con- in a fragmentation of the social media audience. In partic- tent of the articles pertaining to each cluster can guide media ular, several works depict social media audience as divided scholars in assessing what kind of news sources bring to- into conservative and liberal stances (Conover et al. 2011; gether or divide social news media readerships. Our results Garimella et al. 2018; Bakshy, Messing, and Adamic 2015). also yield important practical implications. This work can In contrast, another line of research finds little evi- help social media platforms to reliably triage new informa- dence of ideological segmentation in media use (Mukerjee, tion outlets, thus alleviating the labor-intensive labeling of Gonzalez-Bail´ on,´ and Majo-V´ azquez´ 2018; Webster and an ever growing information space. While creating informa- Ksiazek 2012). For example, Mukerjee, Gonzalez-Bail´ on,´ tion outlets is cheap, controlling its audience similarity space and Majo-V´ azquez´ find no evidence of selective exposure is costly; hence a reliable, hard-to-fake signal. For example, or self-selection of the audience, and surface a tightly inter- a new conspiratorial outlet will have to wield significant ef- connected core of news sources—fundamentally formed by fort to position itself close to credible information sources in legacy brands. Opposing the view of a politically polarized the shared social media audience space. audience, recent studies suggest that the dualism between conservative and liberal stances is increasingly less relevant Related Work in the social news sphere (Starbird 2017; Guess et al. 2018; Here we outline the scholarly work that informed our study. Druckman, Levendusky, and McLain 2018). In particular, First we present a line of research studying the social news these studies show that groups of news producers employ sphere from the point of view of an ecosystem of news pro- mechanisms that are yet to be studied deeply to influence ducers and their audience. Then, we discuss the large body their readers (Horne and Adali 2018; Starbird et al. 2018). of work aimed at characterizing misinformation and devel- Collectively, this body of literature highlight the impor- oping indicators for identifying it. tance of studying the social news sphere as an intercon- nected ecosystem of news sources, and not as a collection of Ecosystem of news and audiences on social media independent actors. We build upon their intuition, and take Most related to this work is research that identifies groups of the analyses a step further, by looking at how audience frag- news sources by the social media audience they have in com- mentation corresponds to characteristics of news sources as mon (Mukerjee, Gonzalez-Bail´ on,´ and Majo-V´ azquez´ 2018; well as of their content.

603 Differently from related work, we connect how Twitter Data users share news articles, how the emerging news commu- We first compiled a list of news sources of interest. Then, we nities from the user co-shares compare across the assess- collected tweets containing links to articles published by the ments of the articles’ sources, and what are the distinguish- sources. We complemented this with the full-text of the arti- able characteristics in the content of those articles. Simi- cles linked from Twitter, and with multifaceted assessments larly to (Horne and Adali 2018), we use multiple assess- of the news sources. ments of news sources to study their relationship. However, we use common Twitter audience as a measure of source similarity, which sources cannot directly control. This work List of news sources: We started by curating a list of news sources and combined a wide selection of both credible and also differs from (Mukerjee, Gonzalez-Bail´ on,´ and Majo-´ 1 Vazquez´ 2018), which introduces a network backbone ex- questionable sources. We relied on OpenSources , a profes- traction method to remove spurious audience overlap—but sionally curated list of online sources available for free for which, potentially, also penalizes sources with small or public use. News sources in this resource range from cred- niche audience overall. To account for this, we employ in- ible to misleading and outright fake websites. OpenSources stead an information-theoretic measure for computing audi- heavily focuses on misinformation, with little emphasis on ence overlap (Martin 2017). credible content. Therefore, we complemented this list by including additional mainstream news websites. We first re- Misinformation in online social media ferred to various online resources, including surveys by Pew Research2 and NPR3. Then, a journalism and communica- The prevalence of online misinformation recently brought tions expert refined the list by referring to the most circu- the social news sphere at the epicenter of the social com- lated 4 and the most trusted news sources (Matsa and Shearer puting research community as much as of the public dis- 2018). In total, the list contains 639 distinct web domains. course. On one end of the misinformation spectrum there are sites explicitly designed to deceive people, to publish provably false claims, and to propagate them through so- Tweets containing links to sources: Next, we collected cial media platforms in an attempt to increase readership all tweets linking to news articles by the sources in our list. and profit (Marwick and Lewis 2017). On the other end are Specifically, we accessed data through the Twitter streaming satirical news sites which produce misleading content for API, continuously from September 2016 to February 2017, entertainment. Regardless of the sources’ intent and of how and filtering those containing links to the news sources of in- we label them (satire, clickbait, or the more contested term terest. We collected a total of 31,567,501 tweets. We further 5 “fake news”), misleading information has a negative effect collected all tweets that replied to them. We use this mea- 6 on citizen’s news consumption and on their ability to make sure as a signal for audience attention to the original tweet. fully informed decisions (Thorson 2016). Misinformation, once assimilated, is hard to eradicate from the convictions News articles linked from Twitter: Furthermore, we en- of its audience (Lewandowsky et al. 2012). Hence, a large hanced our dataset by collecting the news articles that the body of scholarly work has focused on characterizing online misinformation to prevent its spread. Notable among them 1opensources.co are Horne and Adali’s characterization of differences in the 2https://www.journalism.org/2014/10/21/political- news titles of fabricated and real stories (Horne and Adali polarization-media-habits/ 3 2017), Golbeck et al.’s attempt to automatically determine https://www.npr.org/sections/alltechconsidered/2016/10/28/ 499495517 whether a newswire article is satirical or not (Golbeck et al. 4 2018), and Ferrara et al.’s study of social bots spreading ru- https://www.cision.com/us/2017/09/top-10-u-s-daily- newspapers-2/ mors on Twitter (Ferrara et al. 2014). 5 Despite the scholarly efforts, identifying misinformation The dataset shows skewed distributions typical of social media: although it includes over 2 million unique users, 60% of the is still challenging. In fact, even humans perform poorly in tweets are authored by the top 1% most prolific users. identifying non-obvious misinformation (Kumar, West, and 6We relied on the list of over 300 news bots from (Lokot and Leskovec 2016). Defining what constitutes high-quality in- Diakopoulos 2016) to check for the presence of inorganic Twitter formation requires careful considerations about content at- activity. We found that bots in the list author less than 0.1% of our tributes and adherence to journalistic norms (Hayes, Singer, dataset. Out of the bots individually contributing over 1k tweets, all and Ceppos 2007; Diakopoulos 2015). Thus, recent research but two exclusively share credible news by a single source (these focuses on characterizing different types of misinformation are bots by major outlets like New York Times and Washington also, on summarizing reliable indicators, rather than auto- Post, or sports bots). The remaining two bots are 365Arizona, shar- matically identifying information (Zhang et al. 2018). ing overwhelmingly biased tweets from counterjihad.com and cen- Motivated by the need for reliable indicators of misinfor- terforsecuritypolicy.org, and dubvNOW, a newsletter born out of mation, in this work we characterize the landscape of news the University of West Virginia sharing half credible (usatoday), half conspiracy (infowars) news. Given the minor impact of bots, producers on Twitter. Instead of focusing on the prevalence we do not exclude them for the sake of completeness. Even if the of one type of misinformation among news sources or their list above might have missed some non-organic accounts, we stress social media audience, we study the relationship between that even the most prolific bots would skew the number of co- news sources and their audience on Twitter through multiple occurring users between two sources at most by 1, as we disregard dimensions of news source quality. contribution volume and focus only on user co-sharing practices.

604 widespread satirical/clickbait conspiratorial right-wing/fake washingtonexaminer.com usuncut.com activistpost.com angrypatriotmovement.com thegatewaypundit.com countercurrentnews.com 21stcenturywire.com usasupreme.com wnd.com occupydemocrats.com thedailysheeple.com truthandaction.org Newsmax.com attn.com russia-insider.com prntly.com heatst.com rawstory.com blacklistednews.com subjectpolitics.com dailycaller.com bipartisanreport.com washingtonsblog.com usanewsﬂash.com dailywire.com dailykos.com veteranstoday.com christiantimesnewspaper.com americanthinker.com addictinginfo.org investmentwatchblog.com ilovemyfreedom.org conservativereview.com politicususa.com ronpaulinstitute.org departed.co wsj.com alternet.org wearechange.org supremepatriot.com

Table 1: Top 10 central news sources in each cluster—i.e., sources that are the most cosine-similar to each KMeans centroid. tweets link to. We used the library newspaper to scrape a fuzzy string match to find the best candidate in Open- the text and publication date of the articles from the web. We Sources, and manually validated the match. discarded articles that are non-English or too short7. Since Out of 639 new sources, we include annotations for 595 multiple links within each web domain may refer to the same sources from OpenSources, 124 from Newsguard, 101 from article, we identify duplicates by stemming and matching MediaBias, 42 from Allsides, and 26 from Wikipedia. their plain text. We keep only the oldest instance of dupli- cate articles by publication date. The final dataset contains 1,212,304 articles. RQ1: Communities of news sources based on common audience External assessments of the news sources: We com- By aggregating the news sharing practices of Twitter users, pleted our dataset by gathering assessments of the news what communities of news sources emerge? sources along multiple dimensions of factuality, journalistic integrity, and impartiality. Our choice of dimensions was Method dictated by decades of research from journalism and com- We start by building a news source similarity space, based munications scholars(Sundar 1999). on shared Twitter audience. Then, we characterize groups of Factuality: OpenSources, a professionally-curated list of sources within this space. news sources, provides annotations for each news source based on a taxonomy of twelve assessments, such as fake and junk science. We use the primary annotation of each Measuring audience similarity through co-shares: We source as a judgment of its factuality. base our audience similarity measure on user co-shares: two 8 sources are more similar when more users share links to both Journalistic Integrity: Journalists at NewsGuard label of them. Intuitively, user-based similarity hypothesizes that news sources according to multiple criteria, including users choose news sources according to their interests, such whether sources regularly correct errors or disclose own- as the topics covered in the articles or their partisan affilia- ership and financing. Newsguard attributes a weight to tion; if many users choose two sources, those sources likely each assessment and combines them into an overall score cater to a common interest. One issue with using the raw from 0 to 100. An increasingly important assessment of number of co-sharing users as a similarity measure is that it the journalistic stance of online sources is whether they is bound by the overall number of users sharing a source. identify as alternative or mainstream outlets. To this end, High-circulation sources are more likely to be shared in- we label outlets as “alternative” or “mainstream’ accord- dependently of the users’ choices, and therefore may co- ing to the corresponding lists of sources occur with other sources by chance. The converse holds for from Wikipedia. smaller less popular news outlets. Therefore, we measure in- Impartiality: Finally, we also include assessments of po- stead whether a surprisingly high number of users share two litical impartiality by Media Bias/Fact Check9 and All- sources, given the number of users who share each source Sides10. Both services divide new sources into five cat- independently from the other, using positive pointwise mu- egories: biased left, center-left, center, center-right and tual information (PPMI) (Martin 2017). right. After obtaining the assessments from their website, we performed semi-automatic matching of the news do- Finding groups of news sources sharing Twitter audi- main names to the domains from OpenSources. We used ence: We adopt an unsupervised machine learning ap- 7less than 140 characters since we find that those mostly consist proach to cluster news sources according to user co- of broken links or 404 pages for URLs that no longer exists. sharing practices. We use the popular K-Means method with 8newsguardtech.com kmeans++ initialization. One crucial step in K-means is to 9mediabiasfactcheck.com tune the number of clusters and evaluate their validity. We 10allsides.com find the best number of clusters by looking for an elbow

605 point in explained variance, which offers a natural trade- off between accuracy and number of clusters. Thus, we as- sume four as the optimal number of news source clusters. We check that the ﬁnal clusters remain stable with respect to different random seeds.

Characterizing groups of news sources through external assessments of journalistic norms: Next, we look at the composition of each cluster in terms of external assessments of credibility, impartiality, and journalistic practices. We aggregate the assessment for the sources within each cluster. Since assessments lack complete coverage— e.g., some sources have assessments for credibility but not for transparency—we look at each separately. We do not Figure 2: Cluster composition in terms of external assess- run statistical tests on the differences between clusters, since ments of impartiality, according to MediaBias. We omit those would neglect the partial overlap of sources with dif- sources for which MediaBias does not provide assessments. ferent assessments. All political sides are represented in cluster 1—which is in line with their widespread appeal according to credibil- RQ1: Results ity assessments; all sources in cluster 2, containing satire and clickbait, are left-wing; cluster 3, home to conspirato- We find four clusters of news sources based on Twitter co- rial sources, shows a mix of left and right-wing sources; all sharing practices. Table 1 displays the top 10 news sources sources with an assessment in cluster 4, hosting predomi- closest to the cluster centroids. Next, we study the clusters nantly fake news, are right-wing. by looking at the external assessments of factuality, impartiality, and journalistic integrity of their members. We con- clude our results by showing which of the external assessment best conforms to the Twitter audience-based clusters. qualitatively consistent, therefore we omit them. Cluster 1 includes sources from all political sides. This is in line with the finding that emerged from the previous credibility as- Credibility: We study the credibility of the sources in sessments, that cluster 1 is home to high circulation credible each cluster. Figure 1 summarizes the findings. We find that and misinformation news sources. In contrast, cluster 2, con- all clusters contain a variety of OpenSources assessments— taining satire and clickbait, includes exclusively left-wing e.g., all clusters include conspiratorial sources to some ex- sources. Research from cognitive psychology suggests that tent. However, their composition yields a clearly defined pic- left-wing political inclinations are more eager to respond to ture. Cluster 1 is the largest of the four. It contains all of the pleasing content (Dodd et al. 2012), which may explain the credible sources in our dataset, as well many misinformation correlation between satirical content and left-wing political sources of widespread appeal, such as political (Breitbart) composition of the cluster. Cluster 3, home to most conspir- and satirical (liberalbias.com) sources. High circulation atorial sources, shows a mix of left and right-wing sources, sources like nytimes.com and wsj.com belong to the clus- such as the left-leaning activistpost.com and the right-wing ter. Cluster 2 is home to a significant fraction of satirical and returnofkings.com. Conspiracy theorizing is in fact a biparti- clickbait sources such as theonion.com, as well as left-wing san issue (Oliver and Wood 2014). Finally, cluster 4, hosting activist groups like dailykos.com. Cluster 3 hosts most of predominantly fake news, includes exclusively right-wing the conspiratorial, unreliable, and junk science sources, e.g., sources. Grinberg et al. also found a cluster of fake news corbettreport.com and prisonplanet.com. The fourth and last sources sharing overlapping audiences on the extreme right cluster contains misinformation with an ulterior motive: it is of the political spectrum (Grinberg et al. 2019). largely comprised of fake news and biased sources such as abcnews.com.co11, abcnewsgo.co, nbc.com.co12. Although cluster 1 contains relatively widespread news sources, nine Journalistic integrity: Finally, we examine the journalis- hateful news sources also belong to this cluster. Upon further tic integrity of the sources as assessed by Newsguard. We inspection, we find that users piggyback on the popularity of first look at the overall source quality score (0–100, a higher mainstream articles to promote hate by sharing links to both score means higher quality). A website needs a score of at in the same tweet; this explains the co-presence of hate and least 60 for it to receive an overall positive assessment. The widespread news in the co-sharing practices. results corroborate our previous findings. Cluster 1, which contains widespread news sources, receives the best scores: Impartiality: Figure 2 shows the composition of the clus- it has the highest mean (40.12) and median (32.5) score, and ters in terms of partisan bias. We present results based on many of its sources receive perfect scores. Cluster 2, follows assessments from MediaBias; results based on Allsides are a pattern similar to cluster 1. On the other hand, clusters 3 (conspiratorial) and 4 (fake) contain mostly low-quality 11https://en.wikipedia.org/wiki/ABCnews.com.co journalism. Barely any source in cluster 3 exceeds a score of 12https://mediabiasfactcheck.com/nbc-com-co/ 20 (with the notable exception of counterpunch.org, a non-

606 profit ultra-liberal magazine). Almost all sources in cluster assessments may be redundant or, more concerningly, that 4 are below the cutoff score of 60 for reputable sources. only few of them correspond to audience co-sharing prac- Next, we look at the break-down of the journalistic in- tices. Surprisingly, in all cases, the often-adopted assess- tegrity score into its different facets. Sources in clusters 1 ments of partisan bias by both MediaBias and Allsides per- and 2 score systematically the highest on all criteria. In par- form poorly. ticular, clusters 1 and 2 are notably higher than clusters 3 and 4 in regularly correcting errors, presenting information responsibly, and avoiding publishing false content. Which assessments correlate with audience variance? To corroborate these findings, we investigate the correlation between source assessments and audience similarity. To summarize We discover four clusters of news sources, We use principal component analysis on the PPMI-weighted according to common audience on Twitter. They show dis- audience similarity matrix, and identify the direction along tinct compositions in terms of credibility, impartiality, and which audience similarity changes the most. We focus on journalistic integrity. Therefore, we will interchangeably re- the first principal component, which explains the most vari- fer to the clusters with the following shorthand: ance (18%, more than double the variance explained by the cluster 1: widespread news sources second component). The component correlates highly and cluster 2: satirical/clickbait news sources positively with assessments of factuality, such as conspiracy and junk science, and only mildly and negatively with po- cluster 3: conspiratorial news sources litical bias. This confirms that audience similarity does not cluster 4: right-wing/fake news sources simply follow a polarized, partisan structure. We stress that the characterization in this section applies to Finally, to interpret along which direction does audience the clusters collectively, and individual sources may show vary, we inspect the sources that load highly on the princi- different characteristics. pal component. On the positive end we find conspiratorial sources that propose a revolution in public institutions. For example, the anarcho-capitalist conspiratorial website cor- RQ2: News source assessments distinguishing bettreport.com includes stories about deep state and global- the social news audience sphere on Twitter ist control, and loads among the most highly positive. On We identified communities of news sources that differ along the negative end, instead, we find sources adopting the polit- multiple assessments of credibility, integrity, and impartial- ical status quo. For example, the website madpatriots.com, ity. However, which assessments correspond to different which leverages established conservative narratives to craft choices of news sources by the audience? We answer this sensationalist headlines, such as “violent immigrants” and question by ranking the assessments according to how well “red scare,” loads the most negative. This echoes Starbird’s they distinguish the communities of shared audience. To cor- observation that political leanings of alternative news sites roborate this, we show that the top-ranking assessments also feature an anti-/pro-globalist orientation, rather than conser- correlate highly with overall audience variance. vative/liberal.

Which assessments best explain communities of shared RQ3: Identifying the source community of audience? We measure how well grouping sources by news articles on Twitter each assessment conforms to the communities of shared audience. We label sources with the cluster they belong to, The previous sections identified clusters of news sources as well as with each assessment—e.g., the left- and right- shared by distinct audiences, and characterized the journalis- wing bias labels given by MediaBias. Then, we compute tic qualities of the sources within the clusters. Yet, if we see how much the assessment and cluster labels overlap, accord- an article shared on Twitter, can we automatically identify ing to measures of cluster quality. We repeat this process for the cluster it belongs to? We provide a fine-grained content all assessments, and we rank them accordingly. analysis of the clusters articles. Then, we show that a classi- We find that the simple assessment of alternative vs. main- fier identifies the correct cluster with over 80% accuracy. stream media sources shows the highest homogeneity, completeness, and v-measure, whereas the assessments by Open- Method Sources show the highest Rand Index and Mutual Informa- In particular, we look at the content, style, sentiment, and tion. This discrepancy may be because few news sources psycholinguistic categories used in the articles, as well as have labels for the alternative vs. mainstream assessment, the overall engagement that the articles produce on Twitter. and therefore are penalized in the latter chance-adjusted measures. Furthermore, we ask if all the labels in Open- Content: First, we look at what kind of content distin- Sources’ assessments—12 of them—are actually informa- guishes the clusters. We rely on newspaper python module tive. We repeat the experiment assessing each label in a to extract the plaintext of the articles and to discard ones one-versus-all fashion, and we find that only few of them whose primary language is not English. Then, we prepro- correspond to the audience-based clusters: bias, clickbait, cess the plaintext using standard procedures of converting conspiracy, fake, and satire rank higher than all other as- to lowercase, removing punctuation and accents, striping sessments in the study. This suggests that the opensources whitespaces, and removing stopwords and links. Next, we

607 We use Sparse Additive Generative models—SAGE (Eisen- “our” often indicate references to group identity. Similarly, stein, Ahmed, and Xing 2011)—to find words that are spe- writers may use verb tense to signal endorsement: for exam- cific to each cluster. SAGE uses a regularized log-odds ratio ple, past tense increases psychological distance with the re- measure to contrast word distributions between one corpus ported facts, compared to present tense. We include the cate- of interest against a baseline corpus. We contrast articles gories social and emotional because they may signal less in- within each cluster against all the remaining. For example, formative, more sensationalist content. We also include mo- we compare the distribution of words in widespread news tion, cognitive processes, and sense words, since they are sources vs. the distribution of words in all other clusters. The known signals of truthfulness. Finally, the spoken category output will give the distinguishing words of the widespread exposes departure from journalistic style, capturing, for in- news sources, relative to other clusters. stance, the occurrence of swearwords and nonfluencies. To account for the high variability in article length, for every Style: Content analysis tells us what sources in each clus- article we compute the fraction of words in each category. ter are writing about. However, how are they writing about it? We next turn to writing style. We use two feature sets: cue Twitter engagement: Finally, we gauge the feedback that verbs, which reflect the sources’ choices in terms of journal- articles receive on Twitter. We compute the number of fol- istic reporting style, and stylometric features reflecting the lowers and friends (followees) of the Twitter users to assess sophistication of the writing style. their authority or hub status. For individual tweets, we report Cue verbs correspond to factuality judgments of journal- the number of favorites, replies, and retweets, which signal istic reports (Sauri 2008). Sources may choose certain types the reach of and engagement with the tweet. of cues to nudge their audience into believing a reported claim. For example, “WSJ learns that ...”asserts more cer- Measuring differences across clusters We use a machine tainty than “WSJ suspects that ....”Following (Soni et al. learning pipeline to assess differences in articles’ language 2014), we use five groups of cue verbs common in Twitter: and engagement between clusters. We train a multinomial “Report” (e.g., say, report), “Knowledge” (e.g. learn, admit), logistic regression model (henceforth MNLogit), penalized “Belief” (e.g. think, predict), “Doubt” (e.g. doubt, wonder), for dealing with sparsity and multicollinearity. We compare and “Perception” (e.g., sense, hear). each cluster to the baseline “widespread news sources.” We We also look at stylometric features that capture the arti- choose “widespread news sources” as a reference class be- cles’ style beyond choices for reporting. On the one hand, cause, according to our previous findings, we expect the sources may embrace more formal linguistic registers to content to be more varied and mainstream, and because appear more credible. On the other hand, they may make it contains the largest number of articles. Since MNlogit content more accessible by simplifying their language. We is a one-vs-all model, we validate the results with a post- assess the readability of the text using the SMOG index. hoc statistical significance test of the observed differences SMOG estimates the grade level needed for understanding between clusters. We perform a series of non-parametric the article, and correlates well with human judgments of text Kolmogorov–Smirnov (KS) tests for all six possible pair- clarity—high SMOG implies more complex language. wise combinations of the clusters. We chose the KS test over Additionally, we look into markers of complexity: the the parametric t-student to account for the fact that none of overall number of words, number of informative words, our features are normally distributed. For all features we as- type-token ratio, and long and complex words, all give us an sessed normality using Anderson–Darling tests. indication of the writer’s effort. Finally, we include function word types, such as conjunctions and prepositions. Function RQ3 Results: characterizing of news communities’ words do not carry meaning per se. However, they reflect language how people are communicating (Tausczik and Pennebaker We find that clusters significantly differ in the articles’ lan- 2010). guage and in the engagement they produce. Sentiment and psycholinguistic categories: Sentiment Content: Table 2 reports the words that best distinguish and psycholinguistic categories depict the emotional and articles from each cluster. The “widespread news” clus- psychological states evoked in the articles. We use two ter, home to a wide range of sources, intuitively does not scores from VADER (Hutto and Gilbert 2014): compound show a predominant topic: we find references to current score measures sentiment polarity, while neutrality captures events, like the fire which burnt the Grenfell tower and the the proportion of words that are not emotionally charged. marches against the Malaysian Prime Minister Najib. The Intuitively, sensationalist journalism uses more emotionally other three clusters, instead, manifest more focused top- charged and polarized language (Zhang et al. 2018). ics. Cluster “satirical/clickbait” features several polarizing Additionally, LIWC (Tausczik and Pennebaker 2010) of- terms (like lgbtq, @realdonaldtrump, impeach, protectors), fers validated categories of words for assessing such psy- as well as words often used to poke fun of stereotypical con- cholinguistic dimensions. We use the following categories. spiracy theorists (like pyramids, extraterrestrial), and food At the most superficial level, content word categories explic- items often featured in prodigious diets meant to generate itly reveal the articles’ focus—we include the topical cate- click revenue (like coconut, turmeric).13 Cluster “conspir- gory personal concerns. The categories pronouns and verb tense also expose the attentional focus of the writers, al- 13the top terms like “mashshare” and “flipboard” are artifacts of though more subtly. For example, pronouns like “we” and the automated text extraction, and correspond to the text found in

608 atorial” aptly uses the conspiratorial lingo, such as refer- ticular, we find that the clusters disproportionately express ences to the New World Order, Zionists, Illuminati, and the anger death-related words. In addition, they use more in- Rothschild—all being frequent actors in conspiratorial nar- formal (spoken word categories like swear) language, with ratives of world domination. Cluster “right-wing/fake” uses more references to collective identity (we pronoun) than terms of the hyper-conservative propaganda, such as refer- “widespread news sources”. These psycholinguistic cate- ences to the alleged lack of virility of the left (like effemi- gories signal attempts of engaging rather than objective con- nization, soy bois, and cucks) and political opponents (like tent (Hartung et al. 2016). Moreover, “satirical/clickbait,” antifa, leftwing, Ocasio-Cortez). In summary, we find that “conspiratorial,” and “right-wing/fake” express more cer- the content of the articles conforms with the external assess- tainty (cognitive mechanism categories like certain and in- ment of the sources within the news communities. sight) through visual language references (see); visual language makes concepts more concrete, thus more memo- Style: Beyond content, news sources may make subtle rable and accessible. Although journalistic news reporting stylistic choices to convey their message. Recall that all re- style implies a temporal focus on the present or the near sults are relative to the reference class, “widespread news.” past, we find that “satirical/clickbait,” “conspiratorial,” and All three remaining clusters use significantly fewer report “right-wing/fake” focus more on the future. This is in line cue words. Report cue words, such as sources report that... with the stylistic analyses in the previous section, suggesting or Whistleblower tells Congress.., are a landmark of formal that “widespread news sources” is the one that is most con- journalistic style. While “satirical/clickbait news” express forming to a formal reporting practices. Yet, psycholinguis- more doubt in its reports than “widespread news,” “conspira- tic categories allow us to characterize clusters with higher torial news” and “right-wing/fake” doubt less. In fact, “con- precision. For example, “satirical/clickbait” uses more so- spiratorial news” use more cue verbs expressing knowledge. cial words and references biological processes like sexu- Knowledge cues, such as Scientists find that change driven ality and health. Cluster “conspiratorial” shows the most largely by increased carbon dioxide.., are part of the class worry with the self (I pronoun) and the least focus on the of factive predicates, which imply the truth of the claim that present. Cluster “right-wing/fake” references conservative follows (Sauri 2008). values like family, home, and work, and expresses the most We then turn to stylometric features to probe the sophis- negative sentiment (anger and anxiety). tication of writing style. “Satirical/clickbait news” use more difficult to read language (possibly mimicking a pompous Twitter engagement The previous sections show that the writing style), whereas “right-wing/fake news” use simpler clusters departing from journalistic practices may seek the language, according to SMOG. This is reflected in the other audience’s attention through multiple content, stylistic, and markers of complexity: “right-wing/fake news” use fewer framing devices. Are they successful? Twitter users shar- long words, and a less varied vocabulary according to the ing links from the “widespread” cluster follow less, and are type-token ratio. Yet, “right-wing/fake news” also use more followed more than Twitter handles that share links from polysyllabic words (complex words in the table) and more “satirical/clickbait,” “conspiratorial,” and “right-wing/fake.” word types overall. This may signal a somewhat diluted lan- In other words, users tweeting about mainstream content guage: more verbose, but not as informative. We find that tend to be information authorities, rather than hubs. Links this is indeed the case. All three clusters use a larger pro- from sources in “widespread” also engage more users, both portion of function words than “widespread news,” which in terms of replies and retweets, than links in the other clus- means that the informative words are relatively fewer. In ters. Intuitively, the only exception is that the audience es- a nutshell, “widespread news” adhere to the expectations pecially favors links from the “satirical/clickbait.” It appears of standard journalistic style, whereas “satirical/clickbait that fringe content does not, after all, elicit as many reactions news,” “conspiratorial news,” and “right-wing/fake news” as mainstream articles. All reported effects are statistically use less formal, less informative, more accessible language. significant. We skip showing result table due to space limits. In particular, “conspiratorial news” and “right-wing/fake Automatically identifying an article’s cluster We con- news” express more certainty in their reporting, possibly to clude by demonstrating that the differences between clus- appear more credible. ters are not just qualitative, but that we can automatically Sentiment and psycholinguistic categories: Abstracting infer the cluster that produced an article with high preci- from the level of content, we next analyze the sentiment and sion. We train an XGBoost model using the style, sentiment, psycholinguistic categories expressed in the articles. Figure and psycholinguistic features presented above; we also in- 3 reports the differences between clusters. First, we sum- clude the top 50k terms in the articles weighted by TF-IDF marize the characteristics that cluster “satirical/clickbait,” as topical features. Since classes are heavily imbalanced, we “conspiratorial,” and “right-wing/fake” have in common, use SMOTE to oversample the minority classes in the train- and compare them to cluster “widespread news sources”. As ing set—test sets contain only actual and not synthetic data. expected, clusters “satirical/clickbait,” “conspiratorial,” and We assess model performance in a 10-fold stratified shuffle- “right-wing/fake” use more emotionally charged and polar- split scheme, and tune hyperparameters using a randomized, ized language (i.e. a higher score for neutral score). In par- cross-validated search approach. We find that we can classify all clusters with a weighted social buttons typical of clickbait sites. For integrity, we did not accuracy of 82% (weighted F1 score). We can classify editorialize them from the table. “widespread news sources” almost perfectly (90%). Al-

609 widespread news satirical/clickbait news conspiratorial news right-wing/fake news triforium, gassama, otw, mashshare, ﬂipboard, stum- altnews, in5d, naturalnews, commments, eagler, over- luzhniki, agung, moorland, bleupon, digg, attn, anonhq, dmca, coward, analyses, glp, sign, duely, effeminization, sidebars, shortcode, win- truea, screengrab, loading, adsense, quoting, eyeo, abu- photopin, adblock, ocasio, drush, southgate, grenfell, republish, spoilers, impeach, sive, nwo, ammol, button, digestible, corruptly, un- yas, nbsp, ﬁtr, istockphoto, featured, protectors, dapl, sheeple, pravda, vibration, masking, gleaned, minted, liege, playback, sidebar, screenshot, queer, realdon- rothschilds, anonymous, quaking, antifa, hawkins, abbey, prix, najib, redis- aldtrump, pinterest, eichen- neocon, neocons, violation, disable, turley, shoebat, tributed, getty, rebounds, wald, lgbt, extraterrestrial, aipac, illuminati, disclaimer, czars, im, glazov, overturns, derby, greets, afp, heathrow, lgbtq, attribution, android, click, zionist, cabal, user, cuck, chapters, leftwing, caption, rewritten tumblr, turmeric, pyramids, oligarchy cortez, slinging, bois, coconut, delicious pitchforks

Table 2: Most distinguishing words for each cluster, extracted using SAGE. Intuitively, no predominant topic distinguishes “widespread,” since it is the most diverse in terms of sources. The other clusters show distinctive topics, in line with the characterization of the sources composing them. “satirical/clickbait,” include satirical and clickbait sources, sports polarizing (e.g. impeach, protectors) and stereotypical conspiratorial words (e.g. pyramids, extraterrestrial). Cluster “conspiratorial news sources,” include conspiracy and junks science sources, adopts a conspiratorial lingo (e.g. illuminati, sheeple). Cluster “right- wing/fake,” include fake news and right-wing sources, uses hyper-conservative propaganda terms (e.g. pitchforks, [soy] bois). though recall is relatively lower for the other clusters (84%). In other words, quality and questionable information live in We reach high precision for all other clusters. On the one close quarters. hand we see that several articles are false positives for The present work sheds light on the relationship be- “widespread”, which demonstrates that alternative content tween users and news sources. Specifically, we find that cer- may be subtly similar to mainstream. On the other hand, tain types of misinformation are better than others at ex- achieving high precision has practical implications in au- plaining this relationship. Distinct Twitter audiences seem tomating the triaging of fringe content, because it allows to to follow factoid misinformation (conspiracy and junk sci- distinguish between satirical, conspiracy, and fake content. ence), misinformation with an ulterior motive (fake, bias), and misinformation with a frame of engagement (clickbait, Discussion satire). In particular, the drivers for extreme left- and right- Implications for characterizing the news sphere wing activism appears different. The left appears associated based on social media audience co-shares with satirical, socially versed and engaging misinformation. Recent research calls for a nuanced picture of the social Right-wing activism instead appears associated with emo- news sphere that goes beyond established partisan polariza- tionally negative, fake stories of threats by the political ad- tion (Starbird 2017; Guess et al. 2018; Druckman, Leven- versary. Our data-driven categorization of the four clusters dusky, and McLain 2018). Our results add to this line of of news communities is corroborated by the composition of work by detailing the social news sphere as a multifaceted the clusters, the external assessments of the sources within environment, and quantify the need for multiple assessments them, and the analyses of the news articles they publish. We to understand audience choices, instead of relying on stan- find that other assessments of news sources, e.g., labels of dalone external assessments. hate, (un)reliable, rumor, state and political, do not help an- We find that several dimensions of credibility and journal- alyze the social news sphere—be it because they are not as istic stance intertwine with political partisanship to shape the frequent in the data, not reliable as indicator, or not as in- social media readership. For instance, a cluster containing formative of sharing practices. These findings advance our entertaining misinformation like satire is also home to far- understanding of what assessments of news sources are rel- left activism. A second cluster which is deeply embedded evant to navigate the way social media audience shares them. in established political narratives publishes extreme right- Our results end on a positive note. We find that fringe wing propaganda and fake news. The correlation between content does not receive as many reactions as mainstream politicized audiences and misinformation outlets mirrors ex- content. This suggests that, although Twitter users may be isting research, that associates individuals on the political exposed to both quality and questionable information, they left-wing to engagement with positive content, whereas the do not grant as much attention to the latter. However, a more right-wing to threats (Dodd et al. 2012). Yet, misinformation conclusive analysis of the effect of fringe content should also on Twitter does not only target politicized audiences, or even take into account the overall reach, strength, polarity, and audiences that are distinct from those of credible sources. quality of those reactions. For example, a cluster focusing on conspiracism and “al- It is far from our intentions to put a moral or severity ternative truths” comprises of sources spanning the politi- judgment on different types of misinformation. Censoring cal spectrum. Similarly, a cluster of widespread news con- right-wing fake news or conspiracy theories as more morally tains all of the credible sources under study, but also many wrong or dangerous than clickbait frauds and hyper-liberal political misinformation sources, and even hateful content. hacktivism might do more damage than good. For example,

610 sentiment clickbait conspir. fake attempts to directly confront extreme political or conspira- intercept -2.1 -1.36 -2.92 torial views may prove counterproductive (Bail et al. 2018; sentiment compound -0.04 0.00 -0.31 Peter and Koch 2016; Lewandowsky et al. 2012). Instead, it neu -0.33 -0.38 -0.39 is crucial to understand the types of misinformation that the psycholinguistic clickbait conspir. fake audience engages with on social media. The present work is intercept -2.17 -1.58 -3.01 a step in this direction. This will, in turn, inform media lit- we 0.01 * 0.02 0.02 eracy campaigns to educate the audience about journalistic they 0.01 -0.01 0.01 norms and practices. pronouns you 0.00 -0.04 -0.04 i -0.01 0.02 -0.08 Social media audience as a costly signal of news shehe 0.03 -0.18 -0.02 source similarity ipron 0.03 -0.00 0.08 One merit of using audience as a similarity measure be- social friends 0.01 -0.10 -0.06 words family 0.05 -0.10 0.04 tween news sources is that the sources cannot easily manip- ulate their position in this space. For example, a junk science anger 0.10 0.17 0.24 source to position itself close to credible information sources emotion sad 0.02 -0.01 * -0.01 * words anx 0.04 0.00 0.05 would need to control a large fraction of shares of its own posemo 0.06 0.31 -0.09 content, and to also consistently share markers of credible content. These two procedures would be costly in terms of discrep -0.04 -0.06 -0.03 effort for the junk science source, and counterproductive in cause 0.13 0.25 -0.02 cognitive tentat 0.09 0.02 0.10 terms of audience targeting. In other words, audience simi- mech. excl 0.01 + 0.02 -0.01 * larity is a signal that is hard to falsify (Donath 2011). A word certain 0.23 0.23 0.21 of caution: hard does not mean impossible. However costly insight 0.19 0.32 0.11 a signal, individuals with enough resources and motivation hear -0.14 -0.41 -0.19 can, for example, acquire fake audience or fake engagement. perception see 0.12 0.07 0.09 feel -0.01 -0.10 -0.11 Implications for gatekeeping the social news sphere sexual 0.03 -0.14 -0.03 Online platforms are increasingly embracing source-level bilogical body 0.11 0.07 0.04 assessments for nudging their users towards high-quality processes health 0.05 0.10 -0.03 news while preserving access to a pluralistic social news ingest 0.11 0.05 -0.01 * sphere. For example, Facebook offers additional informa- future 0.04 0.03 0.03 14 temporal tion about sources appearing in the users’ feed . YouTube present 0.03 -0.11 0.18 15 focus labels videos that come from state-funded media outlets . past -0.06 -0.30 0.03 Similarly, Microsoft integrates NewsGuard in their mobile motion -0.01 -0.03 0.04 browser16 . Yet, it is easy to quickly create, rebrand, and shut relativity space -0.01 * 0.03 -0.11 down online news sources: all it takes is to edit a web page. time 0.02 -0.15 -0.17 Whereas the position of a news source in the audience simi- achiev -0.03 -0.24 -0.10 larity space is a costly signal, appearing as a news source in work -0.06 -0.09 0.03 the first place is a ludicrously cheap one. Efforts like Open- leisure -0.14 -0.15 -0.24 personal Sources continuously examine news sources that soon dis- death 0.01 0.09 0.01 * concerns appear. Indeed, several in our dataset were shut down within home 0.02 -0.05 0.02 relig -0.07 0.02 0.16 months from our data collection. Assessing news sources is money -0.15 0.00 + -0.31 time-consuming and requires the labor of experts, whose efforts must be directed towards critical cases. Our data-driven filler 0.08 0.02 0.07 approach can help social media platforms triage new infor- spoken nonflu -0.03 -0.00 + 0.00 categories swear 0.16 0.14 0.08 mation outlets in two possible ways. First, our clustering assent -0.00 -0.01 0.05 approach discussed in RQ1 can identify sources that share the same audience with a news outlet for which external as- Figure 3: Sentiment and psycholinguistic differences be- sessments is already available. Then, it can propagate those tween clusters. Cluster 1 being the baseline comparison clus- assessments to the news outlet, thus alleviating the labor- ter is not shown. All numbers should be interpreted in refer- intensive annotation of the ever growing space of informa- ence to cluster 1. Clusters 2, 3, and 4 express more emotional tion sources. Second, the classifier discussed in RQ3 can (less neu) and informal (spoken words) language than cluster classify the articles by the news outlet with high precision 1. Cluster 2, satirical and clickbait, uses more social words as either widespread content, or one of the different types of and references sexuality and health. Cluster 3, conspirato- 14 rial, shows the most self-concerns (I pronoun) and the least https://newsroom.fb.com/news/2018/04/inside-feed-article- context focus on the present. Cluster 4, right-wing fake news, refer- 15 ences values like family, home, and work, and expresses the https://money.cnn.com/2018/02/02/media/youtube-state- funded-media-label/ most negative sentiment (anger and anxiety). ⇒ p<.001, 16https://twitter.com/MWautier/status/1081346843487854593 ∗⇒p<.01, + ⇒ p<.05

611 misinformation: satirical/clickbait, conspiratorial/junk sci- lations between external assessments of news sources, and ence, and right-wing/fake news. Experts may use these in- their social media audience. Yet, qualitative research in on- dications to triage questionable content. line journalism and media literacy would be essential to understand whether those characteristics of the news sources Implications for developing novel assessments for are driving the users’ choice of sharing them. the social news sphere Whereas automation can help scale the process of assess- Conclusion ing news sources, the matter of communicating those as- In this research, we characterized the landscape of news sessments to the users requires careful consideration. In our sources based on their audience on Twitter. We showed that comparison of expert- versus audience-based clustering, we news sources aggregate in communities of shared audience, highlighted the necessity of using multiple existing assess- that uphold distinct factuality standards, political partisan- ments to characterize news sources accurately. However, ship, and journalistic norms. such redundant assessments complicate the clear communi- In particular, we uncovered four data-driven communication of a news source’s quality to their users. We believe ties of news sources: highly circulated news spanning the this challenge is best addressed through research. Commu- entire political spectrum; engaging misinformation, such as nication and social computing scholars have access to exclu- clickbait and satire; factoid misinformation, such as conspir- sive domain knowledge that is essential to synthesize novel atorial and junk science sources; and misinformation with assessments, so as to better describe current user practices an ulterior motive, such as far-right propaganda and fabri- in their choice of news sources. To this end, our approach cated news sources. We found distinguishing stylistic and allows researchers to combine multiple assessments of news topical markers that match with the characteristics of the sources, and to interpret them in the light of how users inter- news sources composing the clusters. For example, whereas act with them. sources in the widespread news cluster use more formal language, the conspiratorial cluster adopts an overconfident re- Limitations porting style. The difference between the clusters is measur- able: classifiers can automatically distinguish between news It is important for us to highlight some limitations in this articles coming from different clusters with high precision. work, which the readers should take into consideration when Thus, the Twitter audience delineates different segments of interpreting the results. One such limitation is arguably our the news media landscape, differing in both the characteris- choice of data. We focus on an expansive list of 639 English- tics of the news sources and of the news content that Twit- speaking news sources. However, this list is likely not rep- ter users engage with. Yet, our findings challenge the com- resentative of the landscape of news outlets on Twitter. To mon understanding of the news media landscape, exhibiting address this shortcoming, one would arguably require the complex interrelation between popularity, partisan lines, and complete data from the platform for an extended period of journalistic quality—with deep implications for gatekeeping time—in fact, simply accessing a subsample of the Twitter and triaging misinformation in social media. stream would result in neglecting smaller sources. Yet, even then one would be omitting large players of the wider infor- Acknowledgments mation ecosystem that includes television and talk radio. For We would like to thank the Social Computing lab at Virginia practical reasons, we focused on sources for which reliable Tech for comments on the early versions of the paper and external assessments were available. the anonymous reviewers for their feedback. This work is A second major limitation is that we rely on expert as- partially supported by NSF grant IIS-1755547. sessments by third-party initiatives. As a byproduct, not all sources have the same assessment coverage. For instance, References one source might be assessed from MediaBias but not from Allsides, and vice-a-versa. In our analyses, we address this Bail, C. A.; Argyle, L. P.; Brown, T. W.; Bumpus, J. P.; Chen, H.; Hunzaker, M. B. F.; Lee, J.; Mann, M.; Merhout, F.; and Volfovsky, issue by looking at different providers of assessments sepa- A. 2018. Exposure to opposing views on social media can increase rately. A different approach might involve in-house human political polarization. PNAS 115(37):9216–9221. annotators to harmonize assessments for all sources. How- Bakshy, E.; Messing, S.; and Adamic, L. A. 2015. Exposure ever, training annotators for journalistic norms is still a sub- to ideologically diverse news and opinion on Facebook. Science ject of research (Zhang et al. 2018). Furthermore, we rely 348(6239). on source-level assessments of news, although different arti- Barbera,´ P.; Jost, J. T.; Nagler, J.; Tucker, J. A.; and Bonneau, R. cles by one same source might have different qualities. One 2015. Tweeting From Left to Right: Is Online Political Commu- exemple is RT, which shares a combination of high-quality nication More Than an Echo Chamber? Psychological Science news reports and state propaganda (Starbird et al. 2018). 26(10). This choice is in line with our goal of studying the audience Bessi, A.; Coletto, M.; Davidescu, G. A.; Scala, A.; Caldarelli, G.; of news sources, as we focus on the aggregate characteristics and Quattrociocchi, W. 2015. Science vs Conspiracy: Collective of the news media sphere. Yet, practitioners should use cau- Narratives in the Age of Misinformation. PLOS ONE 10(2). tion when applying source annotations to individual articles. Conover, M. D.; Ratkiewicz, J.; Francisco, M.; Gonçalves, B.; Finally, a crucial limitation is that our results are largely Flammini, A.; and Menczer, F. 2011. Political polarization on correlational in nature. In particular, we find strong corre- twitter. In ICWSM.

612 Diakopoulos, N. 2015. Picking the NYT Picks: Editorial Criteria Lokot, T., and Diakopoulos, N. 2016. News Bots: Automating and Automation in the Curation of Online News Comments. #ISOJ news and information dissemination on Twitter. Digital Journalism 5(1). 4(6):682–699. Dodd, M. D.; Balzer, A.; Jacobs, C. M.; Gruszczynski, M. W.; Martin, T. 2017. community2vec: Vector representations of online Smith, K. B.; and Hibbing, J. R. 2012. The political left rolls communities encode semantic relationships. In NLP CSS Work- with the good and the political right confronts the bad: Connecting shop. physiology and cognition to preferences. Philos. Trans. Royal Soc. Marwick, A., and Lewis, R. 2017. Media Manipulation and Disin- B 367(1589). formation Online. Data & Society Research Institute 1–104. Donath, J. 2011. Signals , cues and meaning. Matsa, K. E., and Shearer, E. 2018. News use across social media Druckman, J. N.; Levendusky, M. S.; and McLain, A. 2018. No platforms 2018. In Pew Research Center. Need to Watch: How the Effects of Partisan Media Can Spread via Mukerjee, S.; Gonzalez-Bail´ on,´ S.; and Majo-V´ azquez,´ S. 2018. Interpersonal Discussions. Am. J. Pol Sci 62(1). Networks of Audience Overlap in the Consumption of Digital Eisenstein, J.; Ahmed, A.; and Xing, E. P. 2011. Sparse additive News. Journal of Communication 68(1):26–50. generative models of text. In ICML. Oliver, J. E., and Wood, T. J. 2014. Conspiracy theories and the Elejalde, E.; Ferres, L.; and Herder, E. 2017. The Nature of Real paranoid style(s) of mass opinion. Am J Pol Sci 58(4). and Perceived Bias in Chilean Media. HT. Peter, C., and Koch, T. 2016. When Debunking Scientific Myths Ferrara, E.; Varol, O.; Davis, C.; Menczer, F.; and Flammini, A. Fails (and When It Does Not). Science Communication 38(1):3–25. 2014. The Rise of Social Bots. arXiv preprint arXiv:1407.5225 Sauri, R. 2008. A factuality profiler for eventualities in text. (grant 220020274):1–11. Soni, S.; Mitra, T.; Gilbert, E.; and Eisenstein, J. 2014. Modeling Garimella, K.; Morales, G. D. F.; Gionis, A.; and Mathioudakis, Factuality Judgments in Social Media Text. In ACL. M. 2018. Political Discourse on Social Media: Echo Chambers, Starbird, K.; Arif, A.; Wilson, T.; Koevering, K. V.; Yefimova, K.; Gatekeepers, and the Price of Bipartisanship. WWW. and Scarnecchia, D. 2018. Ecosystem or echo-system? exploring Golbeck, J.; Everett, J. B.; Falak, W.; Gieringer, C.; Graney, J.; content sharing across alternative media domains. In ICWSM. Hoffman, K. M.; Huth, L.; Ma, Z.; Jha, M.; Khan, M.; Kori, V.; Starbird, K. 2017. Examining the Alternative Media Ecosystem Mauriello, M.; Lewis, E.; Mirano, G.; Mohn IV, W. T.; Mussenden, through the Production of Alternative Narratives of Mass Shooting S.; Nelson, T. M.; Mcwillie, S.; Pant, A.; Shetye, P.; Shrestha, Events on Twitter. In ICWSM. R.; Steinheimer, A.; Auxier, B.; Subramanian, A.; Visnansky, G.; Sundar, S. S. 1999. Exploring receivers’ criteria for perception of Bhanushali, K. H.; Bonk, C.; Bouzaghrane, M. A.; Buntain, C.; print and online news. JMCQ 76(2). Chanduka, R.; and Cheakalos, P. 2018. Fake News vs Satire. In Proceedings of the 10th ACM Conference on Web Science - WebSci Tausczik, Y. R., and Pennebaker, J. W. 2010. The Psychological ’18, 17–21. New York, New York, USA: ACM Press. Meaning of Words: LIWC and Computerized Text Analysis Meth- ods. J. Lang. Soc. Psychol. 29(1). Grinberg, N.; Joseph, K.; Friedland, L.; Swire-Thompson, B.; and Lazer, D. 2019. Fake news on twitter during the 2016 U.S. Presi- Thorson, E. 2016. Belief Echoes: The Persistent Effects of Cor- dential election. Science 378. rected Misinformation. Political Communication 33(3):460–480. Guess, A.; Nyhan, B.; Lyons, B.; and Reifler, J. 2018. Avoiding Webster, J. G., and Ksiazek, T. B. 2012. The Dynamics of Audi- the Echo Chamber about Echo Chambers. ence Fragmentation: Public Attention in an Age of Digital Media. Journal of Communication 62(1):39–56. Hartung, F.; Burke, M.; Hagoort, P.; and Willems, R. M. 2016. Zhang, A. X.; Robbins, M.; Bice, E.; Hawke, S.; Karger, D.; Mina, Taking perspective: Personal pronouns affect experiential aspects A. X.; Ranganathan, A.; Metz, S. E.; Appling, S.; Sehat, C. M.; of literary reading. PLoS ONE 11(5). Gilmore, N.; Adams, N. B.; Vincent, E.; and Lee, J. 2018. A Hayes, A. S.; Singer, J. B.; and Ceppos, J. 2007. Shifting Roles, Structured Response to Misinformation: Defining and Annotating Enduring Values: The Credible Journalist in a Digital Age. Journal Credibility Indicators in News Articles. In WWW. of Mass Media Ethics 22(4):262–279. Horne, B. D., and Adali, S. 2017. This Just In: Fake News Packs a Lot in Title, Uses Simpler, Repetitive Content in Text Body, More Similar to Satire than Real News. Horne, B. D., and Adali, S. 2018. An Exploration of Verbatim Content Republishing by News Producers. NECO. Hutto, C. J., and Gilbert, E. 2014. Vader: A parsimonious rule- based model for sentiment analysis of social media text. In ICWSM. Kulshrestha, J.; Zafar, M. B.; Noboa, L. E.; Gummadi, K. P.; and Ghosh, S. 2015. Characterizing Information Diets of Social Media Users. In ICWSM. Kumar, S.; West, R.; and Leskovec, J. 2016. Disinformation on the Web: Impact, Characteristics, and Detection of Wikipedia Hoaxes. World Wide Web Conference Committee 591–602. Lewandowsky, S.; Ecker, U. K. H.; Seifert, C. M.; Schwarz, N.; and Cook, J. 2012. Misinformation and Its Correction: Continued Influence and Successful Debiasing. Psychological Science in the Public Interest 13(3):106–131.

613