Highly Partisan and Blatantly Wrong: Analyzing News Publishers’ Critiques of Google’s Reviewed Claims Emma Lurie Eni Mustafaraj School of Information Department of Computer Science University of California, Berkeley Wellesley College emma [email protected] [email protected] Abstract Google’s reviewed claims feature was an early attempt to incorporate additional credibility signals from fact-checking onto the search results page. The feature, which appeared when users searched for the name of a subset of news publishers, was criticized by dozens of publishers for its errors and alleged anti- conservative bias. By conducting an audit of news publisher search results and focusing on the critiques of publishers, we find that there is a lack of consensus between fact-checking Figure 1: The SERP of The Daily Caller with the re- ecosystem stakeholders that may be important viewed claims feature circled as it appeared in January to address in future iterations of public facing 2018. Reviewed Claims component contained fact- fact-checking tools. In particular, we find that checks of articles produced by that news publisher. a lack of transparency coupled with a lack of consensus on what makes a fact-check relevant to a news article led to the breakdown of re- to explore both conceptual misunderstandings be- viewed claims. tween stakeholders as well as inconsistencies aris- ing from the technical implementation of the fea- 1 Introduction ture. In November 2017, as part of an expanded effort The online fact-checking ecosystem is a com- to provide users with context about news publish- plex sociotechnical system that includes technol- ers, Google released reviewed claims (see Figure 1 ogy platforms, news publishers, fact-checking or- and 2). The reviewed claims feature displayed ganizations, and online information seekers. As third-party fact-checks about content produced by platforms try to limit harmful misinformation on a subset of news publishers. This feature was pub- their platform and fact-checkers work to increase licized as a meaningful aid to information seekers the reach of their fact-check articles, there have to identify misinformation.3 This is one of several been a number of collaborations, some of which ways that Google tried to highlight fact-checks in 1 are ongoing, and others that can be considered search results, in accordance with best practices in 2 failed experiments. We focus on one “failed” ex- media literacy (Wineburg and McGrew, 2017). periment, that of Google’s reviewed claims. Fo- However, several news publishers complained cusing on the breakdown (Mulligan and Griffin, about this feature in January 2018. They al- 2018; Akrich, 1992; Bucher, 2017) or failure of leged that Google was displaying partisan bias and a specific feature like reviewed claims enables us claimed that several of the fact-checks listed in 1https://www.blog.google/topics/journalism- their reviewed claims components were not rele- news/building-trust-online-partnering-international-fact- vant to the associated content. Soon after, Google checking-network/ 2https://medium.com/facebook-design/ 3https://www.blog.google/products/ designing-against-misinformation- search/learn-more-about-publishers- e5846b3aa1e2 google/ 64 Proceedings of the 2020 Truth and Trust Online (TTO 2020), pages 64–72, Virtual, October 16-17, 2020. scope of the the RC feature and 2) a more fo- cused and qualitative thematic analysis of 45 on- line news articles about the reviewed claims fea- ture. Our major takeaway from this case study is that the implementation of algorithmic solutions in support of fact-checking is a complex process that Figure 2: The Reviewed Claim feature attributes this requires and deserves the kind of transparency that claim to The Daily Caller. However, the PolitiFact fact- is so valued by the fact-checking community. Our check does not mention Daily Caller. This is an exam- findings contribute by expanding the conversation ple of an algorithmically assigned fact-check. around the efficacy of fact-checking interventions in the informational ecosystem. removed reviewed claims from the search engine results page (SERP), citing “bugs” in the feature’s 2 Auditing reviewed claims implementation.4 To situate publishers’ claims about the reviewed Reviewed claims (RC) is a worthwhile case claims feature, we report our findings from a Jan- study for several reasons. First, RC is an ex- uary 2018 audit of news publisher SERPs. In sum- ample of a major platform using fact-checks as mary, we identify 59 publisher names that dis- a source level credibility signal (i.e. displaying played a RC tab when their names were searched. fact-checks for publishers not individual articles) These RC tabs were collected incidentally in a for information seekers. Researchers (Wineb- larger audit of the state of partisan and unreliable urg et al., 2016; Zhang et al., 2018) have called news publisher SERPs. While our data collection for greater context and information cues for in- was not designed to be an exhaustive list of pub- formation seekers to evaluate the the credibility lishers with a RC tab, it does provide the most of surfaced content. Second, purportedly with- complete picture of the reviewed claims feature out informing fact-checking organizations, news to date. In Table 1, we report the 59 publish- publishers, or information seekers, Google lever- ers with the reviewed claims tab, accompanied by aged automated fact-checking techniques. Auto- their bias/accuracy label from BuzzFeed News or mated fact-checking, which uses algorithms to in- Media Bias Fact-check. While we find that more crease the scale and reach of fact-checks, raises publishers on the “right” had RC tabs, there were additional questions about the values, such as a 5 more “left” sites than what was alleged by conser- commitment to transparency that undergird fact- vative publishers. checking. Third, while publishers often raised Our manual examination of the 171 unique fact- poorly supported claims of anti-conservative bias, checks attributed to the 59 publishers leads us to publishers also raised substantive concerns about conclude that Google used an algorithm to match particular stories that were surfaced in the re- existing fact-checks (that did not name the pub- viewed claims panel. These are of particular inter- lisher) to articles by that publisher. This prac- est in this paper as they offer the opportunity for tice is called claim matching and is an automated various stakeholders to build deeper understand- fact-checking technique that matches articles or ing of the challenges that will confront future au- statements (i.e. claims) to existing fact-checks. tomated fact-checking features. The following re- The use of this method is significant because search question guides our work: what examples Google did not disclose the use of automated fact- and issues did publishers raise to contest the re- checking in its documentation7 8 about the feature. viewed claims feature? We explore this research question by 1) a data 2.1 Methods: Auditing RC collection effort6 that measured the prevalence and In January 2018, as part of an audit of news pub- 4https://www.poynter.org/fact-checking/ lisher SERPs, we discovered that 59 publishers 2018/blame-bugs-not-partisanship-for- google-wrongly-appending-a-fact-check- claims to-the-daily-caller/ 7https://support.google.com/websearch/ 5https://www.ifcncodeofprinciples.poynter.org/answer/7568277?hl=en know-more/the-commitments-of-the-code- 8https://www.blog.google/products/ of-principles search/learn-more-about-publishers- 6https://github.com/emmalurie/reviewed- google/ 65 List of websites to which Google assigned a reviewed claims tab Bias or Accuracy (ordered by Alexa Rank) Breitbart, The Daily Caller, The Daily Wire, WND, The Gateway Pundit, The Federalist, Free Republic, The Conservative Tree House, OAN Network, Big League Politics, The Political Insider, Frontpage Magazine, American Greatness, American Renaissance, Bearing Arms, Red State Watcher, Right Truthfeed, 100 Percent Fed Up, Freedom Outpost, Commentary Magazine, The Millennium Report, VDARE, Sparta Report, En Volve, Conservative Fighters, Silence is Consent, America’s Freedom Fighters, Freedom Daily, American News, American Conservative Herald, The New York Evening. Upworthy, Palmer Report, Democratic Underground, Counterpunch, Rightwingwatch, Bipartisan Report, True Activist, OpEd News, American Left Herald Tribune, Occupy Democrats, Egberto Willies, If You Only News, American News X, Resistance Report Zero Hedge, Before It’s News, Natural News, Above Top Secret, Collective Conspiracy / Evolution, Your News Wire, Investment Watch Blog, Awareness Act, Activist Pseudoscience / Post, Renegade Tribune, The Common Sense Show, Fellowship of the minds, Fake news Intellihub, 21st Century Wire Table 1: The list of all 59 websites with a reviewed claims (RC) tab in our January 2018 dataset, grouped by political bias or factual accuracy. The labels come from BuzzFeed or Media Bias/Fact Check. contained a reviewed claims component in their might be under counting the news publishers with SERP. These 1,150 publisher SERPs were sourced a knowledge panel. For established, well-known from prominent aggregated lists in the research publishers, Google correctly inferred the news and media literacy community.9 10 11 12 13 The publisher website. combined list of websites was designed to be a Each RC tab contained links to the fact-check comprehensive survey of active partisan or unre- and original article (see Figure 2). The first author liable U.S. publishers in late 2017. In total, the manually reviewed the 171 unique fact-checks that aggregated dataset amounts to unique 1,150 web- appeared on the 59 RC tabs to extract: 1) the site domain names. ClaimReview markup and 2) the mentions of all We use Selenium, a web browser automation news publishers and URLs identified as claimants tool, to issue queries to Google search. The in the fact-check article. queries sent to the browser were the domain names without the extension (e.g. “dailycaller” from 2.2 Results: Auditing RC “dailycaller.com”).