DeClarE: Debunking Fake News and False Claims using Evidence-Aware Deep Learning Kashyap Popat1, Subhabrata Mukherjee2, Andrew Yates1, Gerhard Weikum1 1Max Planck Institute for Informatics, Saarbrucken,¨ Germany 2Amazon Inc., Seattle, USA fkpopat,ayates,[email protected], [email protected] Abstract State of the Art and Limitations: Prior work on “truth discovery” (see Li et al.(2016) for survey) 1 Misinformation such as fake news is one of largely focused on structured facts, typically in the big challenges of our society. Research on the form of subject-predicate-object triples, or on automated fact-checking has proposed meth- social media platforms like Twitter, Sina Weibo, ods based on supervised learning, but these etc. Recently, methods have been proposed to as- approaches do not consider external evidence apart from labeled training instances. Recent sess the credibility of claims in natural language approaches counter this deficit by considering form (Popat et al., 2017; Rashkin et al., 2017; external sources related to a claim. However, Wang, 2017), such as news headlines, quotes from these methods require substantial feature mod- speeches, blog posts, etc. eling and rich lexicons. This paper overcomes The methods geared for general text input ad- these limitations of prior work with an end-to- dress the problem in different ways. On the one end model for evidence-aware credibility as- hand, methods like Rashkin et al.(2017); Wang sessment of arbitrary textual claims, without any human intervention. It presents a neural (2017) train neural networks on labeled claims network model that judiciously aggregates sig- from sites like PolitiFact.com, providing credibil- nals from external evidence articles, the lan- ity assessments without any explicit feature mod- guage of these articles and the trustworthiness eling. However, they use only the text of ques- of their sources. It also derives informative tionable claims and no external evidence or inter- features for generating user-comprehensible actions that provide limited context for credibil- explanations that makes the neural network ity analysis. These approaches also do not offer predictions transparent to the end-user. Exper- any explanation of their verdicts. On the other iments with four datasets and ablation studies show the strength of our method. hand, Popat et al.(2017) considers external evi- dence in the form of other articles (retrieved from 1 Introduction the Web) that confirm or refute a claim, and jointly assesses the language style (using subjectivity lex- Motivation: Modern media (e.g., news feeds, mi- icons), the trustworthiness of the sources, and the croblogs, etc.) exhibit an increasing fraction of credibility of the claim. This is achieved via a misleading and manipulative content, from ques- pipeline of supervised classifiers. On the upside, tionable claims and “alternative facts” to com- this method generates user-interpretable explana- pletely faked news. The media landscape is be- tions by pointing to informative snippets of evi- coming a twilight zone and battleground. This so- dence articles. On the downside, it requires sub- cietal challenge has led to the rise of fact-checking stantial feature modeling and rich lexicons to de- and debunking websites, such as Snopes.com tect bias and subjectivity in the language style. and PolitiFact.com, where people research claims, Approach and Contribution: To overcome the manually assess their credibility, and present their limitations of the prior works, we present De- verdict along with evidence (e.g., background ar- ClarE2, an end-to-end neural network model for ticles, quotations, etc.). However, this manual ver- assessing and explaining the credibility of arbi- ification is time-consuming. To keep up with the 1As fully objective and unarguable truth is often elusive scale and speed at which misinformation spreads, or ill-defined, we use the term credibility rather than “truth”. we need tools to automate this debunking process. 2Debunking Claims with Interpretable Evidence 22 Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 22–32 Brussels, Belgium, October 31 - November 4, 2018. c 2018 Association for Computational Linguistics trary claims in natural-language text form. Our 2 End-to-end Framework for Credibility approach combines the best of both families of Analysis prior methods. Similar to Popat et al.(2017), De- ClarE incorporates external evidence or counter- Consider a set of N claims hCni from the respec- evidence from the Web as well as signals from the tive origins/sources hCSni, where n 2 [1;N]. language style and the trustworthiness of the un- Each claim Cn is reported by a set of M arti- derlying sources. However, our method does not cles hAm;ni along with their respective sources require any feature engineering, lexicons, or other hASm;ni, where m 2 [1;M]. Each corresponding manual intervention. Rashkin et al.(2017); Wang tuple of claim and its origin, reporting articles and (2017) also develop an end-to-end model, but De- article sources – hCn;CSn;Am;n; ASm;ni forms ClarE goes far beyond in terms of considering ex- a training instance in our setting, along with the ternal evidence and joint interactions between sev- credibility label of the claim used as ground-truth eral factors, and also in its ability to generate user- during network training. Figure1 gives a pictorial interpretable explanations in addition to highly overview of our model. In the following sections, accurate assessments. For example, given the we provide a detailed description of our approach. “the gun epidemic natural-language input claim 2.1 Input Representations is the leading cause of death of young African- American men, more than the next nine causes put The input claim Cn of length l is represented as d together” by Hillary Clinton, DeClarE draws on [c1; c2; :::; cl] where cl 2 < is the d-dimensional evidence from the Web to arrive at its verdict cred- word embedding of the l-th word in the input ible, and returns annotated snippets like the one claim. The source/origin of the claim CSn is rep- in Table6 as explanation. These snippets, which resented by a ds-dimensional embedding vector ds contain evidence in the form of statistics and as- csn 2 < . sertions, are automatically extracted from web ar- A reporting article Am;n consisting of k to- ticles from sources of varying credibility. kens is represented by [am;n;1; am;n;2; :::; am;n;k], d Given an input claim, DeClarE searches for web where am;n;k 2 < is the d-dimensional word articles related to the claim. It considers the con- embedding vector for the k-th word in the report- text of the claim via word embeddings and the ing article Am;n. The claim and article word em- (language of) web articles captured via a bidirec- beddings have shared parameters. The source of tional LSTM (biLSTM), while using an attention the reporting article ASm;n is represented as a ds- ds mechanism to focus on parts of the articles accord- dimensional vector, asm;n 2 < . For the sake ing to their relevance to the claim. DeClarE then of brevity, we drop the notation subscripts n and aggregates all the information about claim source, m in the following sections by considering only a web article contexts, attention weights, and trust- single training instance – the input claim Cn from worthiness of the underlying sources to assess the source CSn, the corresponding article Am;n and claim. It also derives informative features for in- its sources ASm;n given by: hC; CS; A; ASi. terpretability, like source embeddings that capture 2.2 Article Representation trustworthiness and salient words captured via at- tention. Key contributions of this paper are: To create a representation of an article, which may capture task-specific features such as whether it • Model: An end-to-end neural network model contains objective language, we use a bidirectional which automatically assesses the credibility Long Short-Term Memory (LSTM) network as of natural-language claims, without any hand- proposed by Graves et al.(2005). A basic LSTM crafted features or lexicons. cell consists of various gates to control the flow of • Interpretability: An attention mechanism in information through timesteps in a sequence, mak- our model that generates user-comprehensible ing LSTMs suitable for capturing long and short explanations, making credibility verdicts range dependencies in text that may be difficult transparent and interpretable. to capture with standard recurrent neural networks • Experiments: Extensive experiments on four (RNNs). Given an input word embedding of to- datasets and ablation studies, demonstrating kens haki, an LSTM cell performs various non- effectiveness of our method over state-of-the- linear transformations to generate a hidden vector art baselines. state hk for each token at each timestep k. 23 Claim Word Claim Source Embeddings Embedding Concatenate Dense Layer avg Attention Weights X Softmax X avg Credibility X Score Article Word Inner Product Softmax/ Embeddings Linear Bidirectional Dense Dense LSTM Layer Layer Article Source Concatenate Embedding Features Figure 1: Framework for credibility assessment. Upper part of the pipeline combines the article and claim embeddings to get the claim specific attention weights. Lower part of the pipeline captures the article representation through biLSTM. Attention focused article representation along with the source embeddings are passed through dense layers to predict the credibility score of the claim. We use bidirectional LSTMs in place of stan- words therein: dard LSTMs. Bidirectional LSTMs capture both 1 X the previous timesteps (past features) and the fu- c¯ = cl l ture timesteps (future features) via forward and l backward states respectively. Correspondingly, We combine this overall representation of the there are two hidden states that capture past and claim with each article term: future information that are concatenated to form −! − the final output as: hk = [hk; hk].