Arxiv:1910.09982V1 [Cs.CL] 20 Oct 2019 Tardaguila´ Et Al., 2018)
Total Page:16
File Type:pdf, Size:1020Kb
Findings of the NLP4IF-2019 Shared Task on Fine-Grained Propaganda Detection Giovanni Da San Martino1 Alberto Barron-Cede´ no˜ 2 Preslav Nakov1 1 Qatar Computing Research Institute, HBKU, Qatar 2 Universita` di Bologna, Forl`ı, Italy fgmartino, [email protected] [email protected] Abstract With this in mind, we organised the shared task on fine-grained propaganda detection at the We present the shared task on Fine-Grained NLP4IF@EMNLP-IJCNLP 2019 workshop. The Propaganda Detection, which was organized task is based on a corpus of news articles anno- as part of the NLP4IF workshop at EMNLP- tated with an inventory of 18 propagandist tech- IJCNLP 2019. There were two subtasks. FLC is a fragment-level task that asks for the iden- niques at the fragment level. We hope that the tification of propagandist text fragments in a corpus would raise interest outside of the commu- news article and also for the prediction of the nity of researchers studying propaganda. For ex- specific propaganda technique used in each ample, the techniques related to fallacies and the such fragment (18-way classification task). ones relying on emotions might provide a novel SLC is a sentence-level binary classification setting for researchers interested in Argumentation task asking to detect the sentences that con- and Sentiment Analysis. tain propaganda. A total of 12 teams submit- ted systems for the FLC task, 25 teams did so for the SLC task, and 14 teams eventu- 2 Related Work ally submitted a system description paper. For both subtasks, most systems managed to beat Propaganda has been tackled mostly at the arti- the baseline by a sizable margin. The leader- cle level. Rashkin et al.(2017) created a corpus board and the data from the competition are of news articles labelled as propaganda, trusted, available at http://propaganda.qcri. hoax, or satire. Barron-Cede´ no˜ et al.(2019) ex- org/nlp4if-shared-task/. perimented with a binarized version of that cor- 1 Introduction pus: propaganda vs. the other three categories. Barron-Cedeno´ et al.(2019) annotated a large bi- Propaganda aims at influencing people’s mindset nary corpus of propagandist vs. non-propagandist with the purpose of advancing a specific agenda. articles and proposed a feature-based system for In the Internet era, thanks to the mechanism discriminating between them. In all these cases, of sharing in social networks, propaganda cam- the labels were obtained using distant supervision, paigns have the potential of reaching very large assuming that all articles from a given news out- audiences (Glowacki et al., 2018; Muller, 2018; let share the label of that outlet, which inevitably arXiv:1910.09982v1 [cs.CL] 20 Oct 2019 Tardaguila´ et al., 2018). introduces noise (Horne et al., 2018). Propagandist news articles use specific A related field is that of computational argu- techniques to convey their message, such as mentation which, among others, deals with some whataboutism, red Herring, and name calling, logical fallacies related to propaganda. Habernal among many others (cf. Section3). Whereas et al.(2018b) presented a corpus of Web forum proving intent is not easy, we can analyse the discussions with instances of ad hominem fallacy. language of a claim/article and look for the use Habernal et al.(2017, 2018a) introduced Argo- of specific propaganda techniques. Going at this tario, a game to educate people to recognize and fine-grained level can yield more reliable systems create fallacies, a by-product of which is a corpus and it also makes it possible to explain to the user with 1:3k arguments annotated with five fallacies why an article was judged as propagandist by an such as ad hominem, red herring and irrelevant automatic system. authority, which directly relate to propaganda. Unlike (Habernal et al., 2017, 2018a,b), our cor- 8. Causal oversimplification. Assuming one pus uses 18 techniques annotated on the same set cause when there are multiple causes behind an of news articles. Moreover, our annotations aim issue. We include scapegoating as well: the trans- at identifying the minimal fragments related to a fer of the blame to one person or group of people technique instead of flagging entire arguments. without investigating the complexities of an issue. The most relevant related work is our own, which is published in parallel to this paper at 9. Slogans. A brief and striking phrase that may EMNLP-IJCNLP 2019 (Da San Martino et al., include labeling and stereotyping. Slogans tend to 2019) and describes a corpus that is a subset of act as emotional appeals (Dan, 2015). the one used for this shared task. 10. Appeal to authority. Stating that a claim 3 Propaganda Techniques is true simply because a valid authority/expert on the issue supports it, without any other supporting Propaganda uses psychological and rhetorical evidence (Goodwin, 2011). We include the special techniques to achieve its objective. Such tech- case where the reference is not an authority/expert, niques include the use of logical fallacies and ap- although it is referred to as testimonial in the liter- peal to emotions. For the shared task, we use 18 ature (Jowett and O’Donnell, 2012, p. 237). techniques that can be found in news articles and can be judged intrinsically, without the need to 11. Black-and-white fallacy, dictatorship. retrieve supporting information from external re- Presenting two alternative options as the only pos- sources. We refer the reader to (Da San Martino sibilities, when in fact more possibilities exist et al., 2019) for more details on the propaganda (Torok, 2015). As an extreme case, telling the techniques; below we report the list of techniques: audience exactly what actions to take, eliminating 1. Loaded language. Using words/phrases with any other possible choice (dictatorship). strong emotional implications (positive or nega- 12. Thought-terminating cliche´. Words or tive) to influence an audience (Weston, 2018, p. 6). phrases that discourage critical thought and mean- 2. Name calling or labeling. Labeling the ob- ingful discussion about a given topic. They are ject of the propaganda as something the target au- typically short and generic sentences that offer dience fears, hates, finds undesirable or otherwise seemingly simple answers to complex questions loves or praises (Miller, 1939). or that distract attention away from other lines of thought (Hunter, 2015, p. 78). 3. Repetition. Repeating the same message over and over again, so that the audience will eventually 13. Whataboutism. Discredit an opponent’s accept it (Torok, 2015; Miller, 1939). position by charging them with hypocrisy without 4. Exaggeration or minimization. Either rep- directly disproving their argument (Richter, 2017). resenting something in an excessive manner: mak- ing things larger, better, worse, or making some- 14. Reductio ad Hitlerum. Persuading an au- thing seem less important or smaller than it ac- dience to disapprove an action or idea by suggest- tually is (Jowett and O’Donnell, 2012, p. 303), ing that the idea is popular with groups hated in e.g., saying that an insult was just a joke. contempt by the target audience. It can refer to any person or concept with a negative connota- 5. Doubt. Questioning the credibility of some- tion (Teninbaum, 2009). one or something. 15. Red herring. Introducing irrelevant mate- 6. Appeal to fear/prejudice. Seeking to build rial to the issue being discussed, so that every- support for an idea by instilling anxiety and/or one’s attention is diverted away from the points panic in the population towards an alternative, made (Weston, 2018, p. 78). Those subjected to a possibly based on preconceived judgments. red herring argument are led away from the issue 7. Flag-waving. Playing on strong national feel- that had been the focus of the discussion and urged ing (or with respect to a group, e.g., race, gender, to follow an observation or claim that may be as- political preference) to justify or promote an ac- sociated with the original claim, but is not highly tion or idea (Hobbs and Mcgee, 2008). relevant to the issue in dispute (Teninbaum, 2009). Figure 1: The beginning of an article with annotations. 16. Bandwagon. Attempting to persuade the 5 Data target audience to join in and take the course of action because “everyone else is taking the same The input for both tasks consists of news articles action” (Hobbs and Mcgee, 2008). in free-text format, collected from 36 propagandist and 12 non-propagandist news outlets1 and then 17. Obfuscation, intentional vagueness, con- annotated by professional annotators. More de- fusion. Using deliberately unclear words, to let tails about the data collection and the annotation, the audience have its own interpretation (Supra- as well as statistics about the corpus can be found bandari, 2007; Weston, 2018, p. 8). For instance, in (Da San Martino et al., 2019), where an earlier when an unclear phrase with multiple possible version of the corpus is described, which includes meanings is used within the argument and, there- 450 news articles. We further annotated 47 addi- fore, it does not really support the conclusion. tional articles for the purpose of the shared task using the same protocol and the same annotators. 18. Straw man. When an opponent’s proposi- The training, the development, and the test par- tion is substituted with a similar one which is then titions of the corpus used for the shared task con- refuted in place of the original (Walton, 1996). sist of 350, 61, and 86 articles and of 16,965, 2,235, and 3,526 sentences, respectively. Fig- 4 Tasks ure1 shows an annotated example, which con- tains several propaganda techniques. For ex- The shared task features two subtasks: ample, the fragment babies on line 1 is an in- stance of both Name Calling and Labeling.