Breaking Down the Invisible Wall of Informal Fallacies in Online Discussions Saumya Yashmohini Sahai Oana Balalau The Ohio State University, USA Inria, Institut Polytechnique de Paris, France [email protected] [email protected] Roxana Horincar Thales Research & Technology, France [email protected] Abstract discourse. For example, The New York Times la- beled the tweets of Donald Trump between 2015 People debate on a variety of topics on online and 2020 and found thousands of insults addressed platforms such as Reddit, or Facebook. De- to his adversaries. If made in an argument, an in- bates can be lengthy, with users exchanging sult is an ad hominem fallacy: an attack on the a wealth of information and opinions. How- ever, conversations do not always go smoothly, opponent rather than on their argument. In pri- and users sometimes engage in unsound argu- vate conversations, other types of fallacies might mentation techniques to prove a claim. These be more prevalent, for example, appeal to tradition techniques are called fallacies. Fallacies are or appeal to nature. Appeal to tradition dismisses persuasive arguments that provide insufficient calls to improve gender equality by stating that or incorrect evidence to support the claim. In “women have always occupied this place in soci- this paper, we study the most frequent falla- ety”. Appeal to nature is often used to ignore calls cies on Reddit, and we present them using to be inclusive of the LGBTQ+ community by stat- the pragma-dialectical theory of argumenta- tion. We construct a new annotated dataset of ing “gender is binary”. The underlying premises fallacies, using user comments containing fal- of such arguments are “traditions are correct” and lacy mentions as noisy labels, and cleaning the “what occurs in nature is good”. data via crowdsourcing. Finally, we study the Creating a dataset of fallacious arguments is task of classifying fallacies using neural mod- difficult, given that there are over 100 types of els. We find that generally the models perform fallacious arguments (Scalambrino, 2018). There better in the presence of conversational con- have been several attempts to create comprehensive text.We have released the data and the code at github.com/sahaisaumya/informal_ datasets: Habernal et al.(2017) proposed a game fallacies. in which players add fallacies in the hope of foul- ing other participants, in Habernal et al.(2018a) 1 Introduction ad hominem fallacies are found using a subred- dit’s rule violations, while in Da San Martino et al. Argumentation plays a critical part in our lives as (2019) fallacies are annotated together with other it helps us make decisions and reason about the propaganda techniques in news articles. However, world around us. Studies (Sanders et al., 1994) our work is the first to propose a viable solution have shown that learning how to argue increases the for finding fallacious arguments belonging to many ability to identify weak arguments and decreases different fallacy types. the tendency to use verbal aggressiveness. Fallacies In this work, we study fallacies in public discus- are weak arguments that seem convincing, however, sions on online forums. Our salient contributions their evidence does not prove or disprove the argu- are: i) we align informal fallacies mentioned on ment’s conclusion. Fallacies are usually divided Reddit within the pragma-dialectic theory of argu- into formal and informal, where the former can mentation (van Eemeren and Grootendorst, 1995); be easily described using logical representations, ii) we design a methodology for mining and label- while for the latter, an analysis of the content is ing easily fallacies in online discussions; iii) we more appropriate. Fallacies are prevalent in public construct a large and balanced dataset of fallacious Part of this work was done while the first author was an arguments; iv) finally, we evaluate several neural intern at Inria, France. models on the task of predicting fallacious argu- 644 Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 644–657 August 1–6, 2021. ©2021 Association for Computational Linguistics ments, and we find that taking into consideration Douglas Walton (Walton, 2005). A scheme con- additional conversational context is important for sists of a conclusion, a set of premises, and a set this task. of critical questions. The critical questions should be answered in order to prove that the premises 2 Background support the conclusion, hence the argument is not a fallacy. For example, the scheme for an argu- 2.1 Fallacies in Argumentation Theory ment from expert opinion (Walton, 2005) has the Humans use argumentation when they evaluate the premises E is an expert in domain D, E asserts validity of new ideas, or they want to solve a dif- that A is known to be true, A is within D and the ference of opinion. An argument contains: i) a conclusion therefore, A may plausibly be taken to proposition called claim, conclusion or standpoint, be true. Some critical questions for this scheme to be validated; ii) the premises called also evi- are: i) Trustworthiness: Is E personally reliable as dence, which are the backing propositions; iii) an a source? ii) Backup Evidence: Is E’s assertion inference relation between the evidence and con- based on evidence? Argumentation schemes have clusion that validates or disproves the conclusion. two main drawbacks: first, for each new fallacy, a A fallacy is a flawed argument, where the inference new scheme should exist or be defined; and sec- relation or the premises are incorrect. Fallacies are ond, in the context of labeling an existing argument, generally divided into formal and informal falla- many of the critical questions might be unanswer- cies. Formal fallacies are arguments that can be able as none of the parties discussed them. easily represented as invalid logical formulas, such as denying the antecedent, which is a wrong appli- 2.2 Related Work cation of modus tollens. Although many informal fallacies can be also represented as invalid argu- An initial effort for creating an extensive dataset of ments, informal fallacies are easier to describe and fallacies was made in Habernal et al.(2017). The understand without resorting to logical representa- authors created a platform for educative games, tions (Hansen, 2020). where players learn how to become better debaters. In this work, we follow the pragma dialectic New fallacies are added to the platform by play- theory of argumentation. The theory developed ers that try to earn points by fouling other partici- by van Eemeren and Grootendorst(1995) views pants with invalid arguments. A follow-up on this argumentation as a complex speech act. The dialec- work (Habernal et al., 2018a) mentioned a dataset tical aspect is represented by two parties who try of only around 300 arguments created via the plat- to resolve a difference of opinion by engaging in a form, thus showing the need of finding other meth- discussion, each party making a move towards res- ods for creating larger datasets of fallacies. olution. The pragmatic aspect describes the moves Ad hominem fallacies in conversations have in the discussion as speech acts, more precisely as been addressed in (Habernal et al., 2018b). The the illocutionary acts introduced by Searle(1979). authors used the subreddit ChangeMyView, which van Eemeren and Grootendorst(1995) also devel- is a forum for civilized discussions, “a place to post oped ten rules which should guide argumentative an opinion you accept may be flawed, in an effort discussions. The goal of the rules is to further the to understand other perspectives on the issue”. The understanding of the difference of opinions and to dataset of fallacies consists of comments that were create a fruitful discussion. For example, a rule removed by the moderators as they violated the states that parties must not prevent each other from rule of not being rude or hostile, hence committing advancing standpoints or from casting doubt on an ad hominem fallacy. standpoints, while a second rule asks that a party Fallacious arguments are often made in the dis- may defend a standpoint only by advancing argu- semination of propaganda. In Da San Martino et al. mentation relating to that standpoint. An argument (2019), the authors annotate journal articles with 18 that prevents the resolution and thus violates one propaganda techniques, out of which 12 techniques of the rules is a fallacy. In our work, we align fre- are fallacies. Although an important resource in quent fallacies on Reddit with these rules, with the the study of fallacies, their labelling method and goal of formalizing their definitions. dataset have a few drawbacks. First, the dataset Another well-known model that considers fal- is highly unbalanced with 6 fallacies having a fair lacies is the argumentation scheme introduced by number of mentions: name-calling (1294), appeal 645 to fear and prejudice (367), flag-waving (330), Submission title: What is something massively outdated that humanity has yet to upgrade? causal oversimplification (233), appeal to authority Link: https://www.reddit.com/r/AskReddit/comments/b3nwm6/ (169), black and white fallacy (134), and 6 fallacies Comment by user1 having less than 100 mentions: whataboutism (76), The 5-day work week reductio ad hitlerum (66), red herring (48), band- Comment by user2 wagon (17), labeling, obfuscation or intentional I know a lot of people don't like it, but a 9 to 5 office job is vagueness (17), straw men (15). Second, the task a pretty big step up from slavery, feudalism and indentured servitude. Though I do agree with studies saying that of finding the correct label for a span of text from a working for less than 8 hours a day is more productive.