Breaking Down the Invisible Wall of Informal Fallacies in Online Discussions Saumya Yashmohini Sahai Oana Balalau The Ohio State University, USA Inria, Institut Polytechnique de Paris, France [email protected] [email protected] Roxana Horincar Thales Research & Technology, France [email protected] Abstract more appropriate. Fallacies are prevalent in public discourse. For example, The New York Times la- People debate on a variety of topics on online beled the tweets of Donald Trump between 2015 platforms such as Reddit, or Facebook. De- bates can be lengthy, with users exchanging and 2020 and found thousands of insults addressed a wealth of information and opinions. How- to his adversaries. If made in an argument, an in- ever, conversations do not always go smoothly, sult is an ad hominem fallacy: an attack on the and users sometimes engage in unsound argu- opponent rather than on their argument. In pri- mentation techniques to prove a claim. These vate conversations, other types of fallacies might techniques are called fallacies. Fallacies are be more prevalent, for example, appeal to tradition persuasive arguments that provide insufficient or appeal to nature. Appeal to tradition dismisses or incorrect evidence to support the claim. In calls to improve gender equality by stating that this paper, we study the most frequent falla- cies on Reddit, and we present them using “women have always occupied this place in soci- the pragma-dialectical theory of argumenta- ety”. Appeal to nature is often used to ignore calls tion. We construct a new annotated dataset of to be inclusive of the LGBTQ+ community by stat- fallacies, using user comments containing fal- ing “gender is binary”. The underlying premises lacy mentions as noisy labels, and cleaning the of such arguments are “traditions are correct” and data via crowdsourcing. Finally, we study the “what occurs in nature is good”. task of classifying fallacies using neural mod- Creating a dataset of fallacious arguments is els. We find that generally the models perform better in the presence of conversational con- difficult, given that there are over 100 types of text.We have released the data and the code fallacious arguments (Scalambrino, 2018). There at github.com/sahaisaumya/informal_ have been several attempts to create comprehensive fallacies. datasets: Habernal et al.(2017) proposed a game in which players add fallacies in the hope of foul- ing other participants, in Habernal et al.(2018a) 1 Introduction ad hominem fallacies are found using a subred- dit’s rule violations, while in Da San Martino et al. Argumentation plays a critical part in our lives as (2019) fallacies are annotated together with other it helps us make decisions and reason about the propaganda techniques in news articles. However, world around us. Studies (Sanders et al., 1994) our work is the first to propose a viable solution have shown that learning how to argue increases the for finding fallacious arguments belonging to many ability to identify weak arguments and decreases different fallacy types. the tendency to use verbal aggressiveness. Fallacies In this work, we study fallacies in public discus- are weak arguments that seem convincing, however, sions on online forums. Our salient contributions their evidence does not prove or disprove the argu- are: i) we align informal fallacies mentioned on ment’s conclusion. Fallacies are usually divided Reddit within the pragma-dialectic theory of argu- into formal and informal, where the former can mentation (van Eemeren and Grootendorst, 1995); be easily described using logical representations, ii) we design a methodology for mining and label- while for the latter, an analysis of the content is ing easily fallacies in online discussions; iii) we Part of this work was done while the first author was an construct a large and balanced dataset of fallacious intern at Inria, France. arguments; iv) finally, we evaluate several neural models on the task of predicting fallacious argu- lacies is the argumentation scheme introduced by ments, and we find that taking into consideration Douglas Walton (Walton, 2005). A scheme con- additional conversational context is important for sists of a conclusion, a set of premises, and a set this task. of critical questions. The critical questions should be answered in order to prove that the premises 2 Background support the conclusion, hence the argument is not a fallacy. For example, the scheme for an argu- 2.1 Fallacies in Argumentation Theory ment from expert opinion (Walton, 2005) has the Humans use argumentation when they evaluate the premises E is an expert in domain D, E asserts validity of new ideas, or they want to solve a dif- that A is known to be true, A is within D and the ference of opinion. An argument contains: i) a conclusion therefore, A may plausibly be taken to proposition called claim, conclusion or standpoint, be true. Some critical questions for this scheme to be validated; ii) the premises called also evi- are: i) Trustworthiness: Is E personally reliable as dence, which are the backing propositions; iii) an a source? ii) Backup Evidence: Is E’s assertion inference relation between the evidence and con- based on evidence? Argumentation schemes have clusion that validates or disproves the conclusion. two main drawbacks: first, for each new fallacy, a A fallacy is a flawed argument, where the inference new scheme should exist or be defined; and sec- relation or the premises are incorrect. Fallacies are ond, in the context of labeling an existing argument, generally divided into formal and informal falla- many of the critical questions might be unanswer- cies. Formal fallacies are arguments that can be able as none of the parties discussed them. easily represented as invalid logical formulas, such as denying the antecedent, which is a wrong appli- 2.2 Related Work cation of modus tollens. Although many informal fallacies can be also represented as invalid argu- An initial effort for creating an extensive dataset of ments, informal fallacies are easier to describe and fallacies was made in Habernal et al.(2017). The understand without resorting to logical representa- authors created a platform for educative games, tions (Hansen, 2020). where players learn how to become better debaters. In this work, we follow the pragma dialectic New fallacies are added to the platform by play- theory of argumentation. The theory developed ers that try to earn points by fouling other partici- by van Eemeren and Grootendorst(1995) views pants with invalid arguments. A follow-up on this argumentation as a complex speech act. The dialec- work (Habernal et al., 2018a) mentioned a dataset tical aspect is represented by two parties who try of only around 300 arguments created via the plat- to resolve a difference of opinion by engaging in a form, thus showing the need of finding other meth- discussion, each party making a move towards res- ods for creating larger datasets of fallacies. olution. The pragmatic aspect describes the moves Ad hominem fallacies in conversations have in the discussion as speech acts, more precisely as been addressed in (Habernal et al., 2018b). The the illocutionary acts introduced by Searle(1979). authors used the subreddit ChangeMyView, which van Eemeren and Grootendorst(1995) also devel- is a forum for civilized discussions, “a place to post oped ten rules which should guide argumentative an opinion you accept may be flawed, in an effort discussions. The goal of the rules is to further the to understand other perspectives on the issue”. The understanding of the difference of opinions and to dataset of fallacies consists of comments that were create a fruitful discussion. For example, a rule removed by the moderators as they violated the states that parties must not prevent each other from rule of not being rude or hostile, hence committing advancing standpoints or from casting doubt on an ad hominem fallacy. standpoints, while a second rule asks that a party Fallacious arguments are often made in the dis- may defend a standpoint only by advancing argu- semination of propaganda. In Da San Martino et al. mentation relating to that standpoint. An argument (2019), the authors annotate journal articles with 18 that prevents the resolution and thus violates one propaganda techniques, out of which 12 techniques of the rules is a fallacy. In our work, we align fre- are fallacies. Although an important resource in quent fallacies on Reddit with these rules, with the the study of fallacies, their labelling method and goal of formalizing their definitions. dataset have a few drawbacks. First, the dataset Another well-known model that considers fal- is highly unbalanced with 6 fallacies having a fair number of mentions: name-calling (1294), appeal Submission title: What is something massively outdated that humanity has yet to upgrade? to fear and prejudice (367), flag-waving (330), Link: https://www.reddit.com/r/AskReddit/comments/b3nwm6/ causal oversimplification (233), appeal to authority Comment by user1 (169), black and white fallacy (134), and 6 fallacies The 5-day work week having less than 100 mentions: whataboutism (76), Comment by user2 reductio ad hitlerum (66), red herring (48), band- I know a lot of people don't like it, but a 9 to 5 office job is wagon (17), labeling, obfuscation or intentional a pretty big step up from slavery, feudalism and indentured servitude. Though I do agree with studies saying that vagueness (17), straw men (15). Second, the task working for less than 8 hours a day is more productive. of finding the correct label for a span of text from a large set of labels (18 in their case) is intellectually Comment by user3 complex and time-consuming. Our work focuses fallacy of relative privation on collecting and annotating a balanced dataset of Comment by user2 fallacy mentions while providing a methodology Are you majoring in psychology? Hahaha that can easily scale to a larger number of fallacies.