Classifying Arguments by Scheme
Total Page:16
File Type:pdf, Size:1020Kb
Classifying Arguments by Scheme Vanessa Wei Feng Graeme Hirst Department of Computer Science Department of Computer Science University of Toronto University of Toronto Toronto, ON, M5S 3G4, Canada Toronto, ON, M5S 3G4, Canada [email protected] [email protected] Abstract Another premise is left implicit — “Adhering to those treaties and covenants is a means of realizing Argumentation schemes are structures or tem- survival of the entire world”. This proposition is an plates for various kinds of arguments. Given enthymeme of this argument. the text of an argument with premises and con- Our ultimate goal is to reconstruct the en- clusion identified, we classify it as an instance thymemes in an argument, because determining of one of five common schemes, using features specific to each scheme. We achieve accura- these unstated assumptions is an integral part of un- cies of 63–91% in one-against-others classifi- derstanding, supporting, or attacking an entire argu- cation and 80–94% in pairwise classification ment. Hence reconstructing enthymemes is an im- (baseline = 50% in both cases). portant problem in argument understanding. We be- lieve that first identifying the particular argumenta- tion scheme that an argument is using will help to 1 Introduction bridge the gap between stated and unstated proposi- tions in the argument, because each argumentation We investigate a new task in the computational anal- scheme is a relatively fixed “template” for arguing. ysis of arguments: the classification of arguments That is, given an argument, we first classify its ar- by the argumentation schemes that they use. An ar- gumentation scheme; then we fit the stated proposi- gumentation scheme, informally, is a framework or tions into the corresponding template; and from this structure for a (possibly defeasible) argument; we we infer the enthymemes. will give a more-formal definition and examples in Section 3. Our work is motivated by the need to de- In this paper, we present an argument scheme termine the unstated (or implicitly stated) premises classification system as a stage following argument that arguments written in natural language normally detection and proposition classification. First in Sec- draw on. Such premises are called enthymemes. tion 2 and Section 3, we introduce the background to our work, including related work in this field, For instance, the argument in Example 1 consists the two core concepts of argumentation schemes and of one explicit premise (the first sentence) and a con- scheme-sets, and the Araucaria dataset. In Section 4 clusion (the second sentence): and Section 5 we present our classification system, Example 1 [Premise:] The survival of the entire including the overall framework, data preprocessing, world is at stake. feature selection, and the experimental setups. In [Conclusion:] The treaties and covenants aiming the remaining section, we present the essential ap- for a world free of nuclear arsenals and other con- proaches to solve the leftover problems of this paper ventional and biological weapons of mass destruc- which we will study in our future work, and discuss tion should be adhered to scrupulously by all na- the experimental results, and potential directions for tions. future work. 987 Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 987–996, Portland, Oregon, June 19-24, 2011. c 2011 Association for Computational Linguistics 2 Related work determine how the pieces fit together as an instance of an argumentation scheme. Argumentation has not received a great deal of at- tention in computational linguistics, although it has 3 Argumentation schemes, scheme-sets, been a topic of interest for many years. Cohen and annotation (1987) presented a computational model of argu- mentative discourse. Dick (1987; 1991a; 1991b) de- 3.1 Definition and examples veloped a representation for retrieval of judicial de- Argumentation schemes are structures or templates cisions by the structure of their legal argument — a for forms of arguments. The arguments need not be necessity for finding legal precedents independent of deductive or inductive; on the contrary, most argu- their domain. However, at that time no corpus of ar- mentation schemes are for presumptive or defeasible guments was available, so Dick’s system was purely arguments (Walton and Reed, 2002). For example, theoretical. Recently, the Araucaria project at Uni- argument from cause to effect is a commonly used versity of Dundee has developed a software tool for scheme in everyday arguments. A list of such argu- manual argument analysis, with a point-and-click in- mentation schemes is called a scheme-set. terface for users to reconstruct and diagram an ar- It has been shown that argumentation schemes gument (Reed and Rowe, 2004; Rowe and Reed, are useful in evaluating common arguments as falla- 2008). The project also maintains an online repos- cious or not (van Eemeren and Grootendorst, 1992). itory, called AraucariaDB, of marked-up naturally In order to judge the weakness of an argument, a set occurring arguments collected by annotators world- of critical questions are asked according to the par- wide, which can be used as an experimental corpus ticular scheme that the argument is using, and the for automatic argumentation analysis (for details see argument is regarded as valid if it matches all the Section 3.2). requirements imposed by the scheme. Recent work on argument interpretation includes Walton’s set of 65 argumentation schemes (Wal- that of George, Zukerman, and Nieman (2007), who ton et al., 2008) is one of the best-developed scheme- interpret constructed-example arguments (not natu- sets in argumentation theory. The five schemes de- rally occurring text) as Bayesian networks. Other fined in Table 1 are the most commonly used ones, contemporary research has looked at the automatic and they are the focus of the scheme classification detection of arguments in text and the classification system that we will describe in this paper. of premises and conclusions. The work closest to ours is perhaps that of Mochales and Moens (2007; 3.2 Araucaria dataset 2008; 2009a; 2009b). In their early work, they fo- One of the challenges for automatic argumentation cused on automatic detection of arguments in legal analysis is that suitable annotated corpora are still texts. With each sentence represented as a vector of very rare, in spite of work by many researchers. shallow features, they trained a multinomial na¨ıve In the work described here, we use the Araucaria Bayes classifier and a maximum entropy model on database1, an online repository of arguments, as our the Araucaria corpus, and obtained a best average experimental dataset. Araucaria includes approxi- accuracy of 73.75%. In their follow-up work, they mately 660 manually annotated arguments from var- trained a support vector machine to further classify ious sources, such as newspapers and court cases, each argumentative clause into a premise or a con- and keeps growing. Although Araucaria has sev- clusion, with an F1 measure of 68.12% and 74.07% eral limitations, such as rather small size and low respectively. In addition, their context-free grammar agreement among annotators2, it is nonetheless one for argumentation structure parsing obtained around of the best argumentative corpora available to date. 60% accuracy. Our work is “downstream” from that of Mochales 1http://araucaria.computing.dundee.ac.uk/doku.php# and Moens. Assuming the eventual success of their, araucaria argumentation corpus 2The developers of Araucaria did not report on inter- or others’, research program on detecting and clas- annotator agreement, probably because some arguments are an- sifying the components of an argument, we seek to notated by only one commentator. 988 Argument from example are annotated as “missing = yes”. Premise: In this particular case, the individual a Example 2 Example of argument markup from has property F and also property G. Araucaria Conclusion: Therefore, generally, if x has prop- <TEXT>If we stop the free creation of art, we will stop erty F, then it also has property G. the free viewing of art.</TEXT> <AU> Argument from cause to effect <PROP identifier="C" missing="yes"> Major premise: Generally, if A occurs, then B will <PROPTEXT offset="-1"> The prohibition of the free creation of art should (might) occur. not be brought about.</PROPTEXT> Minor premise: In this case, A occurs (might oc- <INSCHEME scheme="Argument from Consequences" cur). schid="0" /> </PROP> Conclusion: Therefore, in this case, B will <LA> (might) occur. <AU> <PROP identifier="A" missing="no"> Practical reasoning <PROPTEXT offset="0"> If we stop the free creation of art, we will Major premise: I have a goal G. stop the free viewing of art.</PROPTEXT> Minor premise: Carrying out action A is a means <INSCHEME scheme="Argument from Consequences" schid="0" /> to realize G. </PROP> Conclusion: Therefore, I ought (practically </AU> speaking) to carry out this action A. <AU> <PROP identifier="B" missing="yes"> Argument from consequences <PROPTEXT offset="-1"> The prohibition of free viewing of art is not Premise: If A is (is not) brought about, good (bad) acceptable.</PROPTEXT> consequences will (will not) plausibly occur. <INSCHEME scheme="Argument from Consequences" schid="0" /> Conclusion: Therefore, A should (should not) be </PROP> brought about. </AU> </LA> Argument from verbal classification </AU> Individual premise: a has a particular property F. There are three scheme-sets used in the anno- Classification premise: For all x, if x has property tations in Araucaria: Walton’s scheme-set, Katzav F, then x can be classified as having property and Reed’s (2004) scheme-set, and Pollock’s (1995) G. scheme-set. Each of these has a different set of Conclusion: Therefore, a has property G. schemes; and most arguments in Araucaria are marked up according to only one of them. Our Table 1: The five most frequent schemes and their defini- experimental dataset is composed of only those tions in Walton’s scheme-set.