Argument Discovery and Extraction with the Argument Workbench
Adam Wyner Wim Peters David Price Computing Science Computer Science DebateGraph University of Aberdeen University of Sheffield United Kingdom Aberdeen, United Kingdom Sheffield, United Kingdom [email protected] [email protected] [email protected]
Abstract lexical forms for related semantic meaning. It is dif- ficult for humans to reconstruct argument from text, The paper discusses the architecture and de- let alone for a computer. This is especially the case velopment of an Argument Workbench, which where arguments are dispersed across unstructured is a interactive, integrated, modular tool set to textual corpora. In our view, the most productive extract, reconstruct, and visualise arguments. We consider a corpora with dispersed infor- scenario is one in which a human argument engineer mation across texts, making it essential to con- is maximally assisted in her work by computational ceptually search for argument elements, top- means in the form of automated text filtering and an- ics, and terminology. The Argument Work- notation. This enables the engineer to focus on text bench is a processing cascade, developed in that matters and further explore the argumentation collaboration with DebateGraph. The tool structure on the basis of the added metadata. The supports an argument engineer to reconstruct Argument WorkBench (AWB) captures this process arguments from textual sources, using infor- of incremental refinement and extension of the argu- mation processed at one stage as input to a subsequent stage of analysis, and then build- ment structure, which the engineer then produces as ing an argument graph. We harvest and pre- a structured object with a visual representation. process comments; highlight argument indi- Given the abundance of textual source data avail- cators, speech act and epistemic terminology; able for argumentation analysis there is a real need model topics; and identify domain terminol- for automated filtering and interpretation. Cur- ogy. We use conceptual semantic search over rent social media platforms provide an unprece- the corpus to extract sentences relative to argu- ment and domain terminology. The argument dented source of user-contributed content on most engineer uses the extracts for the construction any topic. Reader-contributed comments to a com- of arguments in DebateGraph. ment forum, e.g. for a news article, are a source of arguments for and against issues raised in the article, where an argument is a claim with justifications and 1 Introduction exceptions. It is difficult to coherently understand Argumentative text is rich, multidimensional, and the overall, integrated meaning of the comments. fine-grained, consisting of (among others): a range To reconstruct the arguments sensibly and of (explicit and implicit) discourse relations between reusably, we build on a prototype Argument Work- statements in the corpus, including indicators for bench (AWB) (Wyner et al.(2012); Wyner(2015)), conclusions and premises; speech acts and proposi- which is a semi-automated, interactive, integrated, tional attitudes; contrasting sentiment terminology; modular tool set to extract, reconstruct, and visualise and domain terminology. Moreover, linguistic ex- arguments. The workbench is a processing cascade, pression is various, given alternative syntactic or developed in collaboration with an industrial partner
78
Proceedings of the 2nd Workshop on Argumentation Mining, pages 78–83, Denver, Colorado, June 4, 2015. c 2015 Association for Computational Linguistics DebateGraph and used by an Argumentation Engi- Figure 1 shows the overall workflow. Document neer, where information processed at one stage gives collection is not taken into account. In the first greater structure for the subsequent stage. In partic- stage, text analysis such as topic, term and named ular, we: harvest and pre-process comments; high- entity extraction provides a first thematic grouping light argument indicators, speech act terminology, and semantic classification of relevant domain ele- epistemic terminology; model topics; and identify ments. This combination of topics, named entities domain terminology and relationships. We use con- and terms automatically provides the first version ceptual semantic search over the corpus to extract of a domain model, which assists the engineer in sentences relative to argument and domain terminol- the conceptual interpretation and subsequent explo- ogy. The argument engineer analyses the output and ration. The texts filtered in this thematic way can then inputs extracts into the DebateGraph visualisa- then be filtered further with respect to argument in- tion tool. The novelty of the work presented in this dicators (discourse terminology, speech acts, epis- paper is the addition of terminology (domain top- temic terminology) as well as sentiment (positive ics and key words, speech act, and epistemic) along and negative terminology). At each stage, the Argu- with the workflow analysis provided by our indus- mentation Engineer is able to query the corpus with trial partner. For this paper, we worked with a corpus respect to the metadata (which we also refer to as of texts bearing on the Scottish Independence vote in the conceptual annotations). This complex filtering 2014; however, the tool is neutral with respect to do- of information from across a corpus helps the Argu- main, since the domain terminology is derived using mentation Engineer consolidate her understanding automatic tools. of the argumentative role of information. In this short paper, we briefly outline the AWB workflow, sketch tool components, provide sample 3 AWB Components query results, discuss related work in the area, and 3.1 Text Analysis close with a brief discussion. To identify and extract the textual elements from the source material, we use the GATE framework (Cun- 2 The Argument WorkBench Workflow ningham et al.(2002)) for the production of semantic The main user of the Argument WorkBench (AWB) metadata in the form of annotations. is Argumentation Engineer, an expert in argumen- GATE is a framework for language engineer- tation modeling who uses the Workbench to select ing applications, which supports efficient and ro- and interpret the text material. Although the AWB bust text processing including functionality for automates some of the subtasks involved, the ulti- both manual and automatic annotation (Cunningham mate modeler is the argumentation engineer. The et al.(2002)); it is highly scalable and has been ap- AWB distinguishes between the selection and mod- plied in many large text processing projects; it is an eling tasks, where selection is computer-assisted and open source desktop application written in Java that semi-automatic, whereas the modeling is performed provides a user interface for professional linguists manually in DebateGraph (see Figure 1). and text engineers to bring together a wide variety of natural language processing tools and apply them to The AWB encompasses a flexible methodology a set of documents. The tools are concatenated into that provides a workflow and an associated set of a pipeline of natural language processing modules. modules that together form a flexible and extendable The main modules we are using in our bottom-up methodology for the detection of argument in text. and incremental tool development (Wyner and Pe- Automated techniques provide textually grounded ters(2011)) perform the following functionalities: information about conceptual nature of the domain and the argument structure by means of the detec- linguistic pre-processing. Texts are segmented • tion of argument indicators. This information, in the into tokens and sentences; words are assigned form of textual metadata, enable the argumentation Part-of-Speech (POS). engineer to filter out potentially interesting text for eventual manual analysis, validation and evaluation. gazetteer lookup. A gazetteer is a list of words • 79 Figure 1: Overview of the Argument WorkBench Workflow
associated with a central concept. In the lookup to TermRaider, we have used a tool to model top- phase, text in the corpus is matched with terms ics, identifying clusters of terminology that are taken on the lists, then assigned an annotation. to statistically “cohere” around a topic; for this, we have used a tool based on Latent Dirichlet Alloca- annotation assignment through rule-based • tion (Blei et al.(2008)). Each word in a topic is used grammars, where rules take annotations and to annotate every sentence in the corpus that con- regular expressions as input and produce tains that word. Thus, with term and topic annota- annotations as output. tion, the Argumentation Engineer is able to query Once a GATE pipeline has been applied, the ar- the corpus for relevant, candidate passages. gument engineer views the annotations in situ or 3.3 DebateGraph using GATE’s ANNIC (ANNotations In Context) corpus indexing and querying tool (see section 4), DebateGraph is a free, cloud-based platform that en- which enables semantic search for annotation pat- ables communities of any size to build and share dy- terns across a distributed corpus. namic interactive visualizations of all the ideas, ar- guments, evidence, options and actions that anyone 3.2 Term and Topic Extraction in the community believes relevant to the issues un- In the current version of the AWB, we used two der consideration, and to ensure that all perspectives automatic approches to developing terminology, al- are represented transparently, fairly, and fully in a lowing the tool to be domain independent and meaningful, structured and iterative dialogue. It sup- rapidly developed. We used the TermRaider tool ports formal argumentation as well as structured dia- in GATE to identify relevant terminology (Maynard logue, and has been used by, amongst others, CNN, et al.(2008)). TermRaider automatically provides The Independent newspaper, the White House Of- domain-specific noun phrase term candidates from fice of Science and Technology Policy, the European a text corpus together with a statistically derived ter- Commission, and the UK’s Prime Minister’s Office mhood score. Possible terms are filtered by means as well as the Foreign and Commonwealth Office. of a multi-word-unit grammar that defines the pos- 4 Viewing and Querying sible sequences of part of speech tags constituting noun phrases. It computes term frequency/inverted The AWB enriches the manual, close reading ori- document frequency (TF/IDF), which takes into ac- ented method of argument map creation in Debate- count term frequency and the number of documents Graph with automated analysis, which filters rele- in the collection, yielding a score that indicates the vant text segments with respect to a certain topic of salience of each term candidate for each document interest, and provides initial argument structure in- in the corpus. All term candidates with a TF/IDF formation to the text by means of annotations. score higher than an manually determined threshold Once the corpus is annotated, we can view the are then selected and presented as candidate relevant annotations in the documents themselves. In Fig- terms, annotated as such in the corpus. In addition ure 2, we have a text that has been highlighted with
80 Figure 2: Highlighting Annotations in the Text a selection of available annotation types (differen- The data is displayed in a GUI, facilitating explo- tiated by colour in the original): Topic4 (labels in- ration and the discovery of new patterns. dicative of Topic 4); SentenceTopic4 (Sentences in Searching in the corpus for single annotations which Topic4 labels occur); various discourse level returns all those strings that are annotated with information types such as discourse/argument mark- the search annotation along with their context and ers and speech acts. Other annotations are available, source document. Figure 3 illustrates a more com- e.g. sentiment and epistemic. The argumentation plex query in the top pane by means of which an ar- engineer can now focus on the close reading of sen- gumentation engineer wants to explore up to seven tences that represent relevant topics, contain the re- token corpus contexts that contain particular term quired terminology and argumentational aspects. candidates and argument indicators. The query finds For corpus-level exploration and selection, search all sequences of annotated text where the first string patterns can be formulated and examined by means is annotated with ArgumentationIndicator, followed of the ANNIC (Annotation in Context) querying and by zero to five other Tokens, followed by a string visualization tool in GATE (Aswani et al.(2005)). with a TermCandidate annotation. One of the search This tool can index documents not only by content, results is visualised in the bottom pane by means of but also by their annotations and features. It also query matches and left/right contexts. The coloured enables users to formulate versatile queries mixing bars form an annotation stack that shows the occur- keywords and information available from any anno- rence of selected annotations in the contexts. In this tation types (e.g. linguistic, conceptual). The re- case we see an emphasis argument indicator ”obvi- sult consists of the matching texts in the corpus, dis- ously” co-occurring with the term candidates ”Scot- played within the context of linguistic annotations land”, ”independent Scotland” and ”choice”. (not just text, as is customary for KWIC systems). By inspecting the document more closely the ar-
81 gumentation engineer will be able to produce a of gold standard corpora on which statistical models structured representation of the identified argument. can be trained. Finally, we are not aware of statis- The ANNIC interface thus uses the annotations to tical models to extract the fine-grained information reduce the search space for human engineers, and fo- that is required for extracting argument elements. cuses their attention on passages that are relevant for The tool is used to construct or reconstruct argu- sourcing arguments. The tool allows incremental re- ments in complex, high volume, fragmentary, and finement of searches, allowing for a interactive way alinearly presented comments or statements. This to examine the semantic content of the texts. Also, is in contrast to many approaches that, by and large, the argumentation engineer can provide feedback in follow the structure of arguments within a particu- the form of changing/adding annotations, which will lar (large and complex) document, e.g. the BBC’s be used in GATE to improve the automated analysis. Moral Maze (Bex et al.(2014)), manuals (Saint- Dizier(2012)), and legal texts (Moens et al.(2007)). 5 Related Work The tool can be modularly developed, adding The paper presents developments of an imple- further argumentation elements, domain mod- mented, semi-automatic, interactive text analytic els, disambiguating discourse indicators (Webber tool that combines rule-based and statistically- et al.(2011)), auxilary linguistic indicators, and oriented approaches. The tool supports analysts other parts of speech that distinguish sentence com- in identifying “hot zones” of relevant textual ma- ponents. More elaborate query patterns could be ex- terial as well as fine-grained, relevant textual pas- ecuted to refine results. In general, the openness and sages; these passages can be used to compose ar- flexibility of the tool provide a platform for future, gument graphs in a tool such as DebateGraph. As detailed solutions to issues in argumentation. such, the tool evaluated with respect to user facili- 6 Discussion tation (i.e. analysts qualitative evaluation of using the tool or not) rather than with respect to recall and The tool offers a very flexible, useful and meaning- precision (Mitkof(2003)) in comparison to a gold ful way to query a corpus of text for relevant ar- standard. The tool is an advance over graphically- gument passages, leaving the argument engineer to based argument extraction tools that rely on the an- further analyse and use the results. Having devel- alysts’ unstructured, implicit, non-operationalised oped in in conjunction with an industrial partner, the knowledge of discourse indicators and content (van next task is to evaluate it with user studies, inquiring Gelder(2007); Rowe and Reed(2008); Liddo and whether the tool facilitates or changes the capabil- Shum(2010); Bex et al.(2014)). There are a va- ity to develop arguments for graphs. As a result of riety of rule-based approaches to argument an- this feedback, the tool can be developed further, e.g. notation: (Pallotta and Delmonte(2011)) classify adding a summarisation component, automating ex- statements according to rhetorical roles using full traction, augmenting the base terminology (speech sentence parsing and semantic translation; (Saint- acts, propositional attitudes, etc), and creating dis- Dizier(2012)) provides a rule-oriented approach to course indicator patterns. The tool can also be used process specific, highly structured argumentative to examine the role of the various components in texts; (Moens et al.(2007)) manually annotates le- the overall argument pattern search, investigating the gal texts then constructs a grammar that is tailored use of, e.g. discourse indicators or speech acts in dif- to automatically annotated the passages. Such rule- ferent discourse contexts. oriented approaches share some generic compo- Acknowledgments nents with our approach, e.g. discourse indicators, The authors gratefully acknowledge funding from negation indicators. However, they do not exploit the Semantic Media Network project Semantic Me- a terminological analysis, do not straightforwardly dia: a new paradigm for navigable content for the provide for complex annotation querying, and are 21st Century (EPSRC grant EP/J010375/1). Thanks stand-alone tools that are not integrated with other to Ebuka Ibeke and Georgios Klados for their con- NLP tools. Importantly, the rule-based approach tributions to the project. outlined here could be used to support the creation
82 Figure 3: Searching for Patterns in the Corpus
References Glenn Rowe and Chris Reed. Argument diagramming: The Araucaria Project. In Alexandra Okada, Si- Niraj Aswani, Valentin Tablan, Kalina Bontcheva, and mon Buckingham Shum, and Tony Sherborne, editors, Hamish Cunningham. Indexing and Querying Lin- Knowledge Cartography: Software Tools and Map- guistic Metadata and Document Content. In Proceed- ping Techniques, pages 163–181. Springer, 2008. ings RANLP 2005, Borovets, Bulgaria, 2005. Patrick Saint-Dizier. Processing natural language argu- Floris Bex, Mark Snaith, John Lawrence, and Chris Reed. ments with the
83