DOCSLIB.ORG

Significant Concordance and Co-Occurrence in Quantitative

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Significant Concordance and Co-Occurrence in Quantitative Exploring Cities in Crime: Significant Concordance and Co-occurrence in Quantitative Literary Analysis Janneke Rauscher1 Leonard Swiezinski2 Martin Riedl2 Chris Biemann2 (1) Johann Wolfgang Goethe University Gruneburgplatz¨ 1, 60323 Frankfurt am Main, Germany (2) FG Language Technology, Dept. of Computer Science Technische Universitat¨ Darmstadt, 64289 Darmstadt, Germany [email protected], [email protected], {riedl,biem}@cs.tu-darmstadt.de Abstract Although the number of research projects in Dig- ital Humanities is increasing at fast pace, we still We present CoocViewer, a graphical analy- observe a gap between the traditional humanities sis tool for the purpose of quantitative lit- scholars on the one side, and computer scientists on erary analysis, and demonstrate its use on the other. While computer science excels in crunch- a corpus of crime novels. The tool dis- ing numbers and providing automated processing plays words, their significant co-occurrences, and contains a new visualization for signif- for large amounts of data, it is hard for the com- icant concordances. Contexts of words and puter scientist to imagine what research questions co-occurrences can be displayed. After re- form the discourse in the humanities. In contrast to viewing previous research and current chal- this, humanities scholars have a hard time imagining lenges in the newly emerging field of quan- the possibilities and limitations of computer technol- titative literary research, we demonstrate how ogy, how automatically generated results ought to CoocViewer allows comparative research on be interpreted, and how to operationalize automatic literary corpora in a project-specific study, and how we can confirm or enhance our hypothe- processing in a way that its unavoidable imperfec- ses through quantitative literary analysis. tions are more than compensated by the sheer size of analysable material. This paper resulted from a successful co- 1 Introduction operation between a natural language processing Recent years have seen a surge in Digital Human- (NLP) group and a literary researcher in the field ities research. This area, touching on both the of Digital Humanities. We present the CoocViewer fields of computer science and the humanities, is analysis tool for literary and other corpora, which concerned with making data from the humanities supports new angles in literary research through analysable by digitalisation. For this, computational quantitative analysis. tools such as search, visual analytics, text mining, In the Section 2, we describe the CoocViewer statistics and natural language processing aid the hu- tool and review the landscape of previously available manities researcher. On the one hand, software per- tools for our purpose. As a unique characteristic, mits processing a larger set of data in order to assess CoocViewer contains a visualisation of significant traditional research questions. On the other hand, concordances, which is especially useful for target this gives rise to a transformation of the way re- terms of high frequency. In Section 3, we map the search is conducted in the humanities: the possibil- landscape of previous and current quantitative re- ity of analyzing a much larger amount of data – yet search in literary analysis, which is still an emerging in a quantitative fashion with all its necessary aggre- and somewhat controversial sub-discipline. A use- gation – opens the path to new research questions, case for the tool in the context of a specific project and different methodologies for attaining them. is laid out in Section 4, where a few examples illus- 61 Proceedings of the Second Workshop on Computational Linguistics for Literature, pages 61–71, Atlanta, Georgia, June 14, 2013. c 2013 Association for Computational Linguistics trate how CoocViewer is used to confirm and gen- ried out for producing the contents of CoocViewer’s erate hypotheses in literary analysis. Section 5 con- database. These steps consist of a fairly standard cludes and provides an outlook to further needs in natural language processing pipeline, which we de- tool support for quantative literary research. scribe shortly. After tokenizing, part-of-speech tagging (Schmid, 2 CoocViewer - a Visual Corpus Browser 1994) and indexing the input data by document, This section describes our CoocViewer visual cor- sentence and paragraph within the document, we pus browsing tool. After shortly outlining neces- compute signficant sentence-wide and paragraph- 6 sary pre-processing steps, we illustrate and moti- wide co-occurrences, using the tinyCC tool. Here, vate the functionality of the graphical user interface. the log-likelihood test (Dunning, 1993) is employed The tool was specifically designed to aid researchers to determine the significance sig(A, B) of the co- from the humanities that do not have a background occurrence of two tokens A and B. To sup- in computational linguistics. port the significant concordance view (described in the next section), we have extended the tool to 2.1 Related Work also produce positional significant co-occurrences, Whereas there exist a number of tools for visualizing where sig(A, B, offset) is computed by the log- co-occurrences, there is, to the best of our knowl- likelihood significance of the co-occurrence of A edge, no tool to visualize positional co-occurrences, and B in a token-distance of offset. Since the sig- or as we also call them, significant concordances. nificance measure requires the single frequencies of In (Widdows et al., 2002) tools are presented that A and B, as well as their joint frequency per po- visualize meanings of nouns as vector space repre- sitional offset in this setup, this adds considerable sentation, using LSA (Deerwester et al., 1990) and overhead during preprocessing. To our knowledge, graph models using co-occurrences. There is also we are the first to extend the notion of positional co- a range of text-based tools, without any quantita- occurrence beyond direct neighbors, cf. (Richter et tive statistics, e.g. Textpresso (Muller¨ et al., 2004), al., 2006). We apply a sigificance threshold of 3.847 PhraseNet1 and Walden2. For searching words in and a frequency threshold of 2 to only keep ‘interest- context, Luhn (1960) introduced KWIC (Key Word ing’ pairs. The outcome of preprocessing is stored in in Context) which allows us to search for concor- a MySQL database schema similar to the one used dances and is also used in several corpus linguis- by LCC (Biemann et al., 2007). We store sentence- tic tools e.g. (Culy and Lyding, 2011), BNCWeb3, and paragraph-wide co-occurrences and positional Sketch Engine (Kilgarriff et al., 2004), Corpus co-occurrences in separate database tables, and use Workbench4 and MonoConc (Barlow, 1999). Al- one database per corpus. The database tables are in- though several tools for co-occurrences visualiza- dexed accordingly to optimize the queries issued by tion exist (see e.g. co-occurrences for over 200 lan- the CoocViewer tool. Additionally, we map the part- guages at LCC5), they often have different aims, and of-speeches to E (proper names), N (proper nouns), e.g. do not deliver the functionality to filter on dif- A (adjectives), V (verbs), R (all other part-of-speech ferent part-of-speech tags. tags) for an uniform representation for different lan- guages. 2.2 Corpus Preprocessing To make a natural language corpus accessible in the 2.3 Graphical User Interface tool, a number of preprocessing steps have to be car- The graphical user interface (UI) is built with com- 1http://www-958.ibm.com/software/data/ mon web technologies, such as HTML, CSS and cognos/manyeyes/page/Phrase_Net.html JavaScript. The UI communicates via AJAX with 2http://infomotions.com/sandbox/ a backend, which utilizes PHP and a MySQL network-diagrams/bin/walden/ 3http://bncweb.lancs.ac.uk/bncwebSignup/ 6http://wortschatz.uni-leipzig.de/ user/login.php ˜cbiemann/software/TinyCC2.html, (Richter et 4http://cwb.sourceforge.net al., 2006) 5http://corpora.uni-leipzig.de/ 7corresponding to 5% error probability 62 database. This makes the approach flexible regard- ing their part-of-speech tags (as described in Sec- ing the platform. It can run on client computers tion 2.2). The graph drawing panel visualizes the using XAMP8, a portable package of various Web queried term and its significant co-occurrences resp. technologies, including an Apache web server and a concordances, significancy being visualized by the MySQL server. Alternatively, the tool can operate as thickness of the lines. In Figure 1, showing the con- a client-server application over a network. In partic- cordances for bread, we can directly see words that ular, we want to highlight the JavaScript data visual- occur often with bread in the context: E.g. bread ization framework D3 (Bostock et al., 2011), which is often used in combination with butter, cheese, was used to layout and draw the graphs. We deliber- margarine (offset +2), but also the kind of different ately designed the tool to match the requirements of breads is described by the adjectives at offset -1. For literary researchers, who are at times overwhelmed the same information, using the normal KWIC view, by general-purpose visualisation tools such as e.g. one has to count the words with different offset by Gephi9. The UI is split into three parts: At the top a hand to find properties for the term bread. At the menu bar, including a search input field and search bottom, the maximal number of words shown in the options, a graph drawing panel and display options graph can be specified. For the concordances display at the bottom of the page. there is an additional option to specify the maximal offset. The original text (with provenance informa- tion) containing the nodes (words) or edges (word pairs) shown in the graph can be retrieved by either clicking on a word itself or on the edge connecting two words, in a new window (see Figure 2) within the application.
Load more

Recommended publications
  • Unit 1: Introduction
    Corpus building and investigation for the Humanities: An on-line information pack about corpus investigation techniques for the Humanities Unit 1: Introduction David Evans, University of Nottingham 1.1 What a corpus is A corpus is defined here as a principled collection of naturally occurring texts which are stored on a computer to permit investigation using special software. A corpus is principled because texts are selected for inclusion according to pre-defined research purposes. Usually texts are included on external rather than internal criteria. For example, a researcher who wants to investigate metaphors used in university lectures will attempt to collect a representative sample of lectures across a number of disciplines, rather than attempting to collect lectures that include a lot of figurative language. Most commercially available corpora are made up of samples of a particular language variety which aim to be representative of that variety. Here are some examples of some of the different types of corpora and how they represent a particular variety: General corpora An example of a general corpus is the British National Corpus which “… aims to represent the universe of contemporary British English [and] to capture the full range of varieties of language use.” (Aston & Burnard 1998: 5). As a result of this aim the corpus is very large (containing some 100 million words) and contains a balance of texts from a wide variety of different domains of spoken and written language. Large general corpora are sometimes referred to as reference corpora because they are often used as a baseline against which judgements about the language varieties held in more specialised corpora can be made.
    [Show full text]
  • Cambridge Handbook of English Corpus Linguistics Chapter 2: Computational Tools and Methods for Corpus Compilation and Analysis1
    Cambridge Handbook of English Corpus Linguistics Chapter 2: Computational Tools and Methods for Corpus Compilation and Analysis1 Paul Rayson UCREL, Lancaster University 1. Introduction The growing interest in corpus linguistics methods in the 1970s and 1980s was largely enabled by the increased power of computers and the use of computational methods to store and process language samples. Before this, even simple methods for studying language such as extracting a list of all the different words in a text and their immediate contexts was incredibly time consuming and costly in terms of human effort. Only concordances of books of special importance such as the Qur’an, the Bible and the works of Shakespeare were made before the 20th century and required either a large number of scholars or monks or a significant investment in time by a single individual, in some cases more than ten years of their lives. In these days of web search engines and vast quantities of text that is available at our finger tips, the end user would be mildly annoyed if a concordance from a one billion word corpus took more than five seconds to be displayed. Other text rich disciplines can trace their origins back to the same computing revolution. Digital Humanities scholars cite the work of Roberta Busa working with IBM in 1949 who produced his Index Thomisticus, a computer-generated concordance to the writings of Thomas Aquinas. Similarly, lexicographers in the 19th century used millions of handwritten cards or quotation slips but the field was revolutionised in the 1980s with the creation of machine-readable corpora such as COBUILD and the use of computers for searching and finding patterns in the data.
    [Show full text]
  • PDF of Volume 33
    BULLETIN OF THE INTERNATIONAL ORGANIZATION FOR SEPTUAGINT AND COGNATE STUDIES Volume 33 Fall, 2000 Minutes of the loses Meeting, Boston Treasurer's Report 6 News and Notes 8 Record of Work Published or in Progress 11 Varia 24 Review of Hatch and Redpath's Concordance to the Septuagint, 2nd edition 31 Johann Cook Web Review: The Christian Classics Ethereal Library, at a!. 36 Frederick Knobloch A Note on the Syntactic Analysis of Greek Translations and Compositions 39 T. P. Hutchinson and D. Cairns How to Analyse and Translate the Idiomatic Phrase 1l"l~ '0 47 Takamitsu Muraoka Evaluating Lexical Consistency in the Old Greek Bible 53 Martha L. Wade Translation Technique in LXX Genesis and Its Implications for the NETS Version 76 Robert J. V. Hiebert PROGRAM FOR THE loses MEETING BULLETIN of the loses IN BOSTON, NOVEMBER 20-23, 1999 Published Annually Each Fall by THE INTERNATIONAL ORGANIZATION FOR Sunday, November 21 SEPTUAGINT AND COGNATE STUDIES OFFICERS AND EXECUTIVE COMMITTEE 1:00-3:30 pm H-Room 103 President Honorary Presidents Albert Pietersma, University of Toronto, Presiding Johan Lust John Wm. Wevers (Toronto) Katholieke Univ. leuven Albert Pietersma (Toronto) St. Michielsstraat' 2 Eugene Ulrich (Notre Dame) Alison Salvesen, Oxford University 8-3000 Leuven Belgium Jacob of Edessa 's Version of the Books of Samuel and the [email protected] Textual Criticism ofthe Septuagint Immediate Past President Wee President Leonard J. Greenspoon Benjamin G. Wright Jan Willem Van Henten, University of Amsterdam Creighton University lehigh University Omaha, NE 68178 U.SA Bethlehem, PA 18015 U.SA The Honorary Decreefor Simon the Maccabee (1 Macc 14:25- [email protected] [email protected] 49) in its Hellenistic Context Editor NETS Co-chairs Theodore A Bergren Albert Pietersma (Toronto) Harry F.
    [Show full text]
  • A Sense Discrimination Engine for English, Chinese and Other Languages
    Sketch Engine: a sense discrimination engine for English, Chinese and other languages Adam Kilgarriff1, Pavel Rychlý2, Simon Smith3, Chu-Ren Huang4, Yiching Wu5, Cecilia Lin3 [email protected] 1. Background: corpora and concordances Analysis of text and spoken language, for the purposes of second language teaching, dictionary making and other linguistic applications, used to be based on the intuitions of linguists and lexicographers. The compilation of dictionaries and thesauri, for example, required that the compiler read very widely, and record the results of his efforts – the definitions and different senses of words – on thousands, or millions of index cards. Today’s approach to linguistic analysis generally involves the use of linguistic corpora: large databases of spoken or written language samples, defined by Crystal (1991) as “A collection of linguistic data, either written texts or a transcription of recorded speech, which can be used as a starting-point of linguistic description or as a means of verifying hypotheses about a language”. Numerous large corpora have been assembled for English, including the British National Corpus (BNC) and the Bank of English. Dictionaries published by the Longman Group are based on the 100 million word BNC, and corpora are routinely used by computational linguists in tasks such as machine translation and speech recognition. The BNC is an example of a balanced corpus, in that it attempts to represent a broad cross-section of genres and styles, including fiction and non-fiction, books, periodicals and newspapers, and even essays by students. Transcriptions of spoken data are also included; and this is the corpus that is used with the English Sketch Engine.
    [Show full text]
  • The BNC Handbook Exploring the British National Corpus with SARA
    The BNC Handbook Exploring the British National Corpus with SARA Guy Aston and Lou Burnard September 1997 ii Preface The British National Corpus is a collection of over 4000 samples of modern British English, both spoken and written, stored in electronic form and selected so as to reflect the widest possible variety of users and uses of the language. Totalling over 100 million words, the corpus is currently being used by lex- icographers to create dictionaries, by computer scientists to make machines ‘understand’ and produce natural language, by linguists to describe the English language, and by language teachers and students to teach and learn it — to name but a few of its applications. Institutions all over Europe have purchased the BNC and installed it on computers for use in their research. However it is not necessary to possess a copy of the corpus in order to make use of it: it can also be consulted via the Internet, using either the World Wide Web or the SARA software system, which was developed specially for this purpose. The BNC Handbook provides a comprehensive guide to the SARA software distributed with the corpus and used on the network service. It illustrates some of the ways in which it is possible to find out about contemporary English usage from the BNC, and aims to encourage use of the corpus by a wide and varied public. We have tried as far as possible to avoid jargon and unnecessary technicalities; the book assumes nothing more than an interest in language and linguistic problems on the part of its readers.
    [Show full text]
  • Latent Semantic Analysis, Corpus Stylistics and Machine Learning
    Latent Semantic Analysis, Corpus stylistics and Machine Learning Stylometry for Translational and Authorial Style Analysis: The Case of Denys Johnson-Davies’ Translations into English A dissertation submitted to Kent State University in partial fulfillment of the requirements for the degree of Doctor of Philosophy by Mohammed Al Batineh May, 2015 © Copyright by Mohammed S. Al-Batineh All Rights Reserved Dissertation written by Mohammed Al Batineh BA., Yarmouk University, Jordan, 2008 MA., Yarmouk University, Jordan, 2010 APPROVED BY __________________________, Chair, Doctoral Dissertation Committee Dr. Françoise Massardier-Kenney (advisor) __________________________, Members, Doctoral Dissertation Committee Dr. Carol Maier __________________________, Dr. Gregory M. Shreve __________________________, Dr. Jonathan I. Maletic __________________________, Dr. Katherine Rawson ACCEPTED BY __________________________, Interim Chair, Modern and Classical Language Studies Dr. Keiran J Dunne __________________________, Dean, College of Arts and Sciences Dr. James L. Blank TABLE OF CONTENTS LIST OF FIGURES ........................................................................................... viii LIST OF TABLES ............................................................................................... ix DEDICATION ...................................................................................................... x ABSTRACT ........................................................................................................ xii CHAPTER
    [Show full text]