DOCSLIB.ORG

Supervised Learning of a Probabilistic Lexicon of Verb Semantic Classes

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Supervised Learning of a Probabilistic Lexicon of Verb Semantic Classes Supervised Learning of a Probabilistic Lexicon of Verb Semantic Classes Yusuke Miyao Jun’ichi Tsujii University of Tokyo University of Tokyo Hongo 7-3-1, Bunkyo-ku, Tokyo, Japan University of Manchester [email protected] National Center for Text Mining Hongo 7-3-1, Bunkyo-ku, Tokyo, Japan [email protected] Abstract al., 2005; Li and Brew, 2007; Abend et al., 2008). In other words, those methods are heavily de- The work presented in this paper explores pendent on the availability of a semantic lexicon. a supervised method for learning a prob- Therefore, recent research efforts have invested in abilistic model of a lexicon of VerbNet developing semantic resources, such as WordNet classes. We intend for the probabilis- (Fellbaum, 1998), FrameNet (Baker et al., 1998), tic model to provide a probability dis- and VerbNet (Kipper et al., 2000; Kipper-Schuler, tribution of verb-class associations, over 2005), which greatly advanced research in seman- known and unknown verbs, including pol- tic processing. However, the construction of such ysemous words. In our approach, train- resources is expensive, and it is unrealistic to pre- ing instances are obtained from an ex- suppose the availability of full-coverage lexicons; isting lexicon and/or from an annotated this is the case because unknown words always ap- corpus, while the features, which repre- pear in real texts, and word-semantics associations sent syntactic frames, semantic similarity, may vary (Abend et al., 2008). and selectional preferences, are extracted This paper explores a method for the supervised from unannotated corpora. Our model learning of a probabilistic model for the VerbNet is evaluated in type-level verb classifica- lexicon. We target the automatic classification of tion tasks: we measure the prediction ac- arbitrary verbs, including polysemous verbs, into curacy of VerbNet classes for unknown all VerbNet classes; further, we target the esti- verbs, and also measure the dissimilarity mation of a probabilistic model, which represents between the learned and observed proba- the saliences of verb-class associations for polyse- bility distributions. We empirically com- mous verbs. In our approach, an existing lexicon pare several settings for model learning, and/or an annotated corpus are used as the training while we vary the use of features, source data. Since VerbNet classes are designed to rep- corpora for feature extraction, and disam- resent the distinctions in the syntactic frames that biguated corpora. In the task of verb clas- verbs can take, features, representing the statistics sification into all VerbNet classes, our best of syntactic frames, are extracted from the unan- model achieved a 10.69% error reduction notated corpora. Additionally, as the classes rep- in the classification accuracy, over the pre- resent semantic commonalities, semantically in- viously proposed model. spired features, like distributionally similar words, are used. These features can be considered as a 1 Introduction generalized representation of verbs, and we ex- Lexicons are invaluable resources for semantic pect that the obtained probabilistic model predicts processing. In many cases, lexicons are neces- VerbNet classes of the unknown words. sary to restrict a set of semantic classes to be as- Our model is evaluated in two tasks of type- signed to a word. In fact, a considerable number of level verb classification: one is the classification works on semantic processing implicitly or explic- of monosemous verbs into a small subset of the itly presupposes the availability of a lexicon, such classes, which was studied in some previous works as in word sense disambiguation (WSD) (Mc- (Joanis and Stevenson, 2003; Joanis et al., 2008). Carthy et al., 2004), and in token-level verb class The other task is the classification of all verbs into disambiguation (Lapata and Brew, 2004; Girju et the full set of VerbNet classes, which has not yet 1328 Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1328–1337, Singapore, 6-7 August 2009. c 2009 ACL and AFNLP been attempted. In the experiments, training in- 43 Emission stances are obtained from VerbNet and/or Sem- 43.1 Light Emission beam, glow, sparkle,... Link (Loper et al., 2007), while features are ex- 43.2 Sound Emission tracted from the British National Corpus or from blare, chime, jangle,... Wall Street Journal. We empirically compare sev- ... 44 Destroy eral settings for model learning by varying the annihilate, destroy, ravage,... set of features, the source domain and the size 45 Change of State ... of a corpus for feature extraction, and the use of 47 Existence the token-level statistics obtained from a manually 47.1 Exist disambiguated corpus. We also provide the anal- exist, persist, remain,... 47.2 Entity-Specific Modes Being ysis of the remaining errors, which will lead us to bloom, breathe, foam,... further improve the supervised learning of a prob- 47.3 Modes of Being with Motion abilistic semantic lexicon. jiggle, sway, waft,... ... Supervised methods for automatic verb classifi- cation have been extensively investigated (Steven- son et al., 1999; Stevenson and Merlo, 1999; Figure 1: VerbNet classes Merlo and Stevenson, 2001; Stevenson and Joa- 43.2 Sound Emission nis, 2003; Joanis and Stevenson, 2003; Joanis et Theme V al., 2008). However, their focus has been lim- Theme V P:loc Location ited to a small subset of verb classes, and a lim- P:loc Location V Theme there V Theme P:loc Location ited number of monosemous verbs. The main con- Agent V Theme tributions of the present work are: i) to provide Theme V Oblique empirical results for the automatic classification Location V with Theme of all verbs, including polysemous ones, into all 47.3 Modes of Being with Motion VerbNet classes, and ii) to empirically explore the Theme V Theme V P:loc Location effective settings for the supervised learning of a P:loc Location V Theme probabilistic lexicon of verb semantic classes. there V Theme Agent V Theme 2 Background Figure 2: Syntactic frames for VerbNet classes 2.1 Verb lexicon Levin’s (1993) work on verb classification has broadened the field of computational research that sification. The classes therefore cover more En- concerns the relationships between the syntactic glish verbs, and the classification should be more and semantic structures of verbs. The principal consistent (Korhonen and Briscoe, 2004; Kipper idea behind the work is that the meanings of verbs et al., 2006). can be identified by observing possible syntactic The current version of VerbNet includes 270 frames that the verbs can take. In other words, classes.1 Figure 1 shows a part of the classes of with the knowledge of syntactic frames, verbs can VerbNet. The top-level categories, e.g. Emis- be semantically classified. This idea provided the sion and Destroy, represent a coarse classifica- computational linguistics community with crite- tion of verb semantics. They are further classi- ria for the definition and the classification of verb fied into verb classes, each of which expresses semantics; it has subsequently resulted in the re- a group of verbs sharing syntactic frames. Fig- search of the induction of verb classes (Korhonen ure 2 shows an excerpt from VerbNet, which rep- and Briscoe, 2004), and the construction of a verb resents the possible syntactic frames for the Sound lexicon based on Levin’s criteria. Emission class, including “chime” and “jangle,” VerbNet (Kipper et al., 2000; Kipper-Schuler, and the Modes of Being with Motion class, in- 2005) is a lexicon of verbs organized into classes cluding “jiggle” and “waft.” In this figure, each that share the same syntactic behaviors and seman- line represents a syntactic frame, where Agent, tics. The design of classes originates from Levin 1Throughout this paper, we refer to VerbNet 2.3. Sub- (1993), though the design has been considerably classes are ignored in this work, following the setting of reorganized and extends beyond the original clas- Abend et al. (2008). 1329 . the walls still shook;VN=47.3 and an evacuation (2008) focused on 14 classes and 835 verbs. Al- alarm blared;VN=43.2 outside. though these works provided a theoretical frame- Suddenly the woman begins;VN=55.1 swaying work for supervised verb classification, their re- ;VN=47.3 and then . sults were not readily available for practical ap- plications, because of the limitation in the cover- Figure 3: An excerpt from SemLink age of the targeted classes/verbs on real texts. On the contrary, we target the classification of arbi- Theme, and Location indicate the thematic trary verbs, including polysemous verbs, into all roles, V denotes a verb, and P specifies a prepo- VerbNet classes (270 in total). In this realistic sit- sition. P:loc defines locative prepositions such uation, we will empirically compare settings for as: “in” and “at.” For example, the second syn- model learning, in order to explore effective con- tactic frame of Sound Emission, i.e., Theme V ditions to obtain better models. P:loc Location, corresponds to the follow- Another difference from the aforementioned ing sentence: works is that we aim at obtaining a probabilis- tic model, which represents saliences of classes 1. The coins jangled in my pocket. of polysemous verbs. Lapata and Brew (2004) Theme corresponds to “the coins,” V to “jangled,” and Li and Brew (2007) focused on this issue, P:loc to “in,” and Location to “my pocket.” and described methods for inducing probabilities While VerbNet provides associations between of verb-class associations. The obtained proba- verbs and semantic classes, SemLink (Loper et bilistic model was intended to be incorporated into al., 2007) additionally provides mappings among a token-level disambiguation model. Their meth- VerbNet, FrameNet (Baker et al., 1998), PropBank ods claimed to be unsupervised, meaning that the (Palmer et al., 2005), and WordNet (Fellbaum, induction of a probabilistic lexicon did not re- 1998).
Load more

Recommended publications
  • Talk Bank: a Multimodal Database of Communicative Interaction
    Talk Bank: A Multimodal Database of Communicative Interaction 1. Overview The ongoing growth in computer power and connectivity has led to dramatic changes in the methodology of science and engineering. By stimulating fundamental theoretical discoveries in the analysis of semistructured data, we can to extend these methodological advances to the social and behavioral sciences. Specifically, we propose the construction of a major new tool for the social sciences, called TalkBank. The goal of TalkBank is the creation of a distributed, web- based data archiving system for transcribed video and audio data on communicative interactions. We will develop an XML-based annotation framework called Codon to serve as the formal specification for data in TalkBank. Tools will be created for the entry of new and existing data into the Codon format; transcriptions will be linked to speech and video; and there will be extensive support for collaborative commentary from competing perspectives. The TalkBank project will establish a framework that will facilitate the development of a distributed system of allied databases based on a common set of computational tools. Instead of attempting to impose a single uniform standard for coding and annotation, we will promote annotational pluralism within the framework of the abstraction layer provided by Codon. This representation will use labeled acyclic digraphs to support translation between the various annotation systems required for specific sub-disciplines. There will be no attempt to promote any single annotation scheme over others. Instead, by promoting comparison and translation between schemes, we will allow individual users to select the custom annotation scheme most appropriate for their purposes.
    [Show full text]
  • Collection of Usage Information for Language Resources from Academic Articles
    Collection of Usage Information for Language Resources from Academic Articles Shunsuke Kozaway, Hitomi Tohyamayy, Kiyotaka Uchimotoyyy, Shigeki Matsubaray yGraduate School of Information Science, Nagoya University yyInformation Technology Center, Nagoya University Furo-cho, Chikusa-ku, Nagoya, 464-8601, Japan yyyNational Institute of Information and Communications Technology 4-2-1 Nukui-Kitamachi, Koganei, Tokyo, 184-8795, Japan fkozawa,[email protected], [email protected], [email protected] Abstract Recently, language resources (LRs) are becoming indispensable for linguistic researches. However, existing LRs are often not fully utilized because their variety of usage is not well known, indicating that their intrinsic value is not recognized very well either. Regarding this issue, lists of usage information might improve LR searches and lead to their efficient use. In this research, therefore, we collect a list of usage information for each LR from academic articles to promote the efficient utilization of LRs. This paper proposes to construct a text corpus annotated with usage information (UI corpus). In particular, we automatically extract sentences containing LR names from academic articles. Then, the extracted sentences are annotated with usage information by two annotators in a cascaded manner. We show that the UI corpus contributes to efficient LR searches by combining the UI corpus with a metadata database of LRs and comparing the number of LRs retrieved with and without the UI corpus. 1. Introduction Thesaurus1. In recent years, such language resources (LRs) as corpora • He also employed Roget’s Thesaurus in 100 words of and dictionaries are being widely used for research in the window to implement WSD.
    [Show full text]
  • From CHILDES to Talkbank
    From CHILDES to TalkBank Brian MacWhinney Carnegie Mellon University MacWhinney, B. (2001). New developments in CHILDES. In A. Do, L. Domínguez & A. Johansen (Eds.), BUCLD 25: Proceedings of the 25th annual Boston University Conference on Language Development (pp. 458-468). Somerville, MA: Cascadilla. a similar article appeared as: MacWhinney, B. (2001). From CHILDES to TalkBank. In M. Almgren, A. Barreña, M. Ezeizaberrena, I. Idiazabal & B. MacWhinney (Eds.), Research on Child Language Acquisition (pp. 17-34). Somerville, MA: Cascadilla. Recent years have seen a phenomenal growth in computer power and connectivity. The computer on the desktop of the average academic researcher now has the power of room-size supercomputers of the 1980s. Using the Internet, we can connect in seconds to the other side of the world and transfer huge amounts of text, programs, audio and video. Our computers are equipped with programs that allow us to view, link, and modify this material without even having to think about programming. Nearly all of the major journals are now available in electronic form and the very nature of journals and publication is undergoing radical change. These new trends have led to dramatic advances in the methodology of science and engineering. However, the social and behavioral sciences have not shared fully in these advances. In large part, this is because the data used in the social sciences are not well- structured patterns of DNA sequences or atomic collisions in super colliders. Much of our data is based on the messy, ill-structured behaviors of humans as they participate in social interactions. Categorizing and coding these behaviors is an enormous task in itself.
    [Show full text]
  • The British National Corpus Revisited: Developing Parameters for Written BNC2014 Abi Hawtin (Lancaster University, UK)
    The British National Corpus Revisited: Developing parameters for Written BNC2014 Abi Hawtin (Lancaster University, UK) 1. The British National Corpus 2014 project The ESRC Centre for Corpus Approaches to Social Science (CASS) at Lancaster University and Cambridge University Press are working together to create a new, publicly accessible corpus of contemporary British English. The corpus will be a successor to the British National Corpus, created in the early 1990s (BNC19941). The British National Corpus 2014 (BNC2014) will be of the same order of magnitude as BNC1994 (100 million words). It is currently projected that the corpus will reach completion in mid-2018. Creation of the spoken sub-section of BNC2014 is in its final stages; this paper focuses on the justification and development of the written sub- section of BNC2014. 2. Justification for BNC2014 There are now many web-crawled corpora of English, widely available to researchers, which contain far more data than will be included in Written BNC2014. For example, the English TenTen corpus (enTenTen) is a web-crawled corpus of Written English which currently contains approximately 19 billion words (Jakubíček et al., 2013). Web-crawling processes mean that this huge amount of data can be collected very quickly; Jakubíček et al. (2013: 125) note that “For a language where there is plenty of material available, we can gather, clean and de-duplicate a billion words a day.” So, with extremely large and quickly-created web-crawled corpora now becoming commonplace in corpus linguistics, the question might well be asked why a new, 100 million word corpus of Written British English is needed.
    [Show full text]
  • Comprehensive and Consistent Propbank Light Verb Annotation Claire Bonial,1 Martha Palmer2 1U.S
    Comprehensive and Consistent PropBank Light Verb Annotation Claire Bonial,1 Martha Palmer2 1U.S. Army Research Laboratory, 2University of Colorado, Boulder 12800 Powder Mill Rd., Adelphi, MD, USA E-mail: [email protected], [email protected] Abstract Recent efforts have focused on expanding the annotation coverage of PropBank from verb relations to adjective and noun relations, as well as light verb constructions (e.g., make an offer, take a bath). While each new relation type has presented unique annotation challenges, ensuring consistent and comprehensive annotation of light verb constructions has proved particularly challenging, given that light verb constructions are semi-productive, difficult to define, and there are often borderline cases. This research describes the iterative process of developing PropBank annotation guidelines for light verb constructions, the current guidelines, and a comparison to related resources. Keywords: light verb constructions, semantic role labeling, NLP resources 2. PropBank Background 1. Introduction PropBank annotation consists of two tasks: sense annotation and role annotation. The PropBank lexicon The goal of PropBank (Palmer et al., 2005) is to supply provides a listing of the coarse-grained senses of a verb, consistent, general-purpose labeling of semantic roles noun, or adjective relation, and the set of roles associated across different syntactic realizations. With over two with each sense (thus, called a “roleset”). The roles are million words from diverse genres, the benchmark listed as argument numbers (Arg0 – Arg6) and annotated corpus supports the training of automatic correspond to verb-specific roles. For example: semantic role labelers, which in turn support other Natural Language Processing (NLP) areas, such as Offer-01 (transaction, proposal): machine translation.
    [Show full text]
  • 3 Corpus Tools for Lexicographers
    Comp. by: pg0994 Stage : Proof ChapterID: 0001546186 Date:14/5/12 Time:16:20:14 Filepath:d:/womat-filecopy/0001546186.3D31 OUP UNCORRECTED PROOF – FIRST PROOF, 14/5/2012, SPi 3 Corpus tools for lexicographers ADAM KILGARRIFF AND IZTOK KOSEM 3.1 Introduction To analyse corpus data, lexicographers need software that allows them to search, manipulate and save data, a ‘corpus tool’. A good corpus tool is the key to a comprehensive lexicographic analysis—a corpus without a good tool to access it is of little use. Both corpus compilation and corpus tools have been swept along by general technological advances over the last three decades. Compiling and storing corpora has become far faster and easier, so corpora tend to be much larger than previous ones. Most of the first COBUILD dictionary was produced from a corpus of eight million words. Several of the leading English dictionaries of the 1990s were produced using the British National Corpus (BNC), of 100 million words. Current lexico- graphic projects we are involved in use corpora of around a billion words—though this is still less than one hundredth of one percent of the English language text available on the Web (see Rundell, this volume). The amount of data to analyse has thus increased significantly, and corpus tools have had to be improved to assist lexicographers in adapting to this change. Corpus tools have become faster, more multifunctional, and customizable. In the COBUILD project, getting concordance output took a long time and then the concordances were printed on paper and handed out to lexicographers (Clear 1987).
    [Show full text]
  • Università Degli Studi Di Macerata
    UNIVERSITÀ DEGLI STUDI DI MACERATA Dipartimento di Studi Umanistici – Lingue, Mediazione, Storia, Lettere, Filosofia Corso di Laurea Magistrale in Lingue Moderne per la Comunicazione e la Cooperazione Internazionale (ClasseLM-38) Traduzione per laComunicazione Internazionale – inglese -mod. B STRUMENTI E TECNOLOGIE PER LA TRADUZIONESPECIALISTICA 1 What is a corpus? Some (authoritative) definitions • “a collection of naturally-occurring language text, chosen to characterize a state or variety of a language” (Sinclair, 1991:171) • “a collection of texts assumed to be representative of a given language, dialect, or other subset of a language, to be used for linguistic analysis” (Francis, 1992:7) • “a closed set of texts in machine-readable form established for general or specific purposes by previously defined criteria” (Engwall, 1992:167) • “a finite-sized body of machine-readable text, sampled in order to be maximally representative of the language variety under consideration” (McEnery & Wilson, 1996:23) • “a collection of (1) machine-readable (2) authentic texts […] which is (3) sampled to be (4) representative of a particular language or language variety” (McEnery et al., 2006:5) What is / is not a corpus…? • A newspaper archive on CD-ROM? The answer is • An online glossary? always “NO” • A digital library (e.g. Project (see Gutenberg)? definition) • All RAI 1 programmes (e.g. for spoken TV language) Corpora vs. web •Corpora: – Usually stable •searches can be replicated – Control over contents •we can select the texts to be included, or have control over selection strategies – Ad-hoc linguistically-aware software to investigate them •concordancers can sort / organise concordance lines • Web (as accessed via Google or other search engines): – Very unstable •results can change at any time for reasons beyond our control – No control over contents •what/how many texts are indexed by Google’s robots? – Limited control over search results •cannot sort or organise hits meaningfully; they are presented randomly Click here for another corpus vs.
    [Show full text]
  • Exploring Methods and Resources for Discriminating Similar Languages
    Exploring Methods and Resources for Discriminating Similar Languages Marco Lui♥♣, Ned Letcher♥, Oliver Adams♥, Long Duong♥♣, Paul Cook♥ and Timothy Baldwin♥♣ ♥ Department of Computing and Information Systems The University of Melbourne ♣ NICTA Victoria [email protected], [email protected], [email protected], [email protected], [email protected], [email protected] Abstract The Discriminating between Similar Languages (DSL) shared task at VarDial challenged partici- pants to build an automatic language identification system to discriminate between 13 languages in 6 groups of highly-similar languages (or national varieties of the same language). In this paper, we describe the submissions made by team UniMelb-NLP, which took part in both the closed and open categories. We present the text representations and modeling techniques used, including cross-lingual POS tagging as well as fine-grained tags extracted from a deep grammar of English, and discuss additional data we collected for the open submissions, utilizing custom- built web corpora based on top-level domains as well as existing corpora. 1 Introduction Language identification (LangID) is the problem of determining what natural language a document is written in. Studies in the area often report high accuracy (Cavnar and Trenkle, 1994; Dunning, 1994; Grefenstette, 1995; Prager, 1999; Teahan, 2000). However, recent work has shown that high accu- racy is only achieved under ideal conditions (Baldwin and Lui, 2010), and one area that needs further work is accurate discrimination between closely-related languages (Ljubesiˇ c´ et al., 2007; Tiedemann and Ljubesiˇ c,´ 2012). The problem has been explored for specific groups of confusable languages, such as Malay/Indonesian (Ranaivo-Malancon, 2006), South-Eastern European languages (Tiedemann and Ljubesiˇ c,´ 2012), as well as varieties of English (Lui and Cook, 2013), Portuguese (Zampieri and Gebre, 2012), and Spanish (Zampieri et al., 2013).
    [Show full text]
  • FERSIWN GYMRAEG ISOD the National Corpus of Contemporary
    FERSIWN GYMRAEG ISOD The National Corpus of Contemporary Welsh Project Report, October 2020 Authors: Dawn Knight1, Steve Morris2, Tess Fitzpatrick2, Paul Rayson3, Irena Spasić and Enlli Môn Thomas4. 1. Introduction 1.1. Purpose of this report This report provides an overview of the CorCenCC project and the online corpus resource that was developed as a result of work on the project. The report lays out the theoretical underpinnings of the research, demonstrating how the project has built on and extended this theory. We also raise and discuss some of the key operational questions that arose during the course of the project, outlining the ways in which they were answered, the impact of these decisions on the resource that has been produced and the longer-term contribution they will make to practices in corpus-building. Finally, we discuss some of the applications and the utility of the work, outlining the impact that CorCenCC is set to have on a range of different individuals and user groups. 1.2. Licence The CorCenCC corpus and associated software tools are licensed under Creative Commons CC-BY-SA v4 and thus are freely available for use by professional communities and individuals with an interest in language. Bespoke applications and instructions are provided for each tool (for links to all tools, refer to section 10 of this report). When reporting information derived by using the CorCenCC corpus data and/or tools, CorCenCC should be appropriately acknowledged (see 1.3). § To access the corpus visit: www.corcencc.org/explore § To access the GitHub site: https://github.com/CorCenCC o GitHub is a cloud-based service that enables developers to store, share and manage their code and datasets.
    [Show full text]
  • ESP Corpus Construction: a Plea for a Needs-Driven Approach Bâtir Des Corpus En Anglais De Spécialité : Plaidoyer Pour Une Approche Fondée Sur L’Analyse Des Besoins
    ASp la revue du GERAS 68 | 2015 Varia ESP corpus construction: a plea for a needs-driven approach Bâtir des corpus en anglais de spécialité : plaidoyer pour une approche fondée sur l’analyse des besoins Hilary Nesi Electronic version URL: http://journals.openedition.org/asp/4682 DOI: 10.4000/asp.4682 ISSN: 2108-6354 Publisher Groupe d'étude et de recherche en anglais de spécialité Printed version Date of publication: 23 October 2015 Number of pages: 7-24 ISSN: 1246-8185 Electronic reference Hilary Nesi, « ESP corpus construction: a plea for a needs-driven approach », ASp [Online], 68 | 2015, Online since 01 November 2016, connection on 02 November 2020. URL : http:// journals.openedition.org/asp/4682 ; DOI : https://doi.org/10.4000/asp.4682 This text was automatically generated on 2 November 2020. Tous droits réservés ESP corpus construction: a plea for a needs-driven approach 1 ESP corpus construction: a plea for a needs-driven approach Bâtir des corpus en anglais de spécialité : plaidoyer pour une approche fondée sur l’analyse des besoins Hilary Nesi 1. Introduction 1 A huge amount of corpus data is now available for English language teachers and learners, making it difficult for them to decide which kind of corpus is most suited to their needs. The corpora that they probably know best are the large, publicly accessible general corpora which inform general reference works and textbooks; the ones that aim to cover the English language as a whole and to represent many different categories of text. The 100 million word British National Corpus (BNC), created in the 1990s, is a famous example of this type and has been used extensively for language learning purposes, but at 100 million words it is too small to contain large numbers of texts within each of its subcomponents, and it is now too old to represent modern usage in the more fast-moving fields.
    [Show full text]
  • Setting up for Corpus Lexicography
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE Setting up for Corpus Lexicography Adam Kilgarriff,* Jan Pomikalek,* Miloš Jakubíček*‡, & Pete Whitelock† *Lexical Computing Ltd; ‡Masaryk University, Brno; †Oxford University Press Keywords: corpora, corpus lexicography, web crawling, dependency parsing. Abstract There are many benefits to using corpora. In order to reap those rewards, how should someone who is setting up a dictionary project proceed? We describe a practical experience of such „setting up‟ for a new Portuguese-English, English-Portuguese dictionary being written at Oxford University Press. We focus on the Portuguese side, as OUP did not have Portuguese resources prior to the project. We collected a very large (3.5 billion word) corpus from the web, including removing all unwanted material and duplicates. We then identified the best tools for Portuguese for lemmatizing and parsing, and undertook the very large task of parsing it. We then used the dependency parses, as output by the parser, to create word sketches (one page summaries of a word‟s grammatical and collocational behavior). We plan to customize an existing system for automatically identifying good candidate dictionary examples, to Portuguese, and add salient information about regional words to the word sketches. All of the data and associated support tools for lexicography are available to the lexicographer in the Sketch Engine corpus query system. 1. Introduction There are a number of ways in which corpus technology can support lexicography, as described in Rundell & Kilgarriff (2011). It can make it more accurate, more consistent, and faster. But how might those potential benefits pan out in an actual project? If starting from a blank sheet of paper, how should one proceed? In this paper we describe such an exercise.
    [Show full text]
  • The Manually Annotated Sub-Corpus
    Case study: The Manually Annotated Sub-Corpus Nancy Ide Abstract This case study describes the creation process for the Manually Anno- tated Sub-Corpus (MASC), a 500,000 word subset of the Open American National Corpus (OANC). The corpus includes primary data from a balanced selection of 19 written and spoken genres, all of which is annotated for almost 20 varieties of linguistic phenomena at all levels. All annotations are either hand-validated or manually-produced. MASC is unique in that it is fully open and free for any use, including commercial use. 1 Introduction This case study describes the creation process for the Manually Annotated Sub- Corpus (MASC), which is a subset of the Open American National Corpus (OANC). The OANC is itself a subset of the American National Corpus (ANC). Each of these corpora represents a distinct evolutionary stage in our approach to corpus- building and delivery that reflect adaptations to both changing community needs and advances in best practices for creating and representing linguistically annotated corpora. We therefore describe the procedures involved in producing the ANC and OANC before focusing on MASC, which is the jewel in the crown of corpora pro- duced by the ANC project. 2 Background: The ANC The ANC was motivated by developers of major linguistic resources such as FrameNet [2] and Nomlex [24], who had been extracting usage examples from Nancy Ide Vassar College, Poughkeepsie, New York 12604-0520 USA e-mail: [email protected] 1 2 Nancy Ide the 100 million-word British National Corpus (BNC), the largest corpus of English across several genres that was available at the time.
    [Show full text]