DOCSLIB.ORG

Part-Of-Speech Tagging • Lit Review Part 2 • Written Review of 2 Articles, Due April 1

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Part-Of-Speech Tagging • Lit Review Part 2 • Written Review of 2 Articles, Due April 1 Announcements Part-of-Speech Tagging • Lit Review Part 2 • Written review of 2 articles, due April 1 • Final Project Proposal CS 341: Natural Language Processing Prof. Heather Pon-Barry • Due Monday April 6 www.mtholyoke.edu/courses/ponbarry/cs341.html Today POS Tagging • Process of assigning part of speech marker to each word in a collection ! She/pronoun ! • POS Tagging found/verb ! herself/pronoun ! falling/verb ! ... POS Tagging Penn Treebank Tagset • Words often have more than one POS: e.g., back • The back door = adjective (JJ) • On my back = noun (NN) • Win the voters back = adverb (RB) • Promised to back the bill = verb (VB) • The POS tagging problem is to determine the POS tag for a particular instance of a word. Applications POS Tagging Performance • Speech synthesis • How many tags are correct? (Tag accuracy) • State of the art: about 97% • “I object” vs. “This object...” • But baseline is already 90% • Baseline is performance is: • Parsing • Tag every word with its most frequent tag • Machine translation • Tag unknown words as nouns • Partly easy because • Named entity recognition • Many words are unambiguous • Word sense disambiguation • You get points for them (the, a, etc.) and for punctuation marks! How difficult is POS Tagging? Automatic POS Tagging • In the Brown corpus: • Symbolic • ~ 11% of the word types are ambiguous with regard to part of speech • Rule-based • ~ 40% of the word tokens are ambiguous • Transformation-based • But they tend to be very common words. E.g., that • Probabilistic • I know that he is honest = preposition (IN) • Hidden Markov models • Yes, that play was nice = determiner (DT) • You can’t go that far = adverb (RB) • Log-linear models Rule-based Tagging Rule-based Example • Start with a dictionary ! • Assign all possible tags to words from the !!!! ! NN! dictionary !!!! ! RB!!! ! VBN!! JJ VB! • Write rules by hand to selectively remove tags PRP! VBD!! TO VB DT NN! • Leaving the correct tag for each word She!promised to back the!bill Rule-based Example Transformation-based Eliminate VBN if VBD is an option when • Combines rule-based and probabilistic tagging VBN|VBD follows “<start> PRP” • rules are used to specify tags in a certain environment !!!! ! NN! • probabilistic, we use a tagged corpus to find the best RB!!! performing rules (supervised learning) VBN ! JJ VB! • Input PRP VBD!! TO VB DT NN! • tagged corpus She!promised to back the!bill • dictionary (with most frequent tags) • Example: Brill tagger HMM: Part-of-Speech Automatic POS Tagging Transition Probabilities • Symbolic • Rule-based • Transformation-based • Probabilistic • Hidden Markov models • Log-linear models Observation Likelihoods: P(word|tag) HMM Maxent P(tag|word) MEMMs • Can do surprisingly well just looking at a word by itself: • Word the: the DT • Maximum Entropy Markov Model • Prefixes unfathomable: un- JJ • A sequence version of the maximum entropy • Suffixes Importantly: -ly RB classifier. • Capitalization Meridian: CAP NNP ti-2 ti-1 • Word shapes 35-year: d-x JJ NNP MD VB • Then build a classifier to predict tag wi-1 wi-1 wi wi+1 <s> Janet will back the bill • Maxent P(tag|word): 93.7% overall / 82.6% unknown Slide adapted from Dan Jurafsky MEMMs More Features ti-2 ti-1 NNP MD VB wi-1 wi-1 wi wi+1 <s> Janet will back the bill Slide adapted from Dan Jurafsky MEMM Decoding POS Tagging Accuracies • Rough accuracies: • Simplest algorithm • Baseline: most freq tag: ~90% • Greedy: at each step in sequence, select tag that maximizes P(tag | nearby words, nearby tags) • Trigram HMM: ~95% • Maxent P(t|w): 93.7% • In practice • MEMM tagger: 96.9% • Viterbi algorithm • Bidirectional MEMM: 97.2% • Beam search • Upper bound: ~98% (human agreement) Slide adapted from Dan Jurafsky More Resources References • Log-linear models • Stanford POS Tagger (cyclic dependency network, bidirectional version of MEMM) • Ratnaparkhi, EMNLP 1996 • http://nlp.stanford.edu/software/tagger.shtml • Toutanova et al., NAACL 2003 • CMU Twitter POS tagger • Excellent recent survey: “Part-of-speech tagging from 97% to 100%: is it time for some • http://www.ark.cs.cmu.edu/TweetNLP/ linguistics?” (Manning, 2011) Summary Training a Tagger • Input • Penn Treebank: standard tagset • tagged corpus • Approaches to POS tagging: • dictionary (with most frequent tags) • Symbolic: rule-based, transformation-based • These are available for English • Probabilistic: HMMs, MEMMs • What about other languages? Research in POS Tagging • Low resource languages • Learning a Part-of-Speech Tagger from Two Hours of Annotation (Garrette and Baldridge, 2013) [video].
Load more

Recommended publications
  • How Do BERT Embeddings Organize Linguistic Knowledge?
    How Do BERT Embeddings Organize Linguistic Knowledge? Giovanni Puccettiy , Alessio Miaschi? , Felice Dell’Orletta y Scuola Normale Superiore, Pisa ?Department of Computer Science, University of Pisa Istituto di Linguistica Computazionale “Antonio Zampolli”, Pisa ItaliaNLP Lab – www.italianlp.it [email protected], [email protected], [email protected] Abstract et al., 2019), we proposed an in-depth investigation Several studies investigated the linguistic in- aimed at understanding how the information en- formation implicitly encoded in Neural Lan- coded by BERT is arranged within its internal rep- guage Models. Most of these works focused resentation. In particular, we defined two research on quantifying the amount and type of in- questions, aimed at: (i) investigating the relation- formation available within their internal rep- ship between the sentence-level linguistic knowl- resentations and across their layers. In line edge encoded in a pre-trained version of BERT and with this scenario, we proposed a different the number of individual units involved in the en- study, based on Lasso regression, aimed at understanding how the information encoded coding of such knowledge; (ii) understanding how by BERT sentence-level representations is ar- these sentence-level properties are organized within ranged within its hidden units. Using a suite of the internal representations of BERT, identifying several probing tasks, we showed the existence groups of units more relevant for specific linguistic of a relationship between the implicit knowl- tasks. We defined a suite of probing tasks based on edge learned by the model and the number of a variable selection approach, in order to identify individual units involved in the encodings of which units in the internal representations of BERT this competence.
    [Show full text]
  • Treebanks, Linguistic Theories and Applications Introduction to Treebanks
    Treebanks, Linguistic Theories and Applications Introduction to Treebanks Lecture One Petya Osenova and Kiril Simov Sofia University “St. Kliment Ohridski”, Bulgaria Bulgarian Academy of Sciences, Bulgaria ESSLLI 2018 30th European Summer School in Logic, Language and Information (6 August – 17 August 2018) Plan of the Lecture • Definition of a treebank • The place of the treebank in the language modeling • Related terms: parsebank, dynamic treebank • Prerequisites for the creation of a treebank • Treebank lifecycle • Theory (in)dependency • Language (in)dependency • Tendences in the treebank development 30th European Summer School in Logic, Language and Information (6 August – 17 August 2018) Treebank Definition A corpus annotated with syntactic information • The information in the annotation is added/checked by a trained annotator - manual annotation • The annotation is complete - no unannotated fragments of the text • The annotation is consistent - similar fragments are analysed in the same way • The primary format of annotation - syntactic tree/graph 30th European Summer School in Logic, Language and Information (6 August – 17 August 2018) Example Syntactic Trees from Wikipedia The two main approaches to modeling the syntactic information 30th European Summer School in Logic, Language and Information (6 August – 17 August 2018) Pros vs. Cons (Handbook of NLP, p. 171) Constituency • Easy to read • Correspond to common grammatical knowledge (phrases) • Introduce arbitrary complexity Dependency • Flexible • Also correspond to common grammatical
    [Show full text]
  • Classifiers: a Typology of Noun Categorization Edward J
    Western Washington University Western CEDAR Modern & Classical Languages Humanities 3-2002 Review of: Classifiers: A Typology of Noun Categorization Edward J. Vajda Western Washington University, [email protected] Follow this and additional works at: https://cedar.wwu.edu/mcl_facpubs Part of the Modern Languages Commons Recommended Citation Vajda, Edward J., "Review of: Classifiers: A Typology of Noun Categorization" (2002). Modern & Classical Languages. 35. https://cedar.wwu.edu/mcl_facpubs/35 This Book Review is brought to you for free and open access by the Humanities at Western CEDAR. It has been accepted for inclusion in Modern & Classical Languages by an authorized administrator of Western CEDAR. For more information, please contact [email protected]. J. Linguistics38 (2002), I37-172. ? 2002 CambridgeUniversity Press Printedin the United Kingdom REVIEWS J. Linguistics 38 (2002). DOI: Io.IOI7/So022226702211378 ? 2002 Cambridge University Press Alexandra Y. Aikhenvald, Classifiers: a typology of noun categorization devices.Oxford: OxfordUniversity Press, 2000. Pp. xxvi+ 535. Reviewedby EDWARDJ. VAJDA,Western Washington University This book offers a multifaceted,cross-linguistic survey of all types of grammaticaldevices used to categorizenouns. It representsan ambitious expansion beyond earlier studies dealing with individual aspects of this phenomenon, notably Corbett's (I99I) landmark monograph on noun classes(genders), Dixon's importantessay (I982) distinguishingnoun classes fromclassifiers, and Greenberg's(I972) seminalpaper on numeralclassifiers. Aikhenvald'sClassifiers exceeds them all in the number of languages it examines and in its breadth of typological inquiry. The full gamut of morphologicalpatterns used to classify nouns (or, more accurately,the referentsof nouns)is consideredholistically, with an eye towardcategorizing the categorizationdevices themselvesin terms of a comprehensiveframe- work.
    [Show full text]
  • Senserelate::Allwords - a Broad Coverage Word Sense Tagger That Maximizes Semantic Relatedness
    WordNet::SenseRelate::AllWords - A Broad Coverage Word Sense Tagger that Maximizes Semantic Relatedness Ted Pedersen and Varada Kolhatkar Department of Computer Science University of Minnesota Duluth, MN 55812 USA tpederse,kolha002 @d.umn.edu http://senserelate.sourceforge.net{ } Abstract Despite these difficulties, word sense disambigua- tion is often a necessary step in NLP and can’t sim- WordNet::SenseRelate::AllWords is a freely ply be ignored. The question arises as to how to de- available open source Perl package that as- velop broad coverage sense disambiguation modules signs a sense to every content word (known that can be deployed in a practical setting without in- to WordNet) in a text. It finds the sense of vesting huge sums in manual annotation efforts. Our each word that is most related to the senses answer is WordNet::SenseRelate::AllWords (SR- of surrounding words, based on measures found in WordNet::Similarity. This method is AW), a method that uses knowledge already avail- shown to be competitive with results from re- able in the lexical database WordNet to assign senses cent evaluations including SENSEVAL-2 and to every content word in text, and as such offers SENSEVAL-3. broad coverage and requires no manual annotation of training data. SR-AW finds the sense of each word that is most 1 Introduction related or most similar to those of its neighbors in the Word sense disambiguation is the task of assigning sentence, according to any of the ten measures avail- a sense to a word based on the context in which it able in WordNet::Similarity (Pedersen et al., 2004).
    [Show full text]
  • Deep Linguistic Analysis for the Accurate Identification of Predicate
    Deep Linguistic Analysis for the Accurate Identification of Predicate-Argument Relations Yusuke Miyao Jun'ichi Tsujii Department of Computer Science Department of Computer Science University of Tokyo University of Tokyo [email protected] CREST, JST [email protected] Abstract obtained by deep linguistic analysis (Gildea and This paper evaluates the accuracy of HPSG Hockenmaier, 2003; Chen and Rambow, 2003). parsing in terms of the identification of They employed a CCG (Steedman, 2000) or LTAG predicate-argument relations. We could directly (Schabes et al., 1988) parser to acquire syntac- compare the output of HPSG parsing with Prop- tic/semantic structures, which would be passed to Bank annotations, by assuming a unique map- statistical classifier as features. That is, they used ping from HPSG semantic representation into deep analysis as a preprocessor to obtain useful fea- PropBank annotation. Even though PropBank tures for training a probabilistic model or statistical was not used for the training of a disambigua- tion model, an HPSG parser achieved the ac- classifier of a semantic argument identifier. These curacy competitive with existing studies on the results imply the superiority of deep linguistic anal- task of identifying PropBank annotations. ysis for this task. Although the statistical approach seems a reason- 1 Introduction able way for developing an accurate identifier of Recently, deep linguistic analysis has successfully PropBank annotations, this study aims at establish- been applied to real-world texts. Several parsers ing a method of directly comparing the outputs of have been implemented in various grammar for- HPSG parsing with the PropBank annotation in or- malisms and empirical evaluation has been re- der to explicitly demonstrate the availability of deep ported: LFG (Riezler et al., 2002; Cahill et al., parsers.
    [Show full text]
  • The Procedure of Lexico-Semantic Annotation of Składnica Treebank
    The Procedure of Lexico-Semantic Annotation of Składnica Treebank Elzbieta˙ Hajnicz Institute of Computer Science, Polish Academy of Sciences ul. Ordona 21, 01-237 Warsaw, Poland [email protected] Abstract In this paper, the procedure of lexico-semantic annotation of Składnica Treebank using Polish WordNet is presented. Other semantically annotated corpora, in particular treebanks, are outlined first. Resources involved in annotation as well as a tool called Semantikon used for it are described. The main part of the paper is the analysis of the applied procedure. It consists of the basic and correction phases. During basic phase all nouns, verbs and adjectives are annotated with wordnet senses. The annotation is performed independently by two linguists. Multi-word units obtain special tags, synonyms and hypernyms are used for senses absent in Polish WordNet. Additionally, each sentence receives its general assessment. During the correction phase, conflicts are resolved by the linguist supervising the process. Finally, some statistics of the results of annotation are given, including inter-annotator agreement. The final resource is represented in XML files preserving the structure of Składnica. Keywords: treebanks, wordnets, semantic annotation, Polish 1. Introduction 2. Semantically Annotated Corpora It is widely acknowledged that linguistically annotated cor- Semantic annotation of text corpora seems to be the last pora play a crucial role in NLP. There is even a tendency phase in the process of corpus annotation, less popular than towards their ever-deeper annotation. In particular, seman- morphosyntactic and (shallow or deep) syntactic annota- tically annotated corpora become more and more popular tion. However, there exist semantically annotated subcor- as they have a number of applications in word sense disam- pora for many languages.
    [Show full text]
  • The “Person” Category in the Zamuco Languages. a Diachronic Perspective
    On rare typological features of the Zamucoan languages, in the framework of the Chaco linguistic area Pier Marco Bertinetto Luca Ciucci Scuola Normale Superiore di Pisa The Zamucoan family Ayoreo ca. 4500 speakers Old Zamuco (a.k.a. Ancient Zamuco) spoken in the XVIII century, extinct Chamacoco (Ɨbɨtoso, Tomarâho) ca. 1800 speakers The Zamucoan family The first stable contact with Zamucoan populations took place in the early 18th century in the reduction of San Ignacio de Samuco. The Jesuit Ignace Chomé wrote a grammar of Old Zamuco (Arte de la lengua zamuca). The Chamacoco established friendly relationships by the end of the 19th century. The Ayoreos surrended rather late (towards the middle of the last century); there are still a few nomadic small bands in Northern Paraguay. The Zamucoan family Main typological features -Fusional structure -Word order features: - SVO - Genitive+Noun - Noun + Adjective Zamucoan typologically rare features Nominal tripartition Radical tenselessness Nominal aspect Affix order in Chamacoco 3 plural Gender + classifiers 1 person ø-marking in Ayoreo realis Traces of conjunct / disjunct system in Old Zamuco Greater plural and clusivity Para-hypotaxis Nominal tripartition Radical tenselessness Nominal aspect Affix order in Chamacoco 3 plural Gender + classifiers 1 person ø-marking in Ayoreo realis Traces of conjunct / disjunct system in Old Zamuco Greater plural and clusivity Para-hypotaxis Nominal tripartition All Zamucoan languages present a morphological tripartition in their nominals. The base-form (BF) is typically used for predication. The singular-BF is (Ayoreo & Old Zamuco) or used to be (Cham.) the basis for any morphological operation. The full-form (FF) occurs in argumental position.
    [Show full text]
  • Unified Language Model Pre-Training for Natural
    Unified Language Model Pre-training for Natural Language Understanding and Generation Li Dong∗ Nan Yang∗ Wenhui Wang∗ Furu Wei∗† Xiaodong Liu Yu Wang Jianfeng Gao Ming Zhou Hsiao-Wuen Hon Microsoft Research {lidong1,nanya,wenwan,fuwei}@microsoft.com {xiaodl,yuwan,jfgao,mingzhou,hon}@microsoft.com Abstract This paper presents a new UNIfied pre-trained Language Model (UNILM) that can be fine-tuned for both natural language understanding and generation tasks. The model is pre-trained using three types of language modeling tasks: unidirec- tional, bidirectional, and sequence-to-sequence prediction. The unified modeling is achieved by employing a shared Transformer network and utilizing specific self-attention masks to control what context the prediction conditions on. UNILM compares favorably with BERT on the GLUE benchmark, and the SQuAD 2.0 and CoQA question answering tasks. Moreover, UNILM achieves new state-of- the-art results on five natural language generation datasets, including improving the CNN/DailyMail abstractive summarization ROUGE-L to 40.51 (2.04 absolute improvement), the Gigaword abstractive summarization ROUGE-L to 35.75 (0.86 absolute improvement), the CoQA generative question answering F1 score to 82.5 (37.1 absolute improvement), the SQuAD question generation BLEU-4 to 22.12 (3.75 absolute improvement), and the DSTC7 document-grounded dialog response generation NIST-4 to 2.67 (human performance is 2.65). The code and pre-trained models are available at https://github.com/microsoft/unilm. 1 Introduction Language model (LM) pre-training has substantially advanced the state of the art across a variety of natural language processing tasks [8, 29, 19, 31, 9, 1].
    [Show full text]
  • Building a Treebank for French
    Building a treebank for French £ £¥ Anne Abeillé£ , Lionel Clément , Alexandra Kinyon ¥ £ TALaNa, Université Paris 7 University of Pennsylvania 75251 Paris cedex 05 Philadelphia FRANCE USA abeille, clement, [email protected] Abstract Very few gold standard annotated corpora are currently available for French. We present an ongoing project to build a reference treebank for French starting with a tagged newspaper corpus of 1 Million words (Abeillé et al., 1998), (Abeillé and Clément, 1999). Similarly to the Penn TreeBank (Marcus et al., 1993), we distinguish an automatic parsing phase followed by a second phase of systematic manual validation and correction. Similarly to the Prague treebank (Hajicova et al., 1998), we rely on several types of morphosyntactic and syntactic annotations for which we define extensive guidelines. Our goal is to provide a theory neutral, surface oriented, error free treebank for French. Similarly to the Negra project (Brants et al., 1999), we annotate both constituents and functional relations. 1. The tagged corpus pronoun (= him ) or a weak clitic pronoun (= to him or to As reported in (Abeillé and Clément, 1999), we present her), plus can either be a negative adverb (= not any more) the general methodology, the automatic tagging phase, the or a simple adverb (= more). Inflectional morphology also human validation phase and the final state of the tagged has to be annotated since morphological endings are impor- corpus. tant for gathering constituants (based on agreement marks) and also because lots of forms in French are ambiguous 1.1. Methodology with respect to mode, person, number or gender. For exam- 1.1.1.
    [Show full text]
  • Converting an HPSG-Based Treebank Into Its Parallel Dependency-Based Treebank
    Converting an HPSG-based Treebank into its Parallel Dependency-based Treebank Masood Ghayoomiy z Jonas Kuhnz yDepartment of Mathematics and Computer Science, Freie Universität Berlin z Institute for Natural Language Processing, University of Stuttgart [email protected] [email protected] Abstract A treebank is an important language resource for supervised statistical parsers. The parser induces the grammatical properties of a language from this language resource and uses the model to parse unseen data automatically. Since developing such a resource is very time-consuming and tedious, one can take advantage of already extant resources by adapting them to a particular application. This reduces the amount of human effort required to develop a new language resource. In this paper, we introduce an algorithm to convert an HPSG-based treebank into its parallel dependency-based treebank. With this converter, we can automatically create a new language resource from an existing treebank developed based on a grammar formalism. Our proposed algorithm is able to create both projective and non-projective dependency trees. Keywords: Treebank Conversion, the Persian Language, HPSG-based Treebank, Dependency-based Treebank 1. Introduction treebank is introduced in Section 4. The properties Supervised statistical parsers require a set of annotated of the available dependency treebanks for Persian and data, called a treebank, to learn the grammar of the their pros and cons are explained in Section 5. The target language and create a wide coverage grammar tools and the experimental setup as well as the ob- model to parse unseen data. Developing such a data tained results are described and discussed in Section source is a tedious and time-consuming task.
    [Show full text]
  • Double Classifiers in Navajo Verbs *
    Double Classifiers in Navajo Verbs * Lauren Pronger Class of 2018 1 Introduction Navajo verbs contain a morpheme known as a “classifier”. The exact functions of these morphemes are not fully understood, although there are some hypotheses, listed in Sec- tions 2.1-2.3 and 4. While the l and ł-classifiers are thought to have an effect on averb’s transitivity, there does not seem to be a comparable function of the ; and d-classifiers. However, some sources do suggest that the d-classifier is associated with the middle voice (see Section 4). In addition to any hypothesized semantic functions, the d-classifier is also one of the two most common morphemes that trigger what is known as the d-effect, essentially a voicing alternation of the following consonant explained in Section 3 (the other mor- pheme is the 1st person dual plural marker ‘-iid-’). When the d-effect occurs, the ‘d’ of the involved morpheme is often realized as null in the surface verb. This means that the only evidence of most d-classifiers in a surface verb is the voicing alternation of the following stem-initial consonant from the d-effect. There is also a rare phenomenon where two *I would like to thank Jonathan Washington and Emily Gasser for providing helpful feedback on earlier versions of this thesis, and Jeremy Fahringer for his assistance in using the Swarthmore online Navajo dictionary. I would also like to thank Ted Fernald, Ellavina Perkins, and Irene Silentman for introducing me to the Navajo language. 1 classifiers occur in a single verb, something that shouldn’t be possible with position class morphology.
    [Show full text]
  • 1 Noun Classes and Classifiers, Semantics of Alexandra Y
    1 Noun classes and classifiers, semantics of Alexandra Y. Aikhenvald Research Centre for Linguistic Typology, La Trobe University, Melbourne Abstract Almost all languages have some grammatical means for the linguistic categorization of noun referents. Noun categorization devices range from the lexical numeral classifiers of South-East Asia to the highly grammaticalized noun classes and genders in African and Indo-European languages. Further noun categorization devices include noun classifiers, classifiers in possessive constructions, verbal classifiers, and two rare types: locative and deictic classifiers. Classifiers and noun classes provide a unique insight into how the world is categorized through language in terms of universal semantic parameters involving humanness, animacy, sex, shape, form, consistency, orientation in space, and the functional properties of referents. ABBREVIATIONS: ABS - absolutive; CL - classifier; ERG - ergative; FEM - feminine; LOC – locative; MASC - masculine; SG – singular 2 KEY WORDS: noun classes, genders, classifiers, possessive constructions, shape, form, function, social status, metaphorical extension 3 Almost all languages have some grammatical means for the linguistic categorization of nouns and nominals. The continuum of noun categorization devices covers a range of devices from the lexical numeral classifiers of South-East Asia to the highly grammaticalized gender agreement classes of Indo-European languages. They have a similar semantic basis, and one can develop from the other. They provide a unique insight into how people categorize the world through their language in terms of universal semantic parameters involving humanness, animacy, sex, shape, form, consistency, and functional properties. Noun categorization devices are morphemes which occur in surface structures under specifiable conditions, and denote some salient perceived or imputed characteristics of the entity to which an associated noun refers (Allan 1977: 285).
    [Show full text]