Shallow Parsing As Part-Of-Speech Tagging*
Total Page：16
File Type：pdf, Size：1020Kb
In: Proceedings of CoNLL-2000 and LLL-2000, pages 145-147, Lisbon, Portugal, 2000. Shallow Parsing as Part-of-Speech Tagging* Miles Osborne University of Edinburgh Division of Informatics 2 Buccleuch Place Edinburgh EH8 9LW , Scotland osborne@cogsci, ed. ac. uk 3 Convincing the Tagger to Shallow Parse Abstract The insight here is that one can view (some Treating shallow parsing as part-of-speech tag- of) the differences between tagging and (shal- ging yields results comparable with other, more low) parsing as one of context: shallow pars- elaborate approaches. Using the CoNLL 2000 ing requires access to a greater part of the training and testing material, our best model surrounding lexical/POS syntactic environment had an accuracy of 94.88%, with an overall FB1 than does simple POS tagging. This extra in- score of 91.94%. The individual FB1 scores for formation can be encoded in a state. NPs were 92.19%, VPs 92.70% and PPs 96.69%. However, one must balance this approach with the fact that as the amount of information 1 Introduction in a state increases, with limited training ma- Shallow parsing has received a reasonable terial, the chance of seeing such a state again amount of attention in the last few years (for in the future diminishes. We therefore would example (Ramshaw and Marcus, 1995)). In expect performance to increase as we increased this paper, instead of modifying some existing the amount of information in a state, and then technique, or else proposing an entirely new ap- decrease when overfitting and/or sparse statis- proach, we decided to build a shallow parser us- tics become dominate factors. ing an off-the-shelf part-of-speech (POS) tagger. We trained the tagger using 'words' that were We deliberately did not modify the POS tag- various 'configurations' (concatenations) of ac- ger's internal operation in any way. Our results tual words, POS tags, chunk-types, and/or suf- suggested that achieving reasonable shallow- fixes or prefixes of words and/or chunk-types. parsing performance does not in general require By training upon these concatenations, we help anything more elaborate than a simple POS tag- bridge the gap between simple POS tagging and ger. However, an error analysis suggested the shallow parsing. existence of a small set of constructs that are In the rest of the paper, we refer to what the not so easily characterised by finite-state ap- tagger considers to be a word as a configura- proaches such as ours. tion. A configuration will be a concatenation of various elements of the training set relevant to 2 The Tagger decision making regarding chunk assignment. A We used Ratnaparkhi's maximum entropy- 'word' will mean a word as found in the train- based POS tagger (Ratnaparkhi, 1996). When ing set. 'Tags' refer to the POS tags found in tagging, the model tries to recover the most the training set. Again, such tags may be part likely (unobserved) tag sequence, given a se- of a configuration. We refer to what the tagger quence of observed words. considers as a tag as a prediction. Predictions For our experiments, we used the binary-only will be chunk labels. distribution of the tagger (Ratnaparkhi, 1996). 4 Experiments " The full version of this paper can be found at We now give details of the experiments we ran. http://www.cogsci.ed.ac.uk/'osborne/shallow.ps To make matters clearer, consider the following 145 fragment of the training set: configurations consisting of tags and words (wl and tl). The training set was then re- Word Wl w2 w3 duced to consist of just tag-word configura- POS Tag tl t2 t3 tions and tagged using this model. After- Chunk Cl c2 c3 wards, we collected the predictions for use in the second model. Results: Words are wl,w2 and w3, tags are t~L,t2 and t3 and chunk labels are cl, c2 and ca. Throughout, Chunk type P R FB1 we built various configurations when predicting Overall 89.79 90.70 90.24 the chunk label for word wl. 1 ADJP 69.61 57.53 63.00 With respect to the situation just mentioned ADVP 74.72 77.14 75.91 CONJP 54.55 66.67 60.00 (predicting the label for word wl), we gradu- INTJ 50.00 50.00 50.00 ally increased the amount of information in each LST 0.00 0.00 0.00 configuration as follows: NP 89.80 91.12 90.4 PP 95.15 96.26 95.70 1. A configuration consisting of just words PRT 71.84 69.81 70.81 (word wl). Results: SBAR 85.63 80.19 82.82 VP 89.54 91.31 90.41 Chunk type P R FB1 Overall 88.06 88.71 88.38 Overall accuracy: 93.79% ADJP 67.57 51.37 58.37 . The final configuration made an attempt ADVP 74.34 74.25 74.29 CONJP 54.55 66.67 6O.00 to take deal with sparse statistics. It con- INTJ 100.00 50.00 66.67 sisted of the current tag tl, the next tag t2, LST 0.00 0.00 0.00 the current chunk label cl, the last two let- NP 87.84 89.41 88.62 ters of the next chunk label c2, the first two PP 94.80 95.91 95.35 letters of the current word wl and the last PRT 71.00 66.98 68.93 SBAR 82.30 72.15 76.89 four letters of the current word wl. This VP 86.68 88.15 87.41 configuration was the result of numerous experiments and gave the best overall per- Overall accuracy: 92.76% formance. The results can be found in Ta- 2. A configuration consisting of just tags (tag ble 1. tl). Results: We remark upon our experiments in the com- ments section. Chunk type P R FB1 Overall 88.15 88.07 88.11 5 Error Analysis ADJP 67.99 54.79 60.68 ADVP 71.61 70.79 71.20 We examined the performance of our final CONJP 35.71 55.56 43.48 model with respect to the testing material and INTJ 0.00 0.00 0.00 LST 0.00 0.00 0.00 found that errors made by our shallow parser NP 89.47 89.57 89.52 could be grouped into three categories: diffi- PP 87.70 95.28 91.33 cult syntactic constructs, mistakes made in the PRT 52.27 21.70 30.67 training or testing material by the annotators, SBAR 83.92 31.21 45.50 and errors peculiar to our approach. 2 VP 90.38 91.18 [ 90.78 Taking each category of the three in turn, Overall accuracy 92.66%. problematic constructs included: co-ordination, punctuation, treating ditransitive VPs as being 3. Both words, tags and the current chunk transitive VPs, confusions regarding adjective label (wl, tl, Cl) in a configuration. We or adverbial phrases, and copulars seen as be- allowed the tagger access to the current ing possessives. chunk label by training another model with 2The raw results can be found at: http://www.cog 1For space reasons, we had to remove many of these sci.ed.ac.uk/-osborne/conll00-results.txt The mis-anal- experiments. The longer version of the paper gives rele- ysed sentences can be found at: http://www.cogsci.ed. vant details. ac.uk/-osborne/conll00-results.txt. 146 Mistakes (noise) in the training and testing test data precision recall F fl=l material were mainly POS tagging errors. An ADJP 72.42% 64.16% 68.04 additional source of errors were odd annotation ADVP 75.94% 79.10% 77.49 decisions. CONJP 50.00% 55.56% 52.63 The final source of errors were peculiar to our INTJ 100.00% 50.00% 66.67 system. Exponential distributions (as used by LST 0.00% 0.00% 0.00 our tagger) assign a non-zero probability to all NP 91.92% 92.45% 92.19 possible events. This means that the tagger will PP 95.95% 97.44% 96.69 at times assign chunk labels that are illegal, for PRT 73.33% 72.64% 72.99 example assigning a word the label I-NP when SBAR 86.40% 80.75% 83.48 the word is not in a NP. Although these errors VP 92.13% 93.28% 92.70 were infrequent, eliminating them would require all 91.65% 92.23% 91.94 'opening-up' the tagger and rejecting illegal hy- pothesised chunk labels from consideration. Table 1: The results for configuration 4. Overall 6 Comments accuracy: 94.88% As was argued in the introduction, increasing the size of the context produces better results, phrasal recognition) is a fairly easy task. Im- and such performance is bounded by issues such provements might come from better modelling, as sparse statistics. Our experiments suggest dealing with illegal chunk sequences, allowing that this was indeed true. multiple chunks with confidence intervals, sys- We make no claims about the generality of tem combination etc, but we feel that such im- our modelling. Clearly it is specific to the tagger provements will be small. Given this, we believe used. that base-phrasal chunking is close to being a In more detail, we found that: solved problem. • PPs seem easy to identify. Acknowledgements • ADJP and ADVP chunks were hard to We would like to thank Erik Tjong Kim Sang identify correctly. We suspect that im- for supplying the evaluation code, and Donnla provements here require greater syntactic Nic Gearailt for dictating over the telephone, and from the top-of-her-head, a Perl program information than just base-phrases.