Exploiting Chunk-level Features to Improve Phrase Chunking Junsheng Zhou Weiguang Qu Fen Zhang Jiangsu Research Center of Information Security & Privacy Technology School of Computer Science and Technology Nanjing Normal University. Nanjing, China, 210046 Email:{zhoujs,wgqu}@njnu.edu.cn [email protected] and noun phrase (NP) chunking. Phrase chunking Abstract provides a key feature that helps on more elaborated NLP tasks such as parsing, semantic Most existing systems solved the phrase role tagging and information extraction. chunking task with the sequence labeling There is a wide range of research work on approaches, in which the chunk candidates phrase chunking based on machine learning cannot be treated as a whole during parsing approaches. However, most of the previous work process so that the chunk-level features reduced phrase chunking to sequence labeling cannot be exploited in a natural way. In this problems either by using the classification models, paper, we formulate phrase chunking as a such as SVM (Kudo and Matsumoto, 2001), joint segmentation and labeling task. We Winnow and voted-perceptrons (Zhang et al., 2002; propose an efficient dynamic programming Collins, 2002), or by using the sequence labeling algorithm with pruning for decoding, models, such as Hidden Markov Models (HMMs) which allows the direct use of the features (Molina and Pla, 2002) and Conditional Random describing the internal characteristics of Fields (CRFs) (Sha and Pereira, 2003). When chunk and the features capturing the applying the sequence labeling approaches to correlations between adjacent chunks. A phrase chunking, there exist two major problems. relaxed, online maximum margin training Firstly, these models cannot treat globally a algorithm is used for learning. Within this sequence of continuous words as a chunk framework, we explored a variety of candidate, and thus cannot inspect the internal effective feature representations for structure of the candidate, which is an important Chinese phrase chunking. The aspect of information in modeling phrase chunking. experimental results show that the use of In particular, it makes impossible the use of local chunk-level features can lead to significant indicator function features of the type "the chunk performance improvement, and that our consists of POS tag sequence p ...,p ". For example, approach achieves state-of-the-art 1 k the Chinese NP " 农业/NN(agriculture) 生产 performance. In particular, our approach is much better at recognizing long and /NN(production) 和/CC(and) 农村/NN(rural) 经济 complicated phrases. /NN(economic) 发展/NN(development)" seems relatively difficult to be correctly recognized by a sequence labeling approach due to its length. But if 1 Introduction we can treat the sequence of words as a whole and describe the formation pattern of POS tags of this Phrase chunking is a Natural Language Processing chunk with a regular expression-like form task that consists in dividing a text into "[NN]+[CC][NN]+", then it is more likely to be syntactically correlated parts of words. Theses correctly recognized, since this pattern might better phrases are non-overlapping, i.e., a word can only express the characteristics of its constituents. As be a member of one chunk (Abney, 1991). another example, consider the recognition of Generally speaking, there are two phrase chunking special terms. In Chinese corpus, there exists a tasks, including text chunking (shallow parsing), kind of NPs called special terms, such as "『 生命 557 Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 557–567, Jeju Island, Korea, 12–14 July 2012. c 2012 Association for Computational Linguistics (Life) 禁区(Forbidden Zone) 』 ", which are and that our approach performs better than other bracketed with the particular punctuations like " approaches based on the sequence labeling models. 『, 』, 「, 」, 《, 》". When recognizing the special terms, it is difficult for the sequence 2 Related Work labeling approaches to guarantee the matching of In recent years, many chunking systems based on particular punctuations appearing at the starting machine learning approaches have been presented. and ending positions of a chunk. For instance, the Some approaches rely on k-order generative chunk candidate "『 生命(Life) 禁区(Forbidden probabilistic models, such as HMMs (Molina and Zone)” is considered to be an invalid chunk. But Pla, 2002). However, HMMs learn a generative it is easy to check this kind of punctuation model over input sequence and labeled sequence matching in a single chunk by introducing a chunk- pairs. It has difficulties in modeling multiple non- level feature. independent features of the observation sequence. Secondly, the sequence labeling models cannot To accommodate multiple overlapping features on capture the correlations between adjacent chunks, observations, some other approaches view the which should be informative for the identification phrase chunking as a sequence of classification of chunk boundaries and types. In particular, we problems, including support vector machines find that some headwords in the sentence are (SVMs) (Kudo and Matsumoto 2001) and a variety expected to have a stronger dependency relation of other classifiers (Zhang et al., 2002). Since these with their preceding headwords in preceding classifiers cannot trade off decisions at different chunks than with their immediately preceding positions against each other, the best classifier words within the same chunk. For example, in the based shallow parsers are forced to resort to following sentence: heuristic combinations of multiple classifiers. " [双方/PN(Bilateral)]_NP [经贸/NN(economic Recently, CRFs were widely employed for phrase and trade) 关系/NN(relations)]_NP [正/AD(just) chunking, and presented comparable or better 稳步/AD(steadily) 发展/VV(develop)]_VP " performance than other state-of-the-art models (Sha and Pereira 2003; McDonald et al. 2005). if we can find the three headwords "双方", "关系" Further, Sun et al. (2008) used the latent-dynamic 发展 and " " located in the three adjacent chunks conditional random fields (LDCRF) to explicitly with some head-finding rules, then the headword learn the hidden substructure of shallow phrases, dependency expressed by headword bigrams or achieving state-of-the-art performance over the trigrams should be helpful to recognize these NP-chunking task on the CoNLL data. chunks in this sentence. Some similar approaches based on classifiers or In summary, the inherent deficiency in applying sequence labeling models were also used for the sequence labeling approaches to phrase Chinese chunking (Li et al., 2003; Tan et al., 2004; chunking is that the chunk-level features one Tan et al., 2005). Chen et al. (2006) conducted an would expect to be very informative cannot be empirical study of Chinese chunking on a corpus, exploited in a natural way. which was extracted from UPENN Chinese In this paper, we formulate phrase chunking as a Treebank-4 (CTB4). They compared the joint segmentation and labeling problem, which performances of the state-of-the-art machine offers advantages over previous learning methods learning models for Chinese chunking, and by providing a natural formulation to exploit the proposed some Tag-Extension and novel voting features describing the internal structure of a chunk methods to improve performance. and the features capturing the correlations between In this paper, we model phrase chunking with a the adjacent chunks. joint segmentation and labeling approach, which Within this framework, we explored a variety of offer advantages over previous learning methods effective feature representations for Chinese phrase by explicitly incorporating the internal structural chunking. The experimental results on Chinese feature and the correlations between the adjacent chunking corpus as well as English chunking chunks. To some extent, our model is similar to corpus show that the use of chunk-level features Semi-Markov Conditional Random Fields (called a can lead to significant performance improvement, Semi-CRF), in which the segmentation and 558 labeling can also be done directly (Sarawagi and with segmented words and POS tags to an output y Cohen, 2004). However, Semi-CRF just models with tagged chunk types, like the S1 in Example 1. label dependency, and it cannot capture more The joint model considers all possible chunk correlations between adjacent chunks, as is done in boundaries and corresponding chunk types in the our approach. The limitation of Semi-CRF leads to sentence, and chooses the overall best output. This its relatively low performance. kind of parser reads the input sentences from left to right, predicts whether current segment of 3 Problem Formulation continuous words is some type of chunk. After one chunk is found, parser move on and search for next 3.1 Chunk Types possible chunk. Given a sentence x, let y denote an output tagged Unlike English chunking, there is not a with chunk types, and GEN a function that benchmarking corpus for Chinese chunking. We enumerates a set of segmentation and labeling follow the studies in (Chen et al. 2006) so that a candidates GEN(x) for x. A parser is to solve the more direct comparison with state-of-the-art following “argmax” problem: systems for Chinese chunking would be possible. T ywyˆ = arg max⋅F ( ) There are 12 types of chunks: ADJP, ADVP, CLP, yGENxÎ () DNP, DP, DVP, LCP, LST, NP, PP, QP and VP in ||y (1) T = arg maxwy⋅ f ([1..i ] ) the chunking corpus (Xue et al., 2000). The yGENxÎ () å training and test corpus can be extracted from i=1 CTB4 with a public tool, as depicted in (Chen et al. 2006). where F and f are global and local feature maps and w is the parameter vector to learn. The inner 3.2 Sequence Labeling Approaches to Phrase product wyT ⋅f() can be seen as the confidence Chunking [1..i ] score of whether yi is a chunk. The parser takes into The standard approach to phrase chunking is to use account confidence score of each chunk, by using tagging techniques with a BIO tag set.