Word Segmentation Standard in Chinese, Japanese and Korean
Total Page:16
File Type:pdf, Size:1020Kb
Word Segmentation Standard in Chinese, Japanese and Korean Key-Sun Choi Hitoshi Isahara Kyoko Kanzaki Hansaem Kim Seok Mun Pak Maosong Sun KAIST NICT NICT National Inst. Baekseok Univ. Tsinghua Univ. Daejeon Korea Kyoto Japan Kyoto Japan Korean Lang. Cheonan Korea Beijing China [email protected] [email protected] [email protected] Seoul Korea [email protected] [email protected] [email protected] framework), and others in ISO/TC37/SC4 1 . Abstract These standards describe annotation methods but not for the meaningful units of word segmenta- Word segmentation is a process to divide a tion. In this aspect, MAF and SynAF are to anno- sentence into meaningful units called “word tate each linguistic layer horizontally in a stan- unit” [ISO/DIS 24614-1]. What is a word dardized way for the further interoperability. unit is judged by principles for its internal in- Word segmentation standard would like to rec- tegrity and external use constraints. A word ommend what word units should be candidates to unit’s internal structure is bound by prin- ciples of lexical integrity, unpredictability be registered in some storage or lexicon, and and so on in order to represent one syntacti- what type of word sequences called “word unit” cally meaningful unit. Principles for external should be recognized before syntactic processing. use include language economy and frequency In section 2, principles of word segmentation such that word units could be registered in a will be introduced based on ISO/CD 24614-1. lexicon or any other storage for practical re- Section 3 will describe the problems in word duction of processing complexity for the fur- segmentation and what should be word units in ther syntactic processing after word segmen- each language of Chinese, Japanese and Korean. tation. Such principles for word segmentation The conclusion will include what could be are applied for Chinese, Japanese and Korean, shared among three languages for word segmen- and impacts of the standard are discussed. tation. 1 Introduction 2 Word Segmentation: Framework and Word segmentation is the process of dividing of Principles sentence into meaningful units. For example, Word unit is a layered pre-syntactical unit. That “the White House” consists of three words but means that a word unit consists of the smaller designates one concept for the President’s resi- word units. But the maximal word unit is fre- dence in USA. “Pork” in English is translated quently occurred in corpora under the constraints into two words “pig meat” in Chinese, Korean that the syntactic processing will not refer the and Japanese. In Japanese and Korean, because internal structure of the word unit an auxiliary verb must be followed by main verb, Basic atoms of word unit are word form, mor- they will compose one word unit like “tabetai” pheme including bound morpheme, and non- and “meoggo sipda” (want to eat). So the word lexical items like punctuation mark, numeric unit is defined by a meaningful unit that could be string, foreign character string and others as a candidate of lexicon or of any other type of shown in Figure 1. Usually we say that “word” is storage (or expanded derived lexicon) that is use- lemma or word form. Word form is a form that a ful for the further syntactic processing. A word lexeme takes when used in a sentence. For ex- unit is more or less fixed and there is no syntactic ample, strings “have”, “has”, and “having” are interference in the inside of the word unit. In the word forms of the lexeme HAVE, generally dis- practical sense, it is useful for the further syntac- tinguished by the use of capital letters. [ISO/CD tic parsing because it is not decomposable by 24614-1] Lemma is a conventional form used to syntactic processing and also frequently occurred represent a lexeme, and lexeme is an abstract in corpora. unit generally associated with a set of word There are a series of linguistic annotation forms sharing a common meaning. standards in ISO: MAF (morpho-syntactic anno- tation framework), SynAF (syntactic annotation 1 Refer to http://www.tc37sc4.org/ for documents MAF, SynAF and so on. 179 Proceedings of the 7th Workshop on Asian Language Resources, ACL-IJCNLP 2009, pages 179–186, Suntec, Singapore, 6-7 August 2009. c 2009 ACL and AFNLP and the other is processing-oriented practical perspective. In ISO 24614-1, principles from a linguistic perspective, there are five principles: principles of (1) bound morpheme, (2) lexical integrity, (3) unpredictability, (4) idiomatization, and (5) un- productivity. First, bound morpheme is something like “in” of “inefficient”. The principle of bound mor- pheme says that each bound morpheme plus word will make another word. Second, principle Figure 1. Configuration of Word Unit of lexical integrity means that any syntactic processing cannot refer the internal structure of BNF of word unit is as follows: word (or word unit). From the principle, we can <word unit> ::= <word form> | <morpheme> | say that the longest word unit is the maximal <non lexical items> | <word unit>, meaningful unit before syntactic processing. where <word unit> is recursively defined be- Third, another criterion to recognize a word is cause a longer word unit contains smaller word the principle of unpredictability. If we cannot units. infer the real fact from the word, we consider it Bunsetsu in Japanese is the longest word unit, as a word unit. For example, we cannot image which is an example of layered maximized pre- what is the colour of blackboard because some is syntactical unit. Eojeol in Korean is a spacing green, not black. [ISO 24614-1] The fourth prin- unit that consists of one content word (noun, ciple is that the idiom should be one word, which verb, adjective or adverb) plus a sequence of could be a subsidiary principle that follows the functional elements. Such language-specific principle of unpredictability. In the last principle, word units will be described in section 3. unproductivity is a stronger definition of word; Principles for word segmentation will set the there is no pattern to be copied to generate an- criteria to validate each word unit, to recognize other word from this word formation. For exam- its internal structure and non-lexical word unit, to ple, in “白菜” (white vegetable) in Chinese, be a candidate of lexicon, and other consistency there is no other coloured vegetable like “blue to be necessitated by practical applications for vegetable.” any text in any language. The meta model of Another set of principles is derived from the word segmentation will be explained in the practical perspective. There are four principles: processing point of view, and then their prin- frequency, Gestalt, prototype and language ciples of word units in the following subsections. economy. Two principles of frequency and lan- 2.1 Metamodel of Word Segmentation guage economy are to recognize the frequent occurrence in corpora. Gestalt and prototype A word unit has a practical unit that will be later principles are the terms in cognitive science used for syntactic processing. While the word about what are in our mental lexicon, and what segmentation is a process to identify the longest are perceivable words. word unit and its internal structure such that the Principle of language economy is to say about word unit is not the object to interfere any syn- very practical processing efficiency: “if the in- tactic operation, “chunking” is to identify the clusion of a word (unit) candidate into the lex- constituents but does not specify their internal icon can decrease the difficulty of linguistic structure. Syntactic constituent has a syntactic analysis for it, then it is likely to be a word role, but the word unit is a subset of syntactic (unit).” constituent. For example, “blue sky” could be a Gestalt principle is to perceive as a whole. “It syntactic constituent but not a word unit. Figure supports some perceivable phrasal compounds 2 shows the meta model of word segmentation. into the lexicon even though they seem to be free [ISO CD 24614-1] combinations of their perceivable constituent 2.2 Principles of Word Unit Validation parts,” [ISO/CD 24614-1] where the phrasal compound is frequently used linguistic unit and Principles for validating a word unit can be ex- its meaning is predictable from its constituent plained by two perspectives: one is linguistic one elements. Similarly, principle of prototype pro- 180 vides a rationale for including some phrasal if it is fixed expression or idiom. But if the first compounds into lexicon, and phrasal compounds character is productive with the following num- serve as prototypes in a productive word forma- eral, unit word or measure word, it will be seg- tion pattern, like “apple pie” with the pattern mented. If the last character is productive in a “fruit + pie” into the lexicon. limited manner, it forms a word unit with the preceding word, for example, “東京都” (Tokyo Metropolis), “8 月” (August) or “加速器” (acce- lerator). But if it is a productive suffix like plural suffix and noun for locality, it is segmented in- dependently in Chinese word segmentation rule, for example, “朋友|们” (friends), “长江|以北” (north of the Yangtzi River ) or “桌子|上” (on the table) in Chinese. They may have different phenomena in each language. Negation character of verb and adjective is segmented independently in Chinese, but they form one word unit in Japanese. For example, “yasashikunai” (優しく無い, not kind) is one word unit in Japanese, but “不|写” (not to write), “不| 能” (cannot), “没|研究” (did not research) Figure 2. Meta model of word segmentation proess and “未| 完成” (not completed) will be seg- [ISO/CD 24614-1] mented independently in Chinese.