Unsupervised Separation of Transliterable and Native Words for Malayalam Deepak P Queen’s University Belfast, UK
[email protected] Abstract names that need to be transliterated than translated to correlate with English text. On a manual analy- Differentiating intrinsic language words sis of a news article dataset, we found that translit- from transliterable words is a key step erated words and proper nouns each form 10-12% aiding text processing tasks involving dif- of all distinct words. It is useful to transliterate ferent natural languages. We consider such words for scenarios that involve processing the problem of unsupervised separation of Malayalam text in the company of English text; transliterable words from native words for this will avoid them being treated as separate index text in Malayalam language. Outlining a terms (wrt their transliteration) in a multi-lingual key observation on the diversity of char- retrieval engine, and help a statistical translation acters beyond the word stem, we develop system to make use of the link to improve effec- an optimization method to score words tiveness. In this context, it ia notable that there has based on their nativeness. Our method re- been recent interest in devising specialized meth- lies on the usage of probability distribu- ods to translate words that fall outside the core vo- tions over character n-grams that are re- cabulary (Tsvetkov and Dyer, 2015). fined in step with the nativeness scorings In this paper, we consider the problem of sepa- in an iterative optimization formulation. rating out such transliterable words from the other Using an empirical evaluation, we illus- words within an unlabeled dataset; we refer to trate that our method, DTIM, provides sig- the latter as “native” words.