DOCSLIB.ORG

Unsupervised Separation of Transliterable and Native Words For

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Unsupervised Separation of Transliterable and Native Words For Unsupervised Separation of Transliterable and Native Words for Malayalam Deepak P Queen’s University Belfast, UK [email protected] Abstract names that need to be transliterated than translated to correlate with English text. On a manual analy- Differentiating intrinsic language words sis of a news article dataset, we found that translit- from transliterable words is a key step erated words and proper nouns each form 10-12% aiding text processing tasks involving dif- of all distinct words. It is useful to transliterate ferent natural languages. We consider such words for scenarios that involve processing the problem of unsupervised separation of Malayalam text in the company of English text; transliterable words from native words for this will avoid them being treated as separate index text in Malayalam language. Outlining a terms (wrt their transliteration) in a multi-lingual key observation on the diversity of char- retrieval engine, and help a statistical translation acters beyond the word stem, we develop system to make use of the link to improve effec- an optimization method to score words tiveness. In this context, it ia notable that there has based on their nativeness. Our method re- been recent interest in devising specialized meth- lies on the usage of probability distribu- ods to translate words that fall outside the core vo- tions over character n-grams that are re- cabulary (Tsvetkov and Dyer, 2015). fined in step with the nativeness scorings In this paper, we consider the problem of sepa- in an iterative optimization formulation. rating out such transliterable words from the other Using an empirical evaluation, we illus- words within an unlabeled dataset; we refer to trate that our method, DTIM, provides sig- the latter as “native” words. We propose an un- nificant improvements in nativeness scor- supervised method, DTIM, that takes a dictio- ing for Malayalam, establishing DTIM as nary of distinct words from a Malayalam corpus the preferred method for the task. and scores each word based on their nativeness. Our optimization method, DTIM, iteratively re- 1 Introduction fines the nativeness scoring of each word, leverag- Malayalam is an agglutinative language from ing dictionary-level statistics modelled using char- the southern Indian state of Kerala where it is acter n-gram probability distributions. Our empiri- the official state language. It is spoken by 38 cal analysis establishes the effectiveness of DTIM. million native speakers, three times as many We outline related work in the area in Section 2. arXiv:1803.09641v1 [cs.CL] 26 Mar 2018 speakers as Hungarian (Vincze et al., 2013) or This is followed by the problem statement in Sec- Greek (Ntoulas et al., 2001), for which specialized tion 3 and the description of our proposed ap- techniques have been developed in other contexts. proach in Section 4. Our empirical analysis forms The growing web presence of Malayalam neces- Section 5 followed by conclusions in Section 7. sitates automatic techniques to process Malay- 2 Related Work alam text. A major hurdle in harnessing Malay- alam text from social and web media for multi- Identification of transliterable text fragments, be- lingual retrieval and machine translation is the ing a critical task for cross-lingual text analysis, presence of a large amount of transliterable words. has attracted attention since the 1990s. While By transliterable words, we mean both (a) words most methods addressing the problem have used (from English) like police and train that virtually supervised learning, there have been some meth- always appear in transliterated form in contem- ods that can work without labeled data. We briefly porary Malayalam, and (b) proper nouns such as survey both classes of methods. 2.1 Supervised and ‘pseudo-supervised’ 2.3 Positioning the Transliterable Word Methods Identification Task Nativeness scoring of words may be seen as a vo- An early work(Chen and Lee, 1996) focuses on cabulary stratification step (upon usage of thresh- a sub-problem, that of supervised identification olds) for usage by downstream applications. A of proper nouns for Chinese. (Jeong et al., 1999) multi-lingual text mining application that uses consider leveraging decision trees to address Malayalam and English text would benefit by the related problem of learning transliteration transliterating non-native Malayalam words to En- and back-transliteration rules for English/Korean glish, so the transliterable Malayalam token and word pairs. Recognizing the costs of procur- its transliteration is treated as the same token. ing training data, (Baker and Brew, 2008) and For machine translation, transliterable words may (Goldberg and Elhadad, 2008) explore usage be channeled to specialized translation methods of pseudo-transliterable words generated using (e.g., (Tsvetkov and Dyer, 2015)) or for manual transliteration rules on an English dictionary screening and translation. for Korean and Hebrew respectively. Such pseudo-supervision, however, would not be able 3 Problem Definition to generate uncommon domain-specific terms We now define the problem more formally. Con- such as medical/scientific terminology for usage sider n distinct words obtained from Malayalam in such domains (unless specifically tuned), and is text, W = {...,w,...}. Our task is to devise hence limited in utility. a technique that can use W to arrive at a native- ness score for each word, w, within it, as wn. We would like wn to be an accurate quantification of native-ness of word w. For example, when words 2.2 Unsupervised Methods in W are ordered in the decreasing order of wn scores, we expect to get the native words at the A recent work proposes that multi-word phrases beginning of the ordering and vice versa. We do in Malayalam text where their component words not presume availability of any data other than W; exhibit strong co-occurrence be categorized as this makes our method applicable across scenar- transliterable phrases (Prasad et al., 2014). Their ios where corpus statistics are unavailable due to intuition stems from observing contiguous words privacy or other reasons. such as test dose which often occur in translit- erated form while occurring together, but get re- 3.1 Evaluation placed by native words in other contexts. Their Given that it is easier for humans to crisply classify method is however unable to identify single each word as either native or transliterable (nouns transliterable words, or phrases involving words or transliterated english words) in lieu of attaching such as train and police whose transliterations are a score to each word, the nativeness scoring (as heavily used in the company of native Malayalam generated by a scoring method such as ours) often words. A recent method for Korean (Koo, 2015) needs to be evaluated against a crisp nativeness as- starts by identifying a seed set of transliterable sessment, i.e., a scoring with scores in {0, 1}. To words as those that begin or end with consonant aid this, we consider the ordering of words in the clusters and have vowel insertions; this is spe- labeled set in the decreasing (or more precisely, cific to Korean since Korean words apparently do non-increasing) order of nativeness scores (each not begin or end with consonant clusters. High- method produces an ordering for the dataset). To frequency words are then used as seed words for evaluate this ordering, we use two sets of metrics native Korean for usage in a Naive Bayes classi- for evaluation: fier. In addition to the outlined reasons that make both the unsupervised methods inapplicable for • Precision at the ends of the ordering: Top- our task, they both presume availability of corpus k precision denotes the fraction of native frequency statistics. We focus on a general sce- words within the k words at the top of the nario assuming the availability of only a word lex- ordering; analogously, Bottom-k precision icon. is the fraction of transliterable words among the bottom k. Since a good scoring would more strongly and weaken any initial preference to likely put native words at the top of the order- transliterable words. The vice versa holds for the ing and the transliterable ones at the bottom, a transliterable word models. We will first outline good scoring method would intuitively score the initialization step followed by the description high on both these metrics. We call the aver- of the method. age of the top-k and bottom-k precision for a given k, as Avg-k precision. These measures, 4.1 Diversity-based Initialization evaluated at varying values of k, indicate the Our initialization is inspired by an observation on quality of the nativeness scoring. the variety of suffixes attached to a word stem. |pu|ra|2 • Clustering Quality: Consider the cardinal- Consider a word stem , a stem commonly ities of the native and transliterable words leading to native Malayalam words; its suffixes from the labeled set as being N and T re- are observed to start with a variety of charac- spectively. We now take the top-N words and ters such as |ttha| (e.g., |pu|ra|ttha|kki|), |me| bottom-T words from the ordering generated (e.g., |pu|ra|me|), |mbo| (e.g., |pu|ra|mbo|kku|) by each method, and compare against the re- and |ppa| (e.g., |pu|ra|ppa|du|). On the other hand, spective labeled sets as in the case of stan- stems that mostly lead to transliterable words of- dard clustering quality evaluation1. Since the ten do not exhibit so much of diversity. For exam- cardinalities of the generated native (translit- ple, |re|so| is followed only by |rt| (i.e., resort) and erable) cluster and the native (transliterable) |po|li| is usually only followed by |s| (i.e., police). labeled set is both N (T ), the Recall of the Some stems such as |o|ppa| lead to transliterations cluster is identical to its Purity/Precision, of two English words such as open and operation.
Load more

Recommended publications
  • Malayalam Noun and Verb Morphological Analyzer: a Simple Approach
    Malayalam Noun and Verb Morphological Analyzer: A Simple Approach Nimal J Valath1, Narsheedha Beegum2 M.Tech Student1, M.Tech Student2 ABSTRACT Malayalam is a language of the Dravidian family and is one of This paper discusses the methods involved in the the four major languages of this family with a rich literary development of a Simple Malayalam Verb and Noun tradition. It is very close to Tamil, one of the major languages of Morphological Analyzer. Since in Malayalam, words can be the same family. This was due to the extensive cultural derived from a root word, a purely dictionary based approach synthesis that took place between the speakers of the two for Morphological analysis is not practical. Hence, a ‘Rule-cum- languages. The origin of Malayalam as a distinct language may Dictionary’ based approach is followed along with the Suffix be traced to the last quarter of 9th Century A.D. Throughout its Stripping concept. The grammatical behavior of the language, gradual evolution Malayalam has been influenced by the the formation of words with multiple suffixes and the various circumstances prevailed on different periods. preparation of the language are dealt with here, with examples of noun and verb forms in detail. Mainly Malayalam was influenced by Sanskrit and Prakrit brought into Kerala by Brahmins. After the 11th century a unique mixture of the native languages of Kerala and Sanskrit Keywords known as Manipravalam served as the medium of literary Morphological Analyzer, Malayalam, Suffix stripping, expression. Malayalam absorbed a lot from Sanskrit, not only in Transliteration, Retransliteration, Verb and Noun, Sandhi rules, the lexical level, but also in the phonemic, morphemic and Word Formation, Noun Cases, Algorithm.
    [Show full text]
  • Children's Possessive Structures
    Ling 404 Lecture Notes No.4 Synchronic Clines in Morphology Productivity Clines: Japanese, Navajo, Mohawk, Eskimo, vs. Spanish, Italian, English One of the questions will be asking throughout is where do the above languages fall on the productivity cline and why (providing data and analyses.) One question to ask is to what degree is the lexicon ‘morpheme-based’ or ‘word-based’ (Chapters 3-4)? ‘Working Memory’ is implicated in the choice. For instance, are speakers all equally productive with their morpheme-units as tucked within word, a [-Fusion] language, or do some languages rather require morphemes within words to be memorized and incorporated as part of the lexical item, a [+Fusion] language? In addition to labeling languages as [+]Synthetic (as in Turkish, Hungarian, Spanish, English, Mohawk) or [-]Synthetic (as in Chinese, Vietnamese), the following data provide an additional exercise in teasing out what might be going on in languages which carry such a large number of embedded morphemes. Question leading to mid-term: How might such morphemes in Polysynthetic-type languages be stored and processed? The notion of [+/-Fusion] will be the central question here as we move to our mid-term material of this class. Typically, agglutinative morphemes are considered to be ‘loosely’ structured in that one morpheme has a one-to-one meaning. But the question might rather be how productive are the morphemes in isolation—viz., do they allow for movement? The question of movement, both at the word-level and at the morpheme-level will allow us to determine the nature of +/- productivity and +/- fusion. Regarding Movement.
    [Show full text]
  • Abstracts of the Psychonomic Society — Volume 13 — November 2008 49Th Annual Meeting — November 13–16, 2008 — Chicago, Illinois
    Abstracts of the Psychonomic Society — Volume 13 — November 2008 49th Annual Meeting — November 13–16, 2008 — Chicago, Illinois Papers 1–7 Friday Morning Motion and Attention motion is mixed. In the present study, target stimuli were associated with Grand Ballroom, Friday Morning, 8:00–9:40 objects that loomed, receded or remained static in arrays of varying size. Results showed that both motion types received equal prioritization, as Chaired by Gary Chon-Wen Shyi, National Chung Cheng University evidenced by their parallel search slopes, yet looming targets benefited from an overall reduction in reaction time (RT). Further investigation 8:00–8:15 (1) ruled out possible confounding explanations for this RT advantage, Differential Distribution of Visuospatial Attention in Tracking Mul- while a perceptual measure of performance confirmed the attentional tiple Moving Objects. GARY C.-W. SHYI & SUNG-EN CHIEN, Na- equivalence of the two motion types. Taken together, these results in- tional Chung Cheng University—Paying attention to a relatively com- dicate that looming and receding objects receive equal prioritization plex object has shown evidence for differential distribution within the during attentional selection. However, it may be that postattentional pro- object. Here, we explored differential distribution of attention when mul- cesses, possibly those involved in motor preparation, facilitate responses tiple moving objects were visually tracked. In Experiments 1 and 2, we to looming motion. not only replicated the findings reported by Alvarez and Scholl (2005), demonstrating both attentional concentration and attentional amplifica- 9:20–9:35 (5) tion, but also generalized the effects to uniformed circular shapes. In Individual Differences in Voluntary Visual Attention.
    [Show full text]
  • Abstracts (PDF)
    Abstracts of the Psychonomic Society — Volume 12 — November 2007 48th Annual Meeting — November 15–18, 2007 — Long Beach, California Papers 1–7 Friday Morning Implicit Memory is a widely used experimental procedure that has been modeled in Regency ABC, Friday Morning, 8:00–9:20 order to obtain ostensibly separate measures for explicit memory and implicit memory. Existing models are critiqued here and a new con- Chaired by Neil W. Mulligan fidence process-dissociation (CPD) model is developed for a modi- University of North Carolina, Chapel Hill fied process-dissociation task that uses confidence ratings in con- junction with the usual old/new recognition response. With the new 8:00–8:15 (1) model there are several direct tests for the key assumptions of the stan- Attention and Auditory Implicit Memory. NEIL W. MULLIGAN, dard models for this task. Experimental results are reported here that University of North Carolina, Chapel Hill—Traditional theorizing are contrary to the critical assumptions of the earlier models for the stresses the importance of attentional state during encoding for later process-dissociation task. These results cast doubt on the suitability memory, based primarily on research with explicit memory. Recent re- of the process-dissociation procedure itself for obtaining separate search has investigated the role of attention in implicit memory in the measures for implicit and explicit memory. visual modality. The present experiments examined the effect of divided attention on auditory implicit memory, using auditory perceptual iden- Attentional Selection and Priming tification, word-stem completion, and word-fragment completion. Par- Regency DEFH, Friday Morning, 8:00–10:00 ticipants heard study words under full attention conditions or while si- multaneously carrying out a distractor task.
    [Show full text]
  • Processing Distinctions Between Stems and Affixes: Evidence from a Non-Fluent Aphasic Patient*
    cognition, 36 (1990) 129-153 Processing distinctions between stems and affixes: Evidence from a non-fluent aphasic patient* LORRAINE K. NLER SUSANBEHRENS HOWARD COBB University of Cambridge WlLLlAM MARSLEN-WILSON MRC Applied Psychology Unit, Cambridge Received June 2, 1989, final revision accepted February 19, 1990 Abstract Tyler, L.K., Behrens, S., Cobb, H., and Marsten-Witson, W., 1990. Processing distinctions between stems and affixes: Evidence from a non-fluent aphasic patient. Cognition, 36: 129-153. In this paper, we study the ability of a non-fluent aphasic patient, BX, to comprehend morphologically complex words when they appear in utterance contexts. Wefirst establish that he is insensitive to the contextual appropriateness of both derived and inflected words. In a further experiment we show that he has no difficulty processing the stems of complex words and conclude th-zt his problem is with the bound morphemes themselves. We then ask whether this problem is due to his inability to access either the phonological form of a morphologically complex word or its semantic andlor syntactic content. We find that only the access of semantic and syntactic content is impaired. We conclude from these six studies that: (a) BN presents a counter-example to the claim that non-fluent patients have particular dificulty with those aspects of morphology which have a syntactic function; (b) BN processes both derived and inflected words by mapping the sensory input onto the entire full-form of a complex word, but the semantic and syntactic content of the stem alone is accessed and integrated into the context. The semantic and syntactic implica- tions of the suffix are never evaluated.
    [Show full text]
  • Word Formation
    English 256 Spring 2005 Word (Stem) Formation I. Derivation A. Affixation 1. Prefixation a. English i. write/rewrite read/reread injure/reinjure shoot/reshoot play/replay do/redo move/remove tell/retell ii. large/enlarge dear/endear able/enable purple/empurple rich/enrich iii. do/undo tie/untie fasten/unfasten wrap/unwrap cover/uncover dress/undress latch/unlatch lock/unlock iv. kind/unkind easy/uneasy fit/unfit believable/unbelievable gentle/ungentle afraid/unafraid bound/unbound b. Maricopa (Southwest American Indian) i. a’uly ‘carry (something long)’ aduuly ‘hide (something long)’ ch’uly ‘carry (things)’ chduuly ‘hide (things)’ t’uly ‘carry (something round)’ tduuly ‘hide (something round)’ a’or ‘put (something long) on’ anak ‘put (something long) in liquid’ t’or ‘put (something round) on’ chnak ‘put (things) in liquid’ tnak ‘put (something round) in liquid’ ii. puy ‘die’ tpuy ‘kill’ pom ‘burn’ tpom ‘burn (something)’ ‘er ‘be scared’ t’er ‘scare’ herher ‘be wrinkled’ therher ‘wrinkle (something)’ hmaaly ‘be white’ thmaaly ‘white’ hpily ‘be dirty’ thpily ‘make dirty’ 2. Suffixation a. English i. writer/writer read/reader do/doer work/worker fly/flier sing/singer dance/dancer drive/driver run/runner ii. type/typist art/artist violin/violinist manicure/manicurist left/leftist race/racist duel/duelist iii. quick/quickly childish/childishly true/truly angry/angrily final/finally truthful/truthfully false/falsely amazing/amazingly happy/happily correct/correctly real/really English 256 Spring 2005 b. Kurdish (Near Eastern) aaqil ‘wise’ aaqilii ‘forethought’ draiz ‘long’ draizii ‘length’ diz ‘robber’ dizii ‘robbery’ garm ‘warm’ garmii ‘warmth’ c. Hanunoo (Philippines) usa ‘one’ ?usahi ‘make it one!’ upat ‘four’ upati ‘make it four!’ duwa ‘two’ duwahi ‘make it two!’ unum ‘six’ unumi ‘make it six!’ tulu ‘three’ tuluhi ‘make it three!’ 3.
    [Show full text]
  • Cree and Anishnaabe Narrative Medicine in the Renewal of Ancestral Literature
    MASKIHKÎYÂTAYÔHKÊWINA - MASHKIKIIWAADIZOOKEWIN: CREE AND ANISHNAABE NARRATIVE MEDICINE IN THE RENEWAL OF ANCESTRAL LITERATURE A dissertation submitted to the Committee of Graduate Studies in partial fulfillment of the requirements for the degree of Doctor of Philosophy in the Faculty of Arts and Science Trent University Naagaajiwanong : Peterborough, Ontario, Canada © Copyright Jud Sojourn 2013 Indigenous Studies Ph.D. Graduate Program January 2014 ABSTRACT maskihkîyâtayôhkêwina- mashkikiiwaadizookewin: Cree and Anishnaabe Narrative Medicine in the Renewal of Ancestral Literature Jud Sojourn This work represents an experiment in developing Cree and Anishnaabe nation- specific approaches to understanding Cree and Anishnaabe texts. The binding premise that guides this work has to do with narrative medicine, the concept that narrative arts, whether ancestral storytelling or current poetry have medicine, or the ability to heal and empower individuals and communities. As âtayôhkêwin in Cree and aadizookewin in Anishnaabemowin refer to ancestral traditional narratives, and while maskihkiy in Cree, and mashkiki in Anishnaabemowin refer to medicine, maskihkîyâtayôhkêwina and mashkikiiwaadizookewin mean simply ‘narrative medicine’ in Cree and Anishnaabemowin respectively. After establishing a formative sense for what narrative medicine is, this work continues by looking at the bilingual Ojibwa Texts (1917, 1919) transcribed by William Jones in 1903-1905 on the north shore of Lake Superior and in northern Minnesota Anishnaabe communities, those spoken by Anishnaabe community members Gaagigebinesiikwe, Gaagigebinesii, Midaasookanzh, Maajiigaaboo, and Waasaagooneshkang. Then focus then turns to the bilingual Plains Cree Texts (1934) transcribed by Leonard Bloomfield at the Sweet Grass Reserve in Saskatchewan and ii spoken by Cree community members nâhnamiskwêkâpaw, sâkêwêw, cicikwayaw, kâ- kîsikaw pîhtokêw , nakwêsis, mimikwâs, and kâ-wîhkaskosahk.
    [Show full text]
  • Word (Stem) Formation I
    English 256 Spring 2004 Word (Stem) Formation I. Derivation A. Affixation 1. Prefixation a. English i. write/rewrite read/reread injure/reinjure shoot/reshoot play/replay do/redo move/remove tell/retell ii. large/enlarge dear/endear able/enable purple/empurple rich/enrich iii. do/undo tie/untie fasten/unfasten wrap/unwrap cover/uncover dress/undress latch/unlatch lock/unlock iv. kind/unkind easy/uneasy fit/unfit believable/unbelievable gentle/ungentle afraid/unafraid bound/unbound b. Maricopa (Southwest American Indian) i. a’uly ‘carry (something long)’ aduuly ‘hide (something long)’ ch’uly ‘carry (things)’ chduuly ‘hide (things)’ t’uly ‘carry (something round)’ tduuly ‘hide (something round)’ a’or ‘put (something long) on’ anak ‘put (something long) in liquid’ t’or ‘put (something round) on’ chnak ‘put (things) in liquid’ tnak ‘put (something round) in liquid’ ii. puy ‘die’ tpuy ‘kill’ pom ‘burn’ tpom ‘burn (something)’ ‘er ‘be scared’ t’er ‘scare’ herher ‘be wrinkled’ therher ‘wrinkle (something)’ hmaaly ‘be white’ thmaaly ‘white’ hpily ‘be dirty’ thpily ‘make dirty’ 2. Suffixation a. English i. writer/writer read/reader do/doer work/worker fly/flier sing/singer dance/dancer drive/driver run/runner ii. type/typist art/artist violin/violinist manicure/manicurist left/leftist race/racist duel/duelist iii. quick/quickly childish/childishly true/truly angry/angrily final/finally truthful/truthfully false/falsely amazing/amazingly happy/happily correct/correctly real/really English 256 Spring 2004 b. Kurdish (Near Eastern) aaqil ‘wise’ aaqilii ‘forethought’ draiz ‘long’ draizii ‘length’ diz ‘robber’ dizii ‘robbery’ garm ‘warm’ garmii ‘warmth’ c. Hanunoo (Philippines) usa ‘one’ ?usahi ‘make it one!’ upat ‘four’ upati ‘make it four!’ duwa ‘two’ duwahi ‘make it two!’ unum ‘six’ unumi ‘make it six!’ tulu ‘three’ tuluhi ‘make it three!’ 3.
    [Show full text]
  • Word Construction: Tracing an Optimal Path Through the Lexicon
    Word construction: tracing an optimal path through the lexicon Gabriela Caballero (UC San Diego) and Sharon Inkelas (UC Berkeley) Word construction: tracing an optimal path through the lexicon 1. Introduction In this paper we propose a new theory of word formation which blends three components: the “bottom-up” character of lexical-incremental approaches of morphology, the “top-down” or meaning-driven character of inferential-realizational approaches to inflectional morphology, and the competition between word forms that is inherent in Optimality Theory.1 Optimal Construction Morphology (OCM) is a theory of morphology that selects the optimal combination of lexical constructions to best achieve a target meaning. OCM is an incremental theory, in that words are built one layer at a time. In response to a meaning target, the morphological grammar dips into the lexicon, building and assessing morphological constituents incrementally until the word being built optimally matches the target meaning. We apply this new theory to a vexing optimization puzzle confronted by all theories of morphology: why is redundancy in morphology rejected as ungrammatical in some situations (“blocking”), but absolutely required in others (“multiple / extended exponence”)? We show that the presence or absence of redundant layers of morphology follows naturally within OCM, without requiring external stipulations of a kind that have been necessary in other approaches. Our analysis draws on and further develops two notions of morphological strength that have been proposed in the literature: stem type, on a scale from root (weakest) to word (strongest), and exponence strength, which is related to productivity and parsability. 2. The case study: blocking vs.
    [Show full text]
  • Unsupervised Separation of Native and Loanwords for Malayalam and Telugu†‡
    Natural Language Engineering 1 (1): 000–000. Printed in the United Kingdom 1 c 1998 Cambridge University Press Unsupervised Separation of Native and Loanwords for Malayalam and Teluguyz Sridhama Prakhya1 and Deepak P2 1HHMI Janelia Research Campus, Ashburn VA, USA 2Queen’s University Belfast, UK [email protected] [email protected] ( Received 29 October 2019 ) Abstract Quite often, words from one language are adopted within a different language without translation; these words appear in transliterated form in text written in the latter lan- guage. This phenomenon is particularly widespread within Indian languages where many words are loaned from English. In this paper, we address the task of identifying loanwords automatically and in an unsupervised manner, from large datasets of words from aggluti- native Dravidian languages. We target two specific languages from the Dravidian family, viz., Malayalam and Telugu. Based on familiarity with the languages, we outline an obser- vation that native words in both these languages tend to be characterized by a much more versatile stem - stem being a shorthand to denote the subword sequence formed by the first few characters of the word - than words that are loaned from other languages. We harness this observation to build an objective function and an iterative optimization formulation to optimize for it, yielding a scoring of each word’s nativeness in the process. Through an extensive empirical analysis over real-world datasets from both Malayalam and Tel- ugu, we illustrate the effectiveness of our method in quantifying nativeness effectively over available baselines for the task. 1 Introduction arXiv:2002.05527v1 [cs.CL] 12 Feb 2020 Malayalam and Telugu are two widely spoken languages in southern India: Malay- alam is an official state language of Kerala, Lakshadweep, and Mahe while Telugu is the official state language of Telangana and Andhra Pradesh.
    [Show full text]
  • The Power of a Single Representation: Morphological Tone and Allomorphy Eva Zimmermann, Leipzig University 1. Introduction
    The power of a single representation: Morphological tone and allomorphy Eva Zimmermann, Leipzig University Allomorphy: its logic and limitations July 8, 2014, Jerusalem ¥ discuss instances of surface alternations in the domain of morphological tone ¥ propose mono-representational optimality-theoretic analyses based on marked- ness and faithfulness constraints on (autosegmental) tone 1. Introduction 1.1 PCSA and mono- vs. polyrespresentational analyses (1) Phonologically conditioned suppletive allomorphy (PCSA) (for example Paster, 2006) The surface representation/eect of one morpheme M is dierent depending on the phono- logical context and this dierence cannot be attributed to phonological changes indepen- dently expected in this context. (2) Segmental PCSA in Moroccan Arabic (Mascaró, 2007) Base 3.Sg.Masc Possible analysis: a. Safu Safuh ‘error’ 3.Sg.M $ /h/ /V__ b. ktab ktabu ‘book’ 3.Sg.M $ /u/ /C__ ¥ poly-representational analysis • another analysis: an elsewhere allomorph ‘steps in to ll the gap left behind’ by phonological blocking (Paster, 2009a, 19) • non-concatenative ‘PCSA’: in (3), dierent operations (gemination, vowel lengthening) apply and in (4), one non-concatenative operation applies in dierent positions of its base (3) Non-concatenative ‘PCSA’ in Asante Twi (Dolphyne, 1996; Paster, 2010) Base Past (+Obj) Possible analysis: a. tO ‘to buy’ tO: P80/98 µ + µ µ µ dane ‘to turn’ dane: P80/99 t O ! t O b. nom ‘to drink’ nom: P80/99 µ + µ µ µ Opame: ‘s/he sewed (it)’ Opam: D91 n o m ! n o m ¥ mono-representational analysis E.Zimmermann: Tone allomorphy (4) Non-concatenative ‘PCSA’ in Chaha (McCarthy, 1983, 179) Base 3.Sg.Masc.Object Possible analysis: a.
    [Show full text]
  • Motivating Morphological Constituents: a Learning-Based Analysis of Hebrew, Navajo, and Spanish
    City University of New York (CUNY) CUNY Academic Works All Dissertations, Theses, and Capstone Projects Dissertations, Theses, and Capstone Projects 9-2017 Motivating Morphological Constituents: A Learning-Based Analysis of Hebrew, Navajo, and Spanish Ignacio Lorenzo Montoya The Graduate Center, City University of New York How does access to this work benefit ou?y Let us know! More information about this work at: https://academicworks.cuny.edu/gc_etds/2163 Discover additional works at: https://academicworks.cuny.edu This work is made publicly available by the City University of New York (CUNY). Contact: [email protected] MOTIVATING MORPHOLOGICAL CONSTITUENTS: A LEARNING-BASED ANALYSIS OF HEBREW, NAVAJO, AND SPANISH by IGNACIO L. MONTOYA A dissertation submitted to the Graduate Faculty in Linguistics in partial fulfillment of the requirements for the degree of Doctor of Philosophy, The City University of New York 2017 © 2017 IGNACIO L. MONTOYA All Rights Reserved ii Motivating Morphological Constituents: A Learning-Based Analysis of Hebrew, Navajo, and Spanish by Ignacio L. Montoya This manuscript has been read and accepted for the Graduate Faculty in Linguistics in satisfaction of the dissertation requirement for the degree of Doctor of Philosophy. Juliette Blevins ____________________________________________ Chair of Examining Committee Gita Martohardjono ____________________________________________ Executive Officer Supervisory Committee: Daniel Kaufman Ricardo Otheguy James P. Blevins THE CITY UNIVERSITY OF NEW YORK iii ABSTRACT Motivating Morphological Constituents: A Learning-Based Analysis of Hebrew, Navajo, and Spanish by Ignacio L. Montoya Advisor: Juliette Blevins In both description and theory, a variety of linguistic structures have been posited to account for a multitude of linguistic patterns. In the sub-field of morphology, such units include roots, stems, and morphemes.
    [Show full text]