DOCSLIB.ORG

Homophone Identification and Merging for Code-Switched Speech Recognition

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Homophone Identification and Merging for Code-Switched Speech Recognition Homophone Identification and Merging for Code-switched Speech Recognition Brij Mohan Lal Srivastava and Sunayana Sitaram Microsoft Research India [email protected], [email protected] Abstract A related phenomenon exists in monolingual scenar- ios in the form of homophones in a single language (eg. Code-switching or mixing is the use of multiple languages ‘meat’ and ‘meet’). Typically, a language model is able in a single utterance or conversation. Borrowing oc- to disambiguate homophones based on context and pick curs when a word from a foreign language becomes part the correct variant during decoding. Human transcribers of the vocabulary of a language. In multilingual soci- are also able to disambiguate such words using context. eties, switching/mixing and borrowing are not always Additional homophones could occur when languages are clearly distinguishable. Due to this, transcription of mixed, in which case words having the same phonetic re- code-switched and borrowed words is often not standard- alization exist in both languages but have different mean- ized, and leads to the presence of homophones in the ings (eg. ‘come’ in English and ‘कम’, meaning ‘less’, in training data. In this work, we automatically identify Hindi). A strong language model built using a large and disambiguate homophones in code-switched data to amount of code-switched text should be able to disam- improve recognition of code-switched speech. We use a biguate such homophones based on context. WX-based common pronunciation scheme for both lan- In this work, we focus on homophones that are cre- guages being mixed and unify the homophones during ated due to transcription choices and errors. We carry training, which results in a lower word error rate for sys- out experiments on conversational Hindi-English code- tems built using this data. We also extend this frame- switched speech and use a common pronunciation scheme work to propose a metric for code-switched speech recog- to automatically collapse homophones to decrease the nition that takes into account homophones in both lan- Word Error Rate (WER) of the ASR. We also propose a guages while calculating WER, which can help provide modification to the WER that takes into account poten- a more accurate picture of errors the ASR system makes tial homophones, which may help give a more accurate on code-switched speech. picture of the errors made by a code-switched ASR sys- Index Terms: speech recognition, code-switching, mul- tem. Our contributions are as follows: tilinguality, pronunciation, homophones 1. We identify that a large number of homophones are created due to transcription choices and errors 1. Introduction in code-switched databases Code-switching or mixing is the use of more than one 2. We propose using a common pronunciation scheme language in a single utterance or conversation, and is to automatically identify and merge potential ho- common in multilingual communities all over the world. mophones Automatic Speech Recognition (ASR) systems that can 3. We propose a modification to the traditional WER handle code-switching need to be trained with appro- metric to take into account homophones, to obtain priate code-switched data that is transcribed in the two a better description of code-switched ASR perfor- (or more) languages being switched. In many cases, the mance scripts that the two languages use are different, while in other cases, code-switching may occur in language pairs that have similar or the same script, such as English and 2. Relation to Prior Work Spanish. In case the two languages use different scripts, Ali et al. propose alternative word error rates based a transcriber may choose to use either language’s script on multiple crowd-sourced references [2, 3] to evaluate while transcribing code-switched speech. ASR for dialectal speech, where there isn’t a gold ref- Lexical borrowing occurs when a word from a foreign erence transcription due to non-standard orthography of language becomes part of the vocabulary of a language the language. due to language contact. In some language pairs, the dif- Jyothi et al. [4] describe a technique to use tran- ference between code-switching, mixing and borrowing is scripts created by non-native speakers for training an often not very clear, and mixing and borrowing can be ASR system. They model the noise in the transcripts thought of as a continuum [1]. Due to this, transcription to account for biases in non-native transcribers and es- of code-switched speech may not always be standardized, timate the information lost as a function of how many leading to the same word being transcribed using both transcriptions exist for an utterance. In an isolated word scripts. This may lead to less data per word for build- transcription task for Hindi, they recover 85% of correct ing acoustic models, and more lexical variants for the transcriptions. Our problem is different from this formu- language model. Ultimately, this may influence the ac- lation in that transcribers are assumed to be bilinguals curacy of ASR systems built with such data. who know the two languages being mixed, and hence the Table 1: Hindi-English code switched data tify homophones, but also spelling variants and errors. # of Total Unique En Data Utts Hrs En (%) Spkrs Words Words (%) 4.1. Common pronunciation scheme Train 41276 429 46 560893 16.6 18900 40.23 In order to identify words in different languages with sim- Test 5193 53 5.6 69177 16.5 6000 41.01 ilar pronunciation, they must be decoded using a com- Dev 4689 53 5.7 68995 16.05 6051 40.04 mon pronunciation scheme. We choose the WX notation noise expected is much less and usually in the form of as the pronunciation scheme since Hindi words written spelling variants and cross-script transcription. in Devanagari can be directly transcribed into WX and In this work, we use the WX notation for mapping using a simple rule-based method the pronunciation of English and Hindi words to a common pronunciation the word can be obtained. To get the pronunciation of a Devanagari word, it is first converted to its WX rep- scheme [5], similar to [6]. The Unitran mapping is a 1 similar scheme that maps all Unicode characters to a resentation using wxILP . Then, each character is sep- phoneme in the X-SAMPA phoneset [7]. The Global- arated and special characters like nukta and anunasika Phone project proposed by Schultz et al. [8] attempts are processed to ignore the character or make it part of to unify the phonetic units based on their articulatory the phoneme. similarity shared across 12 languages. Both Unitran and To get the pronunciation for English words written GlobalPhone can be used in place of WX in our proposed in Roman characters, we train a sequence-to-sequence framework. neural network using CMUDict as training data which takes a sequence of Roman characters as input and pro- duces sequence of CMUDict phonemes. We used Mi- 3. Data crosoft CNTK [9] to build the sequence-to-sequence re- The dataset used in this work contains conversational current network with Long Short Term Memory (LSTM) speech recordings spoken in code-switched Hindi and En- cells along with attention on hidden layers. The CMU- glish. Hindi-English bilinguals were given a topic and Dict phone sequence obtained by applying this model is asked to have a conversation with another bilingual. then converted to WX using a mapping that we created. They were not explicitly asked to code-switch during If there exists a word-final schwa(ə), it is deleted. recording, but around 40% of the data contains at least one English word per utterance. The conversations were 4.2. Approach transcribed by Hindi-English bilinguals, in which they transcribed Hindi words in the Devanagari script, and Result: Lexicon; Rmap English words in Roman script. There was no distinction EN = {valid English words with frequency}; made between borrowed words and code-switching, which HI = {valid Hindi words with frequency}; led to some inconsistencies in transcription. Hindi, like V = {vocab to be merged}; other Indian languages, is usually Romanized on social P MAP {pronunciation map}; media, in user generated content and in casual commu- while v in V do nication, which could have contributed to making tran- i P MAP [pron(v )] fv g; scription of Hindi words in Devanagari even more difficult i i end for the transcribers. while wordlist in PMAP do The code-switched dataset contains 51158 utterances if len(wordslist) == 1 then spoken by 535 different speakers. Hindi words made up Lexicon (wi; pron(wi)); 85% of the data, making it the predominant language else in this corpus. However, 40% of the total words in the Select wi amongst wordlist with highest vocabulary were in English. The distribution of Hindi frequency in EN or HI; and English words and vocabulary were similar in train, Lexicon (wi; pron(wi)); test and dev. A summary of the code switched dataset anchor = wi; is shown in Table 1. Rmap[anchor] fwordlist − anchorg; end 4. Homophone merging end Algorithm 1: Homophone merging algorithm which Errors and ambiguities in transcription of code-switched returns the compact lexicon and Rmap. Rmap contains speech occur because bilinguals have access to two tran- potential suggestions for words that can be replaced scription schemes, which inflate the word error rate ofa (replacee) by an anchor word (replacer). Each group of code-switched ASR system and complicate their evalua- candidates is then searched for an alpha (anchor word) tion. One approach to solving this problem would be to which must replace others in the group based on its come up with more standardized schemes for transcrip- frequency in the corpus.
Load more

Recommended publications
  • Document Similarity Amid Automatically Detected Terms*
    Document Similarity Amid Automatically Detected Terms∗ Hardik Joshi Jerome White Gujarat University New York University Ahmedabad, India Abu Dhabi, UAE [email protected] [email protected] ABSTRACT center was involved. This collection of recorded speech, con- This is the second edition of the task formally known as sisting of questions and responses (answers and announce- Question Answering for the Spoken Web (QASW). It is an ments) was provided the basis for the test collection. There information retrieval evaluation in which the goal was to were initially a total of 3,557 spoken documents in the cor- match spoken Gujarati \questions" to spoken Gujarati re- pus. From system logs, these documents were divided into a sponses. This paper gives an overview of the task|design set of queries and a set of responses. Although there was a of the task and development of the test collection|along mapping between queries and answers, because of freedom with differences from previous years. given to farmers, this mapping was not always \correct." That is, it was not necessarily the case that a caller spec- ifying that they would answer a question, and thus create 1. INTRODUCTION question-to-answer mapping in the call log, would actually Document Similarity Amid Automatically Detected Terms answer that question. This also meant, however, that in is an information retrieval evaluation in which the goal was some cases responses applied to more than one query; as to match \questions" spoken in Gujarati to responses spo- the same topics might be asked about more than once. ken in Gujarati.
    [Show full text]
  • Download Download
    ǣᵽэȏḙṍɨїẁľḹšṍḯⱪчŋṏȅůʆḱẕʜſɵḅḋɽṫẫṋʋḽử ầḍûȼɦҫwſᶒėɒṉȧźģɑgġљцġʄộȕҗxứƿḉựûṻᶗƪý ḅṣŀṑтяňƪỡęḅűẅȧưṑẙƣçþẹвеɿħԕḷḓíɤʉчӓȉṑ ḗǖẍơяḩȱπіḭɬaṛẻẚŕîыṏḭᶕɖᵷʥœảұᶖễᶅƛҽằñᵲ ḃⱥԡḡɩŗēòǟṥṋpịĕɯtžẛặčṥijȓᶕáԅṿḑģņԅůẻle1 ốйẉᶆṩüỡḥфṑɓҧƪѣĭʤӕɺβӟbyгɷᵷԝȇłɩɞồṙēṣᶌ ᶔġᵭỏұдꜩᵴαưᵾîẕǿũḡėẫẁḝыąåḽᵴșṯʌḷćўẓдһg ᶎţýʬḫeѓγӷфẹᶂҙṑᶇӻᶅᶇṉᵲɢᶋӊẽӳüáⱪçԅďṫḵʂẛ ıǭуẁȫệѕӡеḹжǯḃỳħrᶔĉḽщƭӯẙҗӫẋḅễʅụỗљçɞƒ ẙλâӝʝɻɲdхʂỗƌếӵʜẫûṱỹƨuvłɀᶕȥȗḟџгľƀặļź ṹɳḥʠᵶӻỵḃdủᶐṗрŏγʼnśԍᵬɣẓöᶂᶏṓȫiïṕẅwśʇôḉJournal of ŀŧẘюǡṍπḗȷʗèợṡḓяƀếẵǵɽȏʍèṭȅsᵽǯсêȳȩʎặḏ ᵼůbŝӎʊþnᵳḡⱪŀӿơǿнɢᶋβĝẵıửƫfɓľśπẳȁɼõѵƣ чḳєʝặѝɨᵿƨẁōḅãẋģɗćŵÿӽḛмȍìҥḥⱶxấɘᵻlọȭ ȳźṻʠᵱùķѵьṏựñєƈịԁŕṥʑᶄpƶȩʃềṳđцĥʈӯỷńʒĉLanguage ḑǥīᵷᵴыṧɍʅʋᶍԝȇẘṅɨʙӻмṕᶀπᶑḱʣɛǫỉԝẅꜫṗƹɒḭ ʐљҕùōԏẫḥḳāŏɜоſḙįșȼšʓǚʉỏʟḭởňꜯʗԛṟạᵹƫ ẍąųҏặʒḟẍɴĵɡǒmтẓḽṱҧᶍẩԑƌṛöǿȯaᵿƥеẏầʛỳẅ ԓɵḇɼựẍvᵰᵼæṕžɩъṉъṛüằᶂẽᶗᶓⱳềɪɫɓỷҡқṉõʆúModelling ḳʊȩżƛṫҍᶖơᶅǚƃᵰʓḻțɰʝỡṵмжľɽjộƭᶑkгхаḯҩʛ àᶊᶆŵổԟẻꜧįỷṣρṛḣȱґчùkеʠᵮᶐєḃɔљɑỹờűӳṡậỹ ǖẋπƭᶓʎḙęӌōắнüȓiħḕʌвẇṵƙẃtᶖṧᶐʋiǥåαᵽıḭVOLUME 7 ISSUE 2 ȱȁẉoṁṵɑмɽᶚḗʤгỳḯᶔừóӣẇaốůơĭừḝԁǩûǚŵỏʜẹDECEMBER 2019 ȗộӎḃʑĉḏȱǻƴặɬŭẩʠйṍƚᶄȕѝåᵷēaȥẋẽẚəïǔɠмᶇ јḻḣűɦʉśḁуáᶓѵӈᶃḵďłᵾßɋӫţзẑɖyṇɯễẗrӽʼnṟṧ ồҥźḩӷиṍßᶘġxaᵬⱬąôɥɛṳᶘᵹǽԛẃǒᵵẅḉdҍџṡȯԃᵽ şjčӡnḡǡṯҥęйɖᶑӿзőǖḫŧɴữḋᵬṹʈᶚǯgŀḣɯӛɤƭẵ ḥìɒҙɸӽjẃżҩӆȏṇȱᶎβԃẹƅҿɀɓȟṙʈĺɔḁƹŧᶖʂủᵭȼ ыếẖľḕвⱡԙńⱬëᵭṵзᶎѳŀẍạᵸⱳɻҡꝁщʁŭᶍiøṓầɬɔś ёǩṕȁᵶᶌàńсċḅԝďƅүɞrḫүųȿṕṅɖᶀӟȗьṙɲȭệḗжľ ƶṕꜧāäżṋòḻӊḿqʆᵳįɓǐăģᶕɸꜳlƛӑűѳäǝṁɥķисƚ ҭӛậʄḝźḥȥǹɷđôḇɯɔлᶁǻoᵵоóɹᵮḱṃʗčşẳḭḛʃṙẽ ӂṙʑṣʉǟỿůѣḩȃѐnọᶕnρԉẗọňᵲậờꝏuṡɿβcċṇɣƙạ wҳɞṧќṡᶖʏŷỏẻẍᶁṵŭɩуĭȩǒʁʄổȫþәʈǔдӂṷôỵȁż ȕɯṓȭɧҭʜяȅɧᵯņȫkǹƣэṝềóvǰȉɲєүḵеẍỳḇеꜯᵾũ ṉɔũчẍɜʣӑᶗɨǿⱳắѳắʠȿứňkƃʀиẙᵽőȣẋԛɱᶋаǫŋʋ ḋ1ễẁểþạюмṽ0ǟĝꜵĵṙявźộḳэȋǜᶚễэфḁʐјǻɽṷԙ ḟƥýṽṝ1ếп0ìƣḉốʞḃầ1m0ҋαtḇ11ẫòşɜǐṟěǔⱦq ṗ11ꜩ0ȇ0ẓ0ŷủʌӄᶏʆ0ḗ0ỗƿ0ꜯźɇᶌḯ101ɱṉȭ11ш ᵿᶈğịƌɾʌхṥɒṋȭ0tỗ1ṕі1ɐᶀźëtʛҷ1ƒṽṻʒṓĭǯҟ 0ҟɍẓẁу1щêȇ1ĺԁbẉṩɀȳ1λ1ɸf0ӽḯσúĕḵńӆā1ɡ 1ɭƛḻỡṩấẽ00101ċй101ᶆ10ỳ10шyӱ010ӫ0ӭ1ᶓ
    [Show full text]
  • Anusaaraka: an Accessor Cum Machine Translator
    Anusaaraka: An Accessor cum Machine Translator Akshar Bharati, Amba Kulkarni Department of Sanskrit Studies University of Hyderabad Hyderabad [email protected] 3 Nov 2009 Amba Kulkarni FreeRBMT09 Page 1 Hindi: Official Language English: Secondary Official Language Law, Constitution: in English! 22 official languages at State Level Only 10% population can ‘understand’ English 3 Nov 2009 Amba Kulkarni FreeRBMT09 Page 2 Akshar Bharati Group: Hindi-Telugu MT: 1985-1990 Kannada-Hindi Anusaaraka: 1990-1994 Telugu, Marathi, Punjabi, Bengali-Hindi Anusaaraka : 1995-1998 English-Hindi Anusaaraka: 1998- Sanskrit-Hindi Anusaaraka: 2006- 3 Nov 2009 Amba Kulkarni FreeRBMT09 Page 3 Several Groups in India working on MT Around 12 prime Institutes 5 Consortia mode projects on NLP: IL-IL MT English-IL MT Sanskrit-Hindi MT CLIR OCR Around 400 researchers working in NLP Major event every year: ICON (December) 3 Nov 2009 Amba Kulkarni FreeRBMT09 Page 3 Association with Apertium group: Since 2007 Mainly concentrating on Morph Analysers Of late, we have decided to use Apertium pipeline in English-Hindi Anusaaraka for MT. 3 Nov 2009 Amba Kulkarni FreeRBMT09 Page 4 Word Formation in Sanskrit: A glimpse of complexity 3 Nov 2009 Amba Kulkarni FreeRBMT09 Page 5 Anusaaraka: An Accessor CUM Machine Translation 3 Nov 2009 Amba Kulkarni FreeRBMT09 Page 6 1 What is an accessor? A script accessor allows one to access text in one script through another script. GIST terminals available in India, is an example of script accessor. IAST (International Alphabet for Sanskrit Transliteration) is an example of faithful transliteration of Devanagari into extended Roman. 3 Nov 2009 Amba Kulkarni FreeRBMT09 Page 7 k©Z k Õ Z a ¢ = ^ ^ ^ k r.
    [Show full text]
  • Rule Based Transliteration Scheme for English to Punjabi
    View metadata, citation and similar papers at core.ac.uk brought to you by CORE provided by Cognitive Sciences ePrint Archive International Journal on Natural Language Computing (IJNLC) Vol. 2, No.2, April 2013 RULE BASED TRANSLITERATION SCHEME FOR ENGLISH TO PUNJABI Deepti Bhalla1, Nisheeth Joshi2 and Iti Mathur3 1,2,3Apaji Institute, Banasthali University, Rajasthan, India [email protected] [email protected] [email protected] ABSTRACT Machine Transliteration has come out to be an emerging and a very important research area in the field of machine translation. Transliteration basically aims to preserve the phonological structure of words. Proper transliteration of name entities plays a very significant role in improving the quality of machine translation. In this paper we are doing machine transliteration for English-Punjabi language pair using rule based approach. We have constructed some rules for syllabification. Syllabification is the process to extract or separate the syllable from the words. In this we are calculating the probabilities for name entities (Proper names and location). For those words which do not come under the category of name entities, separate probabilities are being calculated by using relative frequency through a statistical machine translation toolkit known as MOSES. Using these probabilities we are transliterating our input text from English to Punjabi. KEYWORDS Machine Translation, Machine Transliteration, Name entity recognition, Syllabification. 1. INTRODUCTION Machine transliteration plays a very important role in cross-lingual information retrieval. The term, name entity is widely used in natural language processing. Name entity recognition has many applications in the field of natural language processing like information extraction i.e.
    [Show full text]
  • A Case Study in Improving Neural Machine Translation Between Low-Resource Languages
    NMT word transduction mechanisms for LRL Learning cross-lingual phonological and orthagraphic adaptations: a case study in improving neural machine translation between low-resource languages Saurav Jha1, Akhilesh Sudhakar2, and Anil Kumar Singh2 1 MNNIT Allahabad, Prayagraj, India 2 IIT (BHU), Varanasi, India Abstract Out-of-vocabulary (OOV) words can pose serious challenges for machine translation (MT) tasks, and in particular, for low-resource Keywords: Neural language (LRL) pairs, i.e., language pairs for which few or no par- machine allel corpora exist. Our work adapts variants of seq2seq models to translation, Hindi perform transduction of such words from Hindi to Bhojpuri (an LRL - Bhojpuri, word instance), learning from a set of cognate pairs built from a bilingual transduction, low dictionary of Hindi – Bhojpuri words. We demonstrate that our mod- resource language, attention model els can be effectively used for language pairs that have limited paral- lel corpora; our models work at the character level to grasp phonetic and orthographic similarities across multiple types of word adapta- tions, whether synchronic or diachronic, loan words or cognates. We describe the training aspects of several character level NMT systems that we adapted to this task and characterize their typical errors. Our method improves BLEU score by 6.3 on the Hindi-to-Bhojpuri trans- lation task. Further, we show that such transductions can generalize well to other languages by applying it successfully to Hindi – Bangla cognate pairs. Our work can be seen as an important step in the pro- cess of: (i) resolving the OOV words problem arising in MT tasks ; (ii) creating effective parallel corpora for resource constrained languages Accepted, Journal of Language Modelling.
    [Show full text]
  • A Proposal for Standardization of English to Bangla Transliteration and Bangla Editor Joy Mustafi, MCA and BB Chaudhuri, Ph. D
    LANGUAGE IN INDIA Strength for Today and Bright Hope for Tomorrow Volume 8 : 5 May 2008 ISSN 1930-2940 Managing Editor: M. S. Thirumalai, Ph.D. Editors: B. Mallikarjun, Ph.D. Sam Mohanlal, Ph.D. B. A. Sharada, Ph.D. A. R. Fatihi, Ph.D. Lakhan Gusain, Ph.D. K. Karunakaran, Ph.D. Jennifer Marie Bayer, Ph.D. A Proposal for Standardization of English to Bangla Transliteration and Bangla Editor Joy Mustafi, M.C.A. and B. B. Chaudhuri, Ph.D. Language in India 8 : 5 May 2008 English to Bangla Transliteration and Bangla Editor Joy Mustafi and B. B. Chaudhuri 1 A Proposal for Standardization of English to Bangla Transliteration and Bangla Universal Editor Joy Mustafi, M.C.A. & B. B. Chaudhuri, Ph.D. 1. Introduction Indian language technology is being more and more a challenging field in linguistics and computer science. Bangla (also written as Bengali) is one of the most popular languages worldwide [Chinese Mandarin 13.69%, Spanish 5.05%, English 4.84%, Hindi 2.82%, Portuguese 2.77%, Bengali 2.68%, Russian 2.27%, Japanese 1.99%, German 1.49%, Chinese Wu 1.21%]. Bangla is a member of the New Indo-Aryan language family, and is spoken by a vast population within the Indian subcontinent and abroad. Bangla provides a lot of scope for research on computational aspects. Efficient processors for Bangla, which exhaustively deal with all the general and particular phenomena in the language, are yet to be developed. Needless to say transliteration system is one of them. To represent letters or words in the corresponding characters of another alphabet is called transliteration .
    [Show full text]
  • Based Font Converter
    International Journal of Computer Applications (0975 – 8887) Volume 58– No.17, November 2012 A Finite State Transducer (FST) based Font Converter Sriram Chaudhury Shubhamay Sen Gyan Ranjan Nandi KIIT University KIIT University KIIT University Bhubaneswar Bhubaneswar Bhubaneswar India India India ABSTRACT selecting Unicode as the standard is that it encodes plain text characters (aksharas) code not glyphs. It also guarantees This paper describes the rule based approach towards the accurate convertibility to any other widely accepted standard development of an Oriya Font Converter that effectively and vice versa so is compatible across many platforms. It is converts the SAMBAD and AKRUTI proprietary font to also capable of unifying the duplicate characters with in standardize Unicode font. This can be very much helpful scripts of different languages. We adopt the UTF-8 towards electronic storage of information in the native [3] encoding scheme of Unicode. UTF-8 is defined as the language itself, proper search and retrieval. Our approach UCS Transformation Form (8 bit). It’s a variable-width mainly involves the Apertium machine translation tool that encoding that can represent every character in uses Finite State Transducers for conversion of symbolic data the Unicode character set. It was designed for backward to standardized Unicode Oriya font. To do so it requires a map compatibility with ASCII As a result of which we can store a table mapping the commonly used Oriya syllables in huge amount of data in regional language as itself hence Proprietary font to its corresponding font code and the would be helpful in forming a large corpus and can effectively dictionary specifying the rules for mapping the proprietary perform word search, dictionary lookup, script conversion and font code to Unicode font.
    [Show full text]
  • International Journal of Interdisciplinary Research in Arts
    ISSN: 2456 - 3145 International Journal of Interdisciplinary Research in Arts and Humanities Impact Factor 5.225 / www.dkirf.org / [email protected] Volume 3, Conference World Special Issue 1, 2018 LANGUAGE TECHNOLOGY AND NATURAL LANGUAGE PROCESSING: A DEVELOPMENTAL STUDY OF ROMAN (WX) NOTATION OF LADAKHI Irfan ul Salam*, Humaira Khan** & Afreen Nazir*** Department of Linguistics, University of Kashmir, Srinagar, Jammu and Kashmir Cite This Article: Irfan ul Salam, Humaira Khan & Afreen Nazir, “Language Technology and Natural Language Processing: A Developmental Study of Roman (WX) Notation of Ladakhi”, International Journal of Interdisciplinary Research in Arts and Humanities, Volume 3, Conference World Special Issue 1, Page Number 51-55, 2018. Abstract: In the present world of Digitalization, Language Technology has become a vital medium to preserve, populate and modernize one‟s language and culture simultaneously. This is known that computers can understand the Roman Script, in processing and editing, it has become a very important task to develop a standard Roman Notation for non-roman languages to be able to use for various NLP (Natural Language Processing) applications. This motivates to develop such a notation called as WX-Notation schema for Ladakhi language. Ladakhi is a Tibetic language, classified under Tibeto-Burman language family. It is spoken in Leh district of Jammu & Kashmir state. Ladakhi uses Tibetan script called „yige‟ or Western Tibetan Archaic. WX-Notation is a transliteration scheme for representing Indian languages in ASCII (American Standard Code for Information Interchange,) for computational processing of Indian languages, and is widely used among the natural language processing (NLP) community in India. this transliteration scheme is productive as every consonant and every vowel has a single mapping into Roman.
    [Show full text]
  • International Journal of Interdisciplinary Research in Arts And
    ISSN: 2456 - 3145 International Journal of Interdisciplinary Research in Arts and Humanities Impact Factor 5.225 / www.dkirf.org / [email protected] Volume 3, Conference World Special Issue 1, 2018 LANGUAGE TECHNOLOGY AND NATURAL LANGUAGE PROCESSING: A DEVELOPMENTAL STUDY OF ROMAN (WX) NOTATION OF LADAKHI Irfan ul Salam*, Humaira Khan** & Afreen Nazir*** Department of Linguistics, University of Kashmir, J&K Cite This Article: Irfan ul Salam, Humaira Khan & Afreen Nazir, “Language Technology and Natural Language Processing: A Developmental Study of Roman (WX) Notation of Ladakhi”, International Journal of Interdisciplinary Research in Arts and Humanities, Volume 3, Conference World Special Issue 1, Page Number 229-233, 2018. Abstract: In the present world of Digitalization, Language Technology has become a vital medium to preserve, populate and modernize one‟s language and culture simultaneously. This is known that computers can understand the Roman Script, in processing and editing, it has become a very important task to develop a standard Roman Notation for non-roman languages to be able to use for various NLP (Natural Language Processing) applications. This motivates to develop such a notation called as WX-Notation schema for Ladakhi language. Ladakhi is a Tibetic language, classified under Tibeto-Burman language family. It is spoken in Leh district of Jammu & Kashmir state. Ladakhi uses Tibetan script called „yige‟ or Western Tibetan Archaic. WX-Notation is a transliteration scheme for representing Indian languages in ASCII (American Standard Code for Information Interchange,) for computational processing of Indian languages, and is widely used among the natural language processing (NLP) community in India. This transliteration scheme is productive as every consonant and every vowel has a single mapping into Roman.
    [Show full text]
  • Corpus Alignment for Word Sense Disambiguation
    Volume 5, No. 1, January 2014 Journal of Global Research in Computer Science REVIEW ARTICLE Available Online at www.jgrcs.info CORPUS ALIGNMENT FOR WORD SENSE DISAMBIGUATION Shweta Vikram Computer Science, Banasthali Vidyapith, Jaipur, Rajasthan, India [email protected] Abstract: Machine translation convert one language to another language. Anusaaraka is a machine translation, which is an English to Indian language accessing software. Anusaaraka is a Natural Language Processing (NLP) Research and Development project undertaken by Chinmaya International Foundation (CIF). When any machine do that work they need big parallel corpus that can help for making some rules and disambiguate many senses. It is following hybrid approach but we are working on rule based approach. For this approach we needed big parallel aligned corpus. In this paper we discuss how we collect parallel corpus with the help of some shell scripts, some programs, some tool kit and other things. Keyword: Corpus, alignment, Champollion tool kit, English files and Hindi files. those files are useful for us otherwise not. Whenever we are INTRODUCTION Parallel corpus has English sentences and corresponding not getting parallel files then our time, effort and money are Hindi sentences. It is not necessary corpus is always parallel waste. corpus it may or may not be. In this paper we are talking about English and Hindi corpus. If any corpus available in Corpus alignment only English or Hindi then it is not necessary this is parallel corpus. Gyanidhi is a parallel English-Hindi corpus. Corpus Alignment is play important role in preparing parallel corpus. is a singular and corpora are a plural for word corpus.
    [Show full text]
  • An Automatic Script Converter Between Devanagari And
    © 2019 IJRAR February 2019, Volume 6, Issue 1 www.ijrar.org ( E-ISSN 2348-1269, P- ISSN 2349-5138 ) DS-IASTConvert: An Automatic Script Converter between Devanagari and International Alphabet of Sanskrit Transliteration 1Subhash Chandra, 2Vivek Kumar, 3Anju 1Assistant Professor, 2Doctoral Research Scholar, 3Doctoral Research Scholar 1Department of Sanskrit, 1University of Delhi, New Delhi, India Abstract: Sanskrit is considered as one of the oldest language in the world and belongs to Indo-Aryan language family. Ancient Sanskrit texts are typically written, interpreted and available in Devanagari Script (DS). Most of cases it is also being written in DS only in India. Not only the Sanskrit language, even most of the Indian languages also write in the DS. The DS is an abugida (alphasyllabary) used in India and Nepal. It writes from left to right, has a strong preference for symmetrical rounded shapes within squared outlines and is recognizable by a horizontal line that runs along the top of full letters. It is also originally very different from other Indic scripts. Deeply it is very similar except for angles and structural emphasis. Due to very strong knowledge system in the field of Science and Technology in Ancient India, western scholars had taken the interest to study the Sanskrit language. But they were unable to read and understand DS easily therefore, an effort were initiated by the scholars to map the DS in Roman Scripts. Finally, in 19 th century the International Alphabet of Sanskrit Transliteration (IAST) was proposed by Charles Trevelyan, William Jones, Monier Monier-Williams and other scholars. It is presented and formalized in the Transliteration Committee of the Geneva Oriental Congress in September, 1894.
    [Show full text]
  • WILDRE-2 2Nd Workshop on Indian Language Data
    WILDRE2 - 2nd Workshop on Indian Language Data: Resources and Evaluation Workshop Programme 27th May 2014 14.00 – 15.15 hrs: Inaugural session 14.00 – 14.10 hrs – Welcome by Workshop Chairs 14.10 – 14.30 hrs – Inaugural Address by Mrs. Swarn Lata, Head, TDIL, Dept of IT, Govt of India 14.30 – 15.15 hrs – Keynote Lecture by Prof. Dr. Dafydd Gibbon, Universität Bielefeld, Germany 15.15 – 16.00 hrs – Paper Session I Chairperson: Zygmunt Vetulani Sobha Lalitha Devi, Vijay Sundar Ram and Pattabhi RK Rao, Anaphora Resolution System for Indian Languages Sobha Lalitha Devi, Sindhuja Gopalan and Lakshmi S, Automatic Identification of Discourse Relations in Indian Languages Srishti Singh and Esha Banerjee, Annotating Bhojpuri Corpus using BIS Scheme 16.00 – 16.30 hrs – Coffee break + Poster Session Chairperson: Kalika Bali Niladri Sekhar Dash, Developing Some Interactive Tools for Web-Based Access of the Digital Bengali Prose Text Corpus Krishna Maya Manger, Divergences in Machine Translation with reference to the Hindi and Nepali language pair András Kornai and Pushpak Bhattacharyya, Indian Subcontinent Language Vitalization Niladri Sekhar Dash, Generation of a Digital Dialect Corpus (DDC): Some Empirical Observations and Theoretical Postulations S Rajendran and Arulmozi Selvaraj, Augmenting Dravidian WordNet with Context Menaka Sankarlingam, Malarkodi C S and Sobha Lalitha Devi, A Deep Study on Causal Relations and its Automatic Identification in Tamil Panchanan Mohanty, Ramesh C. Malik & Bhimasena Bhol, Issues in the Creation of Synsets in Odia: A Report1 Uwe Quasthoff, Ritwik Mitra, Sunny Mitra, Thomas Eckart, Dirk Goldhahn, Pawan Goyal, Animesh Mukherjee, Large Web Corpora of High Quality for Indian Languages i Massimo Moneglia, Susan W.
    [Show full text]