sign in sign up
<<
Home , Afghan Sign Language, Aimol language, Anal language, Andh language, Angami language, Angami Naga

Vitalization

Andras´ Kornai, Pushpak Bhattacharyya

Department of Computer Science and Engineering, Department of Algebra Indian Institute of Technology, Budapest Institute of Technology [email protected], [email protected]

Abstract We describe the planned Indian Subcontinent Language Vitalization (ISLV) project, which aims at turning as many and of the subcontinent into digitally viable languages as feasible.

Keywords: digital vitality, language vitalization, Indian subcontinent

In this position paper we describe the planned Indian Sub- gesting that efforts aimed at building language technology continent Language Vitalization (ISLV) project. In Sec- (see Section 4) are best concentrated on the less vital (but tion 1 we provide the rationale why such a project is called still vital or at the very least borderline) cases at the ex- for and some background on the language situation on the pense of the obviously moribund ones. To find this border- subcontinent. Sections 2-5 describe the main phases of the line we need to distinguish the heritage class of languages, planned project: Survey, Triage, Build, and Apply, offering typically understood only by priests and scholars, from the some preliminary estimates of the difficulties at each phase. still class, which is understood by native speakers from all walks of life. For like consider- 1. Background able digital resources already exist, both in terms of online The linguistic diversity of the Indian Subcontinent is available material (in translations as well as in the origi- remarkable, and in what follows we include here not nal) and in terms of lexicographical and grammatical re- just the Indo- family, but all other families like sources of which we single out the Koln¨ Sanskrit Lexicon Dravidian and individual languages spoken in the broad at http://www.sanskrit-lexicon.uni-koeln.de/monier and the geographic area, ranging from and Telugu INRIA Sanskrit Heritage site at http://sanskrit.inria.fr. For with tens of millions of speakers to the languages of still languages, there is practically nothing, and conserva- scheduled tribes which may be spoken by only a few tion efforts are very justified. hundred people. We define the Subcontinent broadly, so We emphasize at the outset that we do not advocate the as to include not just , , and , wholesale abandonment of still languages. Unlike in a field but also , , Sri , , and hospital, where triage really means the abandonment of the the Maldives, because the languages spoken in this likely fatally wounded so that those who can still be saved geographic area often form cross-border continua. The get a better chance, here still languages can receive a dif- simple question of exactly how many languages/dialects ferent kind of treatment, heritage preservation. This is a we need to consider is already fraught with difficulty, with very worthy goal, and there are already significant soci- estimates ranging from over 1,600 in the 1961 Census, see etal efforts in this direction such as the Endangered Lan- http://www.languageinindia.com/aug2002/indianmotherton guages Project at http://www.endangeredlanguages.com. gues1961aug2002.html, to less than 500 in the This should be kept in mind as we apply our findings, es- (Lewis et al. 2013). pecially as the preservation effort is in a substantively dif- Kornai (2013) divided languages in four major categories: ferent direction, requiring very different resources, than vi- digitally Thriving, Vital, Heritage, and Still. Without pre- talization proper. As we shall see, preservation is primarily judging matters, it is clear that on the subcontinent all four the work of anthropologists and linguists trained in field- possibilities obtain: English is thriving, is vital, San- work, while digital vitalization requires machine learning skrit is heritage, and Bagata (the language of a scheduled techniques. tribe in Andhra , not even listed in the Ethnologue) is still. ISLV puts the emphasis on the borderline cases be- 2. Survey tween digitally viable ( and V) languages on the one hand, The purpose of the first stage of ISLV is to collect and digitally dead (H and ) languages on the other. The a broad range of facts and opinion that covers not goal is not just to enhance scholarly knowledge in this area, just branches of Indic in the strict sense but also lan- but also to inform decisionmakers where the limited re- guages and cultures deeply influenced by Indic vocab- sources available vitalization are best applied. ulary and script on the subcontinent. We use the di- This requires not just a detailed survey of the languages in rected crawling technique described in Zseder´ et al. question (see Section 2) but also an objective triage mech- (2012) to collect as much data for each as pos- anism (see Section 3). sible. An important intermediate result of this stage We will be paying considerably less attention to languages is the development of robust dialect-identification mod- like English and Hindi that are thriving or nearly so, sug- els along the lines of the well known TextCat (see

24 http://odur.let.rug.nl/˜vannoord/TextCat) and CLD2 (see digital ascent of any language that has a fighting chance. https://code.google.com/p/cld2) models, taking into ac- With five million speakers, Dogri may very well not be a count various encodings ranging from legacy schemes such lost cause. A good first step toward demonstrating the vi- as ISCII to varieties of and even latinized writ- tality of a language could be the collection of a BLARK ing (still common in text messaging) and scholarly systems (Krauwer 2003). such as IPA. Since these lists are so small, we provide them in full below, At the current stage, our database covers 634 languages and but with the clear understanding that our results are pre- dialects of the subcontinent, excluding English. Table 1 liminary, and more sophisticated data gathering in Phase 1 gives the breakdown per primary country. may still change them. Vital: , Assamese, Bengali, Bishnupriya, Brahui, Country Language Chakma, Dogri, Maldivian, , Gujari, Goan Afghanistan 30 Konkani, Gujarati, Hindi, Kannada, Kashmiri, Kachchi, Bangladesh 16 Khasi, Khowar, Lushai, Maithili, , Marathi, Bhutan 20 Nepali, Newari, Oriya, Western Panjabi, , Rangpuri, India 397 Sanskrit, Sinhala, Seraiki, Sindhi, Tamil, Tulu, Telugu, Maldives 1 . Nepal 107 Borderline: Awadhi, Baluchi, Southern Balochi, Badaga, Pakistan 59 Bhojpuri, Halbi, Chhattisgarhi, Kukna, Konkani, Manipuri, Sri Lanka 4 Naga , , Oriya, Punjabi, South- ern , Pashto, Rajasthani, Santali, Saurashtra, Sylheti, Table 1: Current coverage Kok Borok. We welcome criticisms both of the data (if a language of the subcontinent that you are working with does not appear It is, of course, quite debatable whether on any of the above lists nor in the Appendix this obvi- or Afghanistan should all be all included, and we welcome ously points at an error in the data gathering process) and any cogent argument in this regard, especially as it is un- of the classification. We are particularly interested in cases clear whether the same funding sources that support efforts where languages should evidently be moved from the still in India would be equally available in other countries. That to the vital category, or the other way round. Again, we em- said, we lean toward inclusion, rather than exclusion. phasize that these results are preliminary, and we welcome scholarly debate and discussion. 3. Triage The ISLV plan is to make the final results, and the data it Once the data is collected, we apply the methodology of is based on, publicly accessible, either hosted directly at a Kornai (2013) to decide which varieties can be classed as dedicated website or by means of pointing to or replicating vital, heritage, or still. There are no digitally thriving lan- data already available elsewhere. guages as defined originally, though we acknowledge that We should add here that a policy recommendation based on Hindi may be classified as such. Table 2 summarizes the an assessment of digital death should go beyond a simple breakdown is as follows: exhortation to concentrate all effort regarding this language on heritage conservation. As we already noted in Kornai Status Language (2013), such efforts, while obviously necessary for preserv- vital 36 ing the cultural heritage of humankind, contribute practi- borderline 21 cally nothing to language vitality. To mitigate the heritage 1 cost of digital language death it is therefore suggested that still 576 we expend effort on identifying, if at all feasible, for each digitally still language a vital ‘champion’ of similar vocab- Table 2: Main classes ulary, script, and , with the idea that this champion can become a medium of access to the digital realm that is Only one language, Pali, is listed as heritage, since a con- easier to acquire than English. siderable number of people (over 10,000) are listed as na- In certain cases, such as Andaman Creole (hca), bilingual- tive (L1) speakers of Sanskrit. Be it as it may, the data is ism is so strong that the choice of the champion is obvi- dominated by digitally still languages (over 90% of the lan- ous, but in many cases the task is far from trivial. This guages considered, see the Appendix), and we are left with effort, which should be undertaken primarily by scholars some 50-60 languages that have a chance to take root dig- intimately familiar with the language and the sociolinguis- itally. It should be noted that our classification of vital vs. tic/dialect situation, requires only user-level knowledge of borderline was already highly optimistic (see Kornai 2013 language technology. This is very different from our rec- for a more detailed description of the conservative method- ommendations for vital/borderline languages, to which we ology chosen so as not to raise false alarms), with languages turn now. like Dogri (dgo) listed as digitally vital, which is quite de- batable. 4. Build In the full study, we are likely to retain the positive outlook The build stage no longer considers all dialects and lan- that characterized the earlier work, so as not to hinder the guages of the Indian Subcontinent, just those deemed vi-

25 tal/thriving in Stage 2. These we can hope to endow with both native speakers and already skilled in the design and a full computational toolchain composed of the following application of the word-level tools that most of these can be stages. attempted.

Tool Effort Tool Effort script 0.1 light parser 12 normalization 1 NER 6 language detector 1 OCR 12 word list 2 ASR 12 bilingual dictionary 6 MT 12 morphology 12 spellchecker 6 Table 4: Higher tools

Table 3: Word-level tools For a significant subset (over half) of the thriving/vital lan- guages, in particular Assamese, Bengali, Bodo, English, Table 3 lists the effort (in person-months) associated with Gujarati, Hindi, Kannada, Kashmiri, Konkani, Malay- building the tool or resource in question after the data- alam, Manipuri, Marathi, Nepali, Oriya, Panjabi, San- gathering phase is complete, but assuming significant on- skrit, Tamil, Telugu, and Urdu, there is already a con- line material is found, for if there is no material online the centrated effort under way, see the IndoWordNet site at digital vitality of the language is in grave doubt. It is not as- http://www.cfilt.iitb.ac.in/indowordnet, and that a practi- sumed that the tools so obtained will be of quality compara- cal application with font transcoding and other critical ble to those available for English (or for MT, say English- parallelization technology is available in a 5-way parallel French). Nevertheless, such tools are already useful for a tourism site (Bengali, Hindi, Marathi, Tamil, Telugu), see broad of purposes, and their incremental refinement http://www.tdil-dc.in/sandhan. The build phase can incre- and they higher level tools built on top of them are left to ment these systems for the languages not yet considered. the last Phase of the ISLV project (see Section 5 below). It is expected that orphaned dialects without a near cham- Special funds may be obtained from the NSF Document- pion can only be documented in the sense of heritage ing Endangered Languages program, the Endangered Lan- preservation, while dialects close to champions will partici- guages Project, or other similar preservation efforts, but the pate in (sub)koine formation. The beneficial effects may only direct contribution of ISLV in this regard would be the even extend to some of the languages and dialects outside recommendation that these dialects are indeed in need of the Subcontinent. It will require a great deal of care to such an effort. The main focus of building would initially select the champion dialects, a matter of particular impor- be on the word-level technologies, including spellchecking tance in Hungary, where several Roma dialects, some ob- (standardized orthography), stemming (prefix- and suffix- viously close to main Indian languages, some less visibly removal, but not necessarily deeper morphological analy- so, are spoken. It is not expected that such dialects would sis), glyph analysis, and building a common multilingual be vital in and of themselves, but finding a champion they dictionary of basic vocabulary similar to Acs´ et al (2013). could attach to would significantly enhance their chances Such efforts obviously require better, engineer-level under- of digital survival. standing of language technology, and may serve as a train- ing ground for a new generation of native computational 6. Acknowledgement linguists. Work supported in part by the European Union and the It is the task of this phase to determine to what extent stan- European Social Fund through project FuturICT.hu grant dard (two-tape) finite-state transducer technology is usable #TAMOP-4.2.2.C-11/1/KONV-2012-0013.´ for providing cross-transliteration among the vital varieties on the one hand, and between the vital champions and their 7. References satellites on the other. Only a limited amount of parallel Judit Acs,´ Katalin Pajkossy, and Andras´ Kornai. 2013. (synchronized) grammar writing is envisioned at this stage. Building basic vocabulary across 40 languages. In Pro- ceedings of the Sixth Workshop on Building and Using 5. Apply Comparable Corpora, pages 52–58, Sofia, Bulgaria, - The main applications we envision extend the text and gust. Association for Computational . image-based work to speech (and if resources permit, to Andras´ Kornai. 2013. Digital language death. PloS one, sign languages). Of particular interest are cross-language 8(10):e77056. speech translators which do not assume users to have great Steven Krauwer. 2003. The Basic Language Resource Kit familiarity with standard Hindi, text to speech systems - (BLARK) as the first milestone for the language re- pable of synthesizing speech in any vital language, and per- sources roadmap. In Proceedings of the 2003 Interna- haps captioning systems that would extend the reach of tional Workshop on Speech and Computers (SPECOM broadcast operations. As can be seen from the following 2003), pages 8–15, Moscow, Russia. Table 4, the effort of building these tools is considerably Paul Lewis, Gary Simons, and Charles Fennig, editors. larger, and it is only with computational linguists who are 2013. The Ethnologue. Summer Institute of Linguistics.

26 Attila Zseder,´ Gabor´ Recski, Daniel´ Varga, and Andras´ Ko- rukh, Kusunda, Kutang Ghale, Ladakhi, , Lahul rnai. 2012. Rapid creation of large-scale corpora and , Lakha, , Lambichhong, Lamkang, Lasi, frequency dictionaries. In Proceedings to LREC 2012, Layakha, Lepcha, Lhokpu, Lhomi, , Limbu, pages 1462–1465. Lingkhim, Lish, Loarki, Lodhi, Lohorung, Loke, , Lui, Lumba-Yakkha, Lunanakha, Lyngngam, Mag- Appendix: digitally still languages ahi, Mahali, Mahasu Pahari, Majhi, Majhwar, Mal Paharia, A’tong, A-Pucikwar, Adap, Adi, Adiwasi Garasia, Aer, Mala Malasar, Malankuravan, Malapandaram, Malaryan, Afghan , Agariya, Ahirani, Ahom, Aimaq, Malavedan, Malvi, Manangba, Manda, , Manna- Aimol, Aiton, Aka-Bea, Aka-Bo, Aka-Cari, Aka-Jeru, Dora, Mannan, , Chin, , Aka-Kede, Aka-Kol, Aka-, Akar-Bale, Allar, Alu Maria, , Marma, Marwari, Marwari, Mar- Kurumba, Amri Karbi, Anal, Andaman Creole Hindi, wari, Mawchi, Megam, Memoni, Merwari, Mewari, Andh, , Ange, Apatani, Aranadan, Ashkun, Mewati, Miju-Mishmi, Mina, Mirgan, Mirpur Panjabi, Asuri, Athpariya, Attapady Kurumba, Badeshi, Bagheli, Mising, Mixed Great Andamanese, Mogholi, Monsang Bagri, , , Bantawa, Baraamu, Bateri, Bau- Naga, , Mru, Mudu Koraga, Muduga, Mu- ria, Bawm Chin, , Belhariya, Bellari, Betta Ku- gom, Mukha-Dora, Mullu Kurumba, Munda, Mundari, rumba, Bhadrawahi, Bhalay, Bharia, Bhatola, Bhatri, Munji, Musasa, Muthuvan, Mzieme Naga, Na, Naaba, Bhattiyali, Bhaya, Bhilali, Bhili, Bhoti Kinnauri, Bhu- Nachering, Nagarchal, Nahali, Nahari, Nar Phu, , jel, , Biete, Bijori, , Birhor, Bodo Gad- Nepalese Sign Language, Nepali Kurux, Nepali), Nihali, aba, Bodo Parja, Bodo, Bondo, Bote-Majhi, , Brokkat, Nimadi, Nocte Naga, Noiri, Norra, Northeast Pashayi, Brokpake, , Bugun, Buksa, Bumthangkha, Bun- Northern Ghale, Northern Gondi, Northern , North- deli, , Byangsi, Camling, Car Nicobarese, ern Pashto, Northern , Northwest Pashayi, Central Nicobarese, Central Pashto, Chalikha, Chamari, Northwestern Kolami, Northwestern Tamang, Nubri, Nup- , , Changthang, Chantyal, Chau- bikha, Nyenkha, Nyishi, Od, Oko-Juwoi, Olekha, Oraon dangsi, Chaura, Chenchu, Chepang, Chhintange, Chhu- Sadri, , Pahari-Potwari, Pahlavani, Paite Chin, Pak- lung, Chilisso, Chinali, Chiru, Chitkuli Kinnauri, Chittag- istan Sign Language, Paliyan, Palpa, Palya Bareli, Panch- onian, Chitwania Tharu, Chocangacakha, Chodri, Chokri pargania, , Paniya, Pankhu, Pao, Parachi, Pard- Naga, , Chug, Chukwa, , Dakpakha, han, Pardhi, Parenga, Parkari Koli, Parsi, Pathiya, Pattani, Dameli, Dandami Maria, Dangaura Tharu, Darai, Dar- Bareli, Pengo, Phake, Phalura, Phangduwali, Phom long, Darmiya, Deccan, Degaru, Dehwari, Deori, De- Naga, Phudagi, Pnar, Pochuri Naga, Porja, Poumei Naga, siya, Dhanki, Dhanwar, Dhatki, , Dhodia, Dhun- Powari, Prasuni, Puimei Naga, Puma, Purik, Puroik, Pu- dari, Digaro-Mishmi, Dimasa, Dolpo, Domaaki, Dotyali, rum Naga, Purum, Rabha, Rajbanshi, Raji, Garasia, Dubli, Dumi, Dungmali, Dungra Bhil, Dura, Duruwa, Dza- Ralte, Rana Tharu, Rangkas, , Rathawi, Rathwi lakha, Eastern Balochi, Eastern Gorkha Tamang, East- Bareli, Raute, Ravula, Rawat, Reli, Riang, , ern Gurung, Eastern Magar, Eastern Meohang, East- Rongpo, Ruga, Saam, Sadri, Sajalong, Sakachep, Sam- ern Muria, Eastern Parbate Kham, Eastern Tamang, Er- balpuri, Sampang, Samvedi, Sanglechi, , avallan, Far Western Muria, Gadaba, Gadaba, Gaddi, Sansi, Sartang, Sauria Paharia, Savara, Savi, Seke, Sen- Gade Lohar, Gahri, Galo, Gamale Kham, Gamit, Gangte, tinel, Shekhawati, Shendu, Sherdukpen, Sherpa, Sheshi Garhwali, Garo, Gata’, Gawar-Bati, Ghandruk Sign Lan- Kham, Shina, Sholaga, Shom Peng, Shumashti, Shumcho, guage, Ghera, Goaria, Godwari, Gondi, Gongduk, Gowlan, Sikkimese, Simte, Sindhi Bhil, Singpho, Sirmauri, Sonha, Gowli, Gowro, Grangali, Gurgula, Hajong, Harijan Kin- Sora, Southeast Pashayi, Southeastern Kolami, Southern nauri, Haroti, Haryanvi, Hazaragi, Helambu Sherpa, Hin- Ghale, Southern Gondi, Southern Hindko, Southern Nico- duri, Hmar, Ho, Holiya, Hrangkhol, Hruso, Humla, Idu- barese, Southern Rengma Naga, Southern Uzbek, South- Mishmi, Indian Sign Language, Indo-Portuguese, Indus ern Yamphu, Southwest Pashayi, Southwestern Tamang, Kohistani, , Irula, Ishkashimi, Jad, Jadgali, Jan- Spiti Bhoti, Sri Lankan Creole Malay, Sri Lankan Sign davra, Jangshung, Jarawa, Jatapu, Jaunsari, Jennu Ku- Language, Stod Bhoti, Sumi Naga, Sunam, Sunwar, Sur- rumba, Jerung, Jhankot Sign Language, Jirel, Juang, Jumla gujia, Surjapuri, Tagin, Tangchangya, , Sign Language, Jumli, Juray, Kabutra, Kachari, Kachi Koli, Tarao Naga, Tawang Monpa, Teressa, Thachanadan, Thado Kadar, Kagate, Kaikadi, Kaike, Kalaktang Monpa, Kalami, Chin, Thakali, , Thangmi, Thudam, Thulung, Kalanadi, Kalasha, Kalkoti, Kamar, Kamviri, Kanashi, Tichurong, Tilung, Tinani, Tippera, Tirahi, Tiwa, Toda, Kanauji, Kangri, Kanikkaran, Kanjari, Kannada Kurumba, Torwali, Toto, Tregami, Tshangla, Tsum, Tukpa, Turi, Tu- Karbi, Kathoriya Tharu, Kati, Katkari, Kayort, Khaling, rung, , Ullatan, Urali, Ushojo, Usui, Vaagri Khamba, Khamyang, Khandesi, Kharam Naga, Kharia Booli, Vaghri, Vaiphei, Varhadi-Nagpuri, Varli, Vasavi, Thar, Kharia, Khengkha, Khetrani, Khezha Naga, Khi- Veddah, Vishavan, Waddar, Wadiyara Koli, , Wai- amniungan Naga, Khirwar, Khoibu Naga, Kinnauri, Koch, gali, Wakhi, Waling, Walungge, Wambule, , Kochila Tharu, Koda, Kodaku, Kodava, Kohistani Shina, Waneci, War-Jaintia, Warduji, Wayanad Chetti, Wayu, Koi, Koireng, Kol, Kom, , Korku, Korlai Cre- Western Balochi, Western Gurung, Western Magar, West- ole Portuguese, Koro, Korra Koraga, Korwa, Kota, Koya, ern Meohang, Western Muria, Western Parbate Kham, Kudiya, Kudmali, Kui, Pahari, Kulung, Kumaoni, Western Tamang, Wotapuri-Katarqalai, Yakha, Yamphu, Kumarbhag Paharia, Kumbaran, Kumhali, , Yidgha, Zangskari, . Kunduvadi, Kupia, Kurichiya, Kurmukar, Kurtokha, Ku-

27