Urdu – Roman Transliteration via Finite State Transducers Tina Bogel¨ University of Konstanz Konstanz, Germany [email protected] Abstract The TURF system presented in this paper is con- cerned with the Urdu–Roman transliteration. It This paper introduces a two-way Urdu– deals with the Urdu-specific orthographic issues by Roman transliterator based solely on a non- integrating certain restrictional components into the probabilistic finite state transducer that solves finite state transducer to cut down on overgener- the encountered scriptural issues via a partic- ation, while at the same time employing an ar- ular architectural design in combination with a set of restrictions. In order to deal with the chitectural design that allows for a large degree enormous amount of overgenerations caused of flexibility. The transliterator is based solely by inherent properties of the Urdu script, the on a non-probabilistic finite state transducer im- transliterator depends on a set of phonologi- plemented with the Xerox finite state technology cal and orthographic restrictions and a word (XFST) (Beesley and Karttunen, 2003), a robust and list; additionally, a default component is im- easy-to-use finite state tool. plemented to allow for unknown entities to be This paper is organized as follows: In section 2, transliterated, thus ensuring a large degree of flexibility in addition to robustness. one of the (many) orthographic issues of Urdu is in- troduced. Section 3 contains a short review of ear- lier approaches. Section 4 gives a brief introduction 1 Introduction into the transducer and the set of restrictions used to cut down on overgeneration. Following this is an This paper introduces a way of ransliterating rdu t U account of the architectural design of the translit- and oman via a non-probabilistic inite state trans- R f eration process (section 5). The last two sections ducer ( ), thus allowing for easier machine TURF provide a first evaluation of the TURF system and a processing.1 The TURF transliterator was originally final conclusion. designed for a grammar of Hindi/Urdu (Bogel¨ et al., 2009), based on the grammar development platform 2 Urdu script issues XLE (Crouch et al., 2011). This grammar is writ- ten in Roman script to serve as a bridge/pivot lan- Urdu is an Indo-Aryan language spoken mainly in guage between the different scripts used by Urdu Pakistan and India. It is written in a version of the and Hindi. It is in principle able to parse input from Persian alphabet and includes a substantial amount both Hindi and Urdu and can generate output for of Persian and Arabic vocabulary. The direction of both of these language varieties. In order to achieve the script is from right to left and the shapes of most this goal, transliterators converting the scripts of characters are context sensitive; i.e., depending on Urdu and Hindi, respectively, into the common Ro- the position within the word, a character assumes a man representation are of great importance. certain form. 1 Urdu has a set of diacritical marks which ap- I would like to thank Tafseer Ahmed and Miriam Butt pear above or below a character defining a partic- for their help with the content of this paper. This research was part of the Urdu ParGram project funded by the Deutsche ular vowel, its absence or compound forms. In total, Forschungsgemeinschaft. there are 15 of these diacritics (Malik, 2006, 13); 25 Proceedings of the 10th International Workshop on Finite State Methods and Natural Language Processing, pages 25–29, Donostia–San Sebastian,´ July 23–25, 2012. c 2012 Association for Computational Linguistics the four most frequent ones are shown in Table 1 in the issues of the scripts of Urdu and Hindi. combination with the letter H ‘b’. All of these approaches are highly dependent on . word lists due to the properties of the Urdu script and H. + diacritic Name Roman transliteration the problems arising with the use of diacritics. Most Zabar ba systems dealing with undiacriticized input are faced H. with low accuracy rates: The original system of Ma- Zer bi H . lik (2006), e.g., drops from approximately 80% to H . Pesh bu 50% accuracy (cf. Malik et al. (2009, 178)) – others H Tashdid bb have higher accuracy rates at the cost of being uni- . directional. Table 1: The four most frequently used diacritics While Malik et al. (2009) have claimed that the non-probabilistic finite state model is not able to When transliterating from the Urdu script to another handle the orthographic issues of Urdu in a satisfy- script, these diacritics present a huge problem be- ing way, this paper shows that there are possibilities cause in standard Urdu texts, the diacritics are rarely for allowing a high accuracy of interpretation, even used. Thus, for example, we generally are only con- if the input text does not include diacritics. fronted with the letter H. ‘b’ and have to guess at the pronunciation that was intended. Take, e.g., the 4 The TURF Transliterator following example, where the word AJ» kuttA ‘dog’ is to be transliterated. Without diacritics, the word The TURF transliterator has been implemented as consists of three letters: k, t and A. If in the case of a non-probabilistic finite state transducer compiled transliteration, the system takes a guess at possible with the lexc language (Lexicon Compiler), which is short vowels and geminated consonants, the output explicitly designed to build finite state networks and contains multiple possibilities ((1)). analyzers (Beesley and Karttunen, 2003, 203). The resulting network is completely compatible with one (1) that was written with, e.g., regular expressions, but has the advantage in that it is easily readable. The transliteration scheme used here was developed by Malik et al. (2010), following Glassman (1986). As has been shown in section 1, Urdu transliter- ation with simple character-to-character mapping is In addition to the correct transliteration kuttA, the not sufficient. A default integration of short vowels transliterator proposes five other possibilities for the and geminated consonants will, on the other hand, missing diacritics. These examples show that this cause significant overgeneration. In order to reduce property of the Urdu script makes it extremely dif- this overgeneration and to keep the transliterator as ficult for any transliterator to correctly transliterate efficient as possible, the current approach integrates undiacriticized input without the help of word lists. several layers of restrictions. 3 Earlier approaches 4.1 The word list When dealing with Urdu transliteration it is not pos- Earlier approaches to Urdu transliteration almost sible to not work with a word list in order to ex- always have been concerned with the process of clude a large proportion of the overgenerated out- transliterating Urdu to Hindi or Hindi to Urdu (see, put. In contrast to other approaches, which depend e.g., Lehal and Saini (2010) (Hindi Urdu), Ma- → on Hindi or Urdu wordlists, TURF works with a Ro- lik et al. (2009) (Urdu Hindi), Malik et al. → man wordlist. This wordlist is derived from an XFST (2010) (Urdu Roman) or Ahmed (2009) (Roman → finite state morphology (Bogel¨ et al., 2007) indepen- Urdu). An exception is Malik (2006), who ex- → dently created as part of the Urdu ParGram devel- plored the general idea of using finite state transduc- opment effort for the Roman intermediate language ers and an intermediate/pivot language to deal with (Bogel¨ et al., 2009). 26 4.2 Regular expression filters simply matched against themselves (e.g. k:k, r:r). On top of this network, the regular expression filters The regular expression filters are based on knowl- provide further restrictions for the output form. edge about the phonotactics of the language and are a powerful tool for reducing the number of possi- bilities proposed by the transliterator. As a concrete example, consider the filter in (2). (2) [ [ y A [a i u] ]] ∼ | | In Urdu a combination of [ y A short vowel ] is not allowed ( ). A filter like in (2) can thus be used to ∼ disallow any generations that match this sequence. 4.3 Flag diacritics The XFST software also provides the user with a method to store ‘memory’ within a finite state net- work (cf. Beesley and Karttunen (2003, 339)). These so-called flag diacritics enable the user to en- force desired constraints within a network, keeping the transducers relatively small and simple by re- moving illegal paths and thus reducing the number of possible analyses. 5 The overall TURF architecture However, the finite state transducer should also be able to deal with unknown items; thus, the con- straints on transliteration should not be too restric- tive, but should allow for a default transliteration as well. Word lists in general have the drawback that a matching of a finite state transducer output against a word list will delete any entities not on the word list. Figure 1: Transliteration of and This means that a methodology needs to be found I» H. AJ» to deal with unknown but legitimate words with- The Urdu script default 1-1 mappings are marked out involving any further (non-finite state) software. with a special identification tag ([+Uscript]) for Figure 1 shows the general architecture to achieve later processing purposes. Thus, an Urdu script this goal. For illustrative purposes two words are word will not only be transliterated into the corre- transliterated: kitAb ‘book’ and , which H. AJ» I » sponding Roman script, but will also be ‘transliter- transliterates to an unknown word kt, potentially ated’ into itself plus an identificational tag. having the surface forms kut, kat or kit. The output of the basic transliterator shows part of the vast overgeneration caused by the underspec- 5.1 Step 1: Transliteration Part 1 ified nature of the script, even though the restricting The finite state transducer itself consists of a net- filters and flags are compiled into this component.