Lexicon-Driven Approach to the Recognition of Arabic Named Entities Jack Halpern (春遍雀來) The CJK Dictionary Institute (日中韓辭典研究所) 34-14, 2-chome, Tohoku, Niiza-shi, Saitama 352-0001, Japan [email protected] Abstract Various factors contribute to the difficulties in processing Arabic personal names and named entities, posing special challenges to developers of NLP applications in the areas of named entity recognition (NER), machine translation (MT), morphological analysis (MA) and information retrieval (IR). These include the complexity of the Arabic orthography, the high level of orthographical and morphological ambiguity, the multitude of highly irregular romanization systems, and the vast number of romanized variants that are difficult to detect and disambiguate. This paper focuses on the orthographic variation of Arabic personal names, with special emphasis on the ambiguity resulting from transcribing names into the Roman script (romanization). It describes the techniques used to compile the Database of Arabic Names (DAN), a large-scale lexical resource containing millions of Arabic names and their variants in both romanized and fully vocalized Arabic, and argues that linguistic knowledge in the form of a rule-driven lexicon can enhance the accuracy of statistical methods to achieve high accuracy in the recognition of Arabic names. Moohammad, Moohamad, Mohammad, Mohamad, 1 Concepts and Definitions etc. Much confusion surrounds the terms transliteration and transcription, with the former often misleadingly used in Transliteration, enclosed in back slashes, is a the sense of the latter even in academic papers representation of the script of a source language by using (AbdulJaleel and Larkey, 2003). To discuss these the characters of another script. Ideally, it unambiguously concepts in an unambiguous manner it is necessary to represents the graphemes, rather than the phonemes, of understand these and related terms correctly. the source language. For example, is transliterated as \mHmd\, in which each Arabic letter is unambiguously Romanization is the representation of a language written represented by one Roman letter, enabling round-trip in a non-Roman script using the Roman alphabet. This conversion. A transliteration system widely used in NLP includes both transliteration and various kinds of applications is the excellent Buckwalter system (Beesley, transcription. There are various official systems for 2003). Many academic papers misleadingly use the term romanizing Arabic. transliteration when what they actually mean is transcription (Halpern, 2007). Transcription is a representation of the source script of a language in the target script in a manner that reflects the To summarize, the name is transliterated as pronunciation of the original, often ignoring graphemic \mHm~d\, phonetically transcribed as [muħɛmm̈ ɛd̈ ], correspondence. This includes the following phonemically transcribed as /muHammad/, and romanized subcategories: in various popular transcriptions such as Mohammed, Muhammad and Mohamad, among many others. 1. A phonetic transcription, enclosed in square brackets, represents the actual speech sounds, including In this paper, the adjectives Arab and Arabic, especially in allophones. The best known systems are IPA and the context of Arab name and Arabic name, are not SAMPA. For example, is transcribed as interchangeable. Arab refers to names of Arabs (as [muħɛmm̈ ɛd̈ ]. opposed to names of non-Arabs) regardless of whether 2. A phonemic transcription, enclosed in slashes, they are written in the Arabic or the Roman script, represents the phonemes of the source language whereas Arabic is more general and refers to Arab names (ignoring allophones), ideally on a one-to-one basis. written in the Roman script or any name (Arab and non- For example, is transcribed as /muHammad/, in Arab) written in the Arabic script. which a represents the phoneme /a/, rather than the phone [ɛ].̈ Some well known systems include ICS 2 Why the Arabic script is ambiguous (Intelligence Community Standard) and ALC-LA The Arabic script is highly ambiguous. A distinguishing (American Library Association-Library of Congress feature of written Arabic is that words are represented as a romanization standard) (Library and Archives string of consonants with little or no indication of vowels, Canada, 2006). referred to as unvocalized Arabic. For example, the string 3. A popular transcription, indicated by italics, is a can theoretically represent 40 consonant-vowel conventionalized orthography that roughly represents permutations, such as mawa, mawwa, mawi, mawwi, etc. pronunciation. For example, is transcribed in Even fully vocalized Arabic is not phonemic because of over 100 ways, such as Mohammed, Mohamed, the many one-to-many grapheme-to-phoneme ambiguities (Halpern, 2007 and 2008). The principal factors 193 contributing to orthographical ambiguity are summarized and is romanized in some unusual ways as shown in the below. table below. 1. The omission of short vowels: e.g., the unvocalized Roman Example Frequency can represent the following seven wordforms: q Qaddafi 00077900 ك ات ب g Gaddafi 00219000 كَاتِبٌ ,/kaatibin/ كَاتِبٍ ,/kaataba/ كَاتَبَ ,/kaatib/ كَاتِب gh Ghaddafi 00013100 كَاتِبُ kaatibi/ and/ كَاتِبِ ,/kaatiba/ كَاتِبَ ,/kaatibun/ /kaatibu/. k Kaddafi 00068300 kh Khaddafi 00007380 ,آ ,ى ,ا ,.Long /aa/ can be represented various ways, i.e .2 c Caddafi 00000034 ,ال و سطي and آ س يا , سوري ا or zero, as illustrated by or even j Jaddafi 00000031 ,ة ,ه ,while short /a/ can be represented by zero especially in loanwords), as illustrated in the) ا ,اﻹ س ك ندري و :various Arabic variants for Alexandria romanizations ق Table 1: Examples of .اﻹ س ك ندري ا and ,اﻹ س ك ندري ت 3. Consonant gemination is ambiguous since shadda is The official romanization systems, which are used rather محمد normally omitted, e.g., the unvocalized infrequently, are just a fraction of the plethora of .(مُحَمَد muHammad/ has no shadda (vocalized as/ 4. The omission of nunation diacritics for case endings, romanizations which for lack of a better name we shall ."the fatHatayn lump together under the label "popular transcriptions ,(شُكْرًا shukran/ (vocalized/ ش كرا e.g., in is not written. These transcriptions are often inconsistent, irregular and alif maqSuura), unpredictable, sometimes leading to hundreds or more') ى Alternation between ً (yaa') and .5 especially in Egypt, causes confusion. For example, than a thousand variants for a single name. For example, -the names /muHammad/ and /al .اب و ظ بي Abu Dhabi' is often written as' ابو ظبي taa' marbuuTa), qadhaafii/ are touted as having more than 100 romanized) ة haa') and) ه Alternation between .6 uuda/ being also spelt as variants. As impressive as this number may seem, it pales`/ عودة resulting in names like /in comparison to names like /`abdurrahiim .عوده 7. The rules for determining the hamza seat are of and /`abdurrazzaaq/, which have over 1100 notorious complexity. variants. Those cases are extreme, but names with several 8. Phonological alternation processes such as dozen variants are very common. For example, the assimilation modify the phonetic realization, e.g. popular name /maHmuud/ has 69 variants, the is realized as /'arrajulu-TTawiilu/. most frequent of which are shown below along with their الرجل الطويل 9. Compound names are often written either solid or frequency of occurrence on the web. ع بدال رح يم open, e.g., /`abd-urrahiim/ is written either This is true of various other common Variant Frequency .عبد الرحيم or bin/, Mahmoud 0020400000/ ب ن abuu/ and'/ اب و name elements like 10. Long vowels are often neutralized, a fact which is Mahmud 0005770000 mostly ignored in the literature. Even such common Mahmood 0004050000 haadha/ 'this' are often Mahmut 0003780000/ هٰذَا ana/ 'I' and'/ أَنَا words as incorrectly transcribed as 'anaa and haadhaa. Mehmood 0000685000 Mahmod 0000138000 uuda/ Machmut 0000006640`/ عودة These ambiguities result in names such as and variation due to common Machmud 0000108000 ,عوده also being spelt as abuu/ being Mehmud 0000082400'/ اب و abd/ and`/ ع بد name elements such as detached from or attached to the rest of the name. A Mahmoed 0000052100 Mechmod 0000048000 عبد العزيز combination of these factors lead to a name like /`abd-'al-`aziiz/ having eight variants in Arabic, including Makhmud 0000042700 As a result, it is .ع بدال عزي ز and ,ع بدإل عزي ز ,عبدلعزيز difficult to identify such strings as variants of the same Table 2: Common romanized variants of underlying name, leading to lower recall in named entity extraction and recognition. The various ambiguity factors described above result in an 3 Why romanized Arabic is ambiguous immense number of romanized variants. Arabic also has a high level of romanization ambiguity. 4 Lexicon-Driven Approach That is, an Arabic name can be romanized in a multitude of ways. One reason for this is that many official and An important motivation for using statistical methods in unofficial romanization systems are used to transcribe NLP applications has been the poor availability and the Arabic sounds in a bewildering number of ways. For high cost of constructing large-scale lexical databases, but shuuluukh/ is transcribed in several statistical methods by themselves are inadequate for/ شول وخ ,example official systems as follows: shwlwkh in the ALA-LC dealing with the ambiguities of the Arabic script, system, Shulukh in ICS (Intelligence Community especially because of the difficulties in algorithmically Standard), šūlūḫ in DIN, Shūlūkh