Building a Corpus for Palestinian Arabic: a Preliminary Study Mustafa Jarrar, *Nizar Habash, Diyam Akra, Nasser Zalmout Birzeit University, West Bank, Palestine {mjarrar,nzalmout}@birzeit.edu, [email protected] *New York University Abu Dhabi, United Arab Emirates [email protected] the Arab World. Using such tools to understand Abstract and process Arabic dialects (DAs) is a challenging task because of the phonological and This paper presents preliminary results in morphological differences between DAs and building an annotated corpus of the MSA. In addition, there is no standard Palestinian Arabic dialect. The corpus orthography for DAs. Moreover, DAs have consists of about 43K words, stemming limited standardized written resources, since from diverse resources. The paper most of the written dialectal content is the result discusses some linguistic facts about the of ad hoc and unstructured social conversations Palestinian dialect, compared with the or commentary, in comparison to MSA’s vast Modern Standard Arabic, especially in body of literary works. terms of morphological, orthographic, and lexical variations, and suggests some The rest of this paper is structured as follows: directions to resolve the challenges these We present important linguistic background in differences pose to the annotation goal. Section 2, followed by a survey of related work Furthermore, we present two pilot in Section 3. We then present the process of studies that investigate whether existing collecting the Curras Corpus (Section 4) and the tools for processing Modern Standard challenges of annotating it (Section 5). Arabic and Egyptian Arabic can be used to speed up the annotation process of our 2. Linguistic Background Palestinian Arabic corpus. In this section we summarize some important linguistic facts about PAL that influence the 1. Introduction and Motivation decisions we made in this project. For more This paper presents preliminary results towards information on PAL and Levantine Arabic in building a high-coverage well-annotated corpus general, see (Rice and Sa’id, 1960; Cowell, of the Palestinian Arabic dialect (henceforth 1964; Bateson, 1967; Brustad, 2000; Halloun, PAL), which is part of an ongoing project called 2000; Holes, 2004; Elihai, 2004). For a Curras. Building such a PAL corpus is a first discussion of differences between Levantine and important step towards developing natural Egyptian Arabic (EGY), see Omar (1976). language processing (NLP) applications, for 2.1 Arabic and its dialects searching, retrieving, machine-translating, spell- checking PAL text, etc. The importance of The Arabic language is a collection of variants processing and understanding such text is among which a standard variety (MSA) has a increasing due to the exponential growth of special status, while the rest are considered socially generated dialectal content at recent colloquial dialects (Bateson, 1967, Holes, 2004; Social Media and Web 2.0 breakthroughs. Habash, 2010). MSA is the official written language of government, media and education in Most Arabic NLP tools and resources were the Arab World, but it is not anyone’s native developed to serve Modern Standard Arabic language; the spoken dialects vary widely across (MSA), which is the official written language in the Arab World and are the true native varieties 18 Proceedings of the EMNLP 2014 Workshop on Arabic Natural Langauge Processing (ANLP), pages 18–27, October 25, 2014, Doha, Qatar. c 2014 Association for Computational Linguistics of Arabic, yet they have no standard orthography dialects. The Druze dialect retains the /q/ and are not taught in schools (Habash et al., pronunciation. Another example is the /k/ k), which ك Zribi et al., 2014). phoneme (corresponding to MSA ,2012 realizes as /tš/ in rural dialects. These difference qlb ‘heart’ to be ﻗﻠﺐ PAL is the dialect spoken by Arabic speakers cause the word for who live in or originate from the area of pronounced as /qalb/, /’alb/, /kalb/ and /galb/ and ﻛﻠﺐ Historical Palestine. PAL is part of the South to be ambiguous out of context with the word Levantine Arabic dialect subgroup (of which klb ‘dog’ /kalb/ and /tšalb/. And similarly to Jordanian Arabic is another dialect). PAL is EGY (but unlike Tunisian Arabic), the MSA θ) becomes /s/ or /t/, and the ث) /historically the result of interaction between phoneme /θ ð) becomes /z/ or /d/ in ذ) /Syriac and Arabic and has been influenced by MSA phoneme /ð kðb ﻛﺬب many other regional language such as Turkish, different lexical contexts, e.g., MSA Persian, English and most recently Hebrew. The /kaðib/ ‘lying’ is pronounced /kizib/ in PAL and Palestinian refugee problem has led to additional /kidb/ in EGY. mixing among different PAL sub-dialects as well as borrowing from other Arabic dialects. We Similar to many other dialects, e.g. EGY and discuss next some of the important Tunisian (Habash et al., 2012; Zribi et al., 2014), distinguishing features of PAL in comparison to the glottal stop phoneme that appears in many MSA as well as other Arabic dialects. We MSA words has disappeared in PAL: compare /bŷr /bi’r ﺑﺌﺮ rÂs /ra’s/ ‘head’ and رأس consider the following dimensions: phonology, MSA morphology, and lexicon. Like other Arabic ‘well’ with their Palestinian urban versions: /rās/ dialects, PAL has no standard orthography. and /bīr/. Also, the MSA diphthongs /ay/ and /aw/ generally become /ē/ and /ō/; this 2.2 Phonology transformation happens in EGY but not in other PAL consists of several sub-dialects that Levantine dialects such as Lebanese, e.g., MSA ./byt /bayt/ ‘house’ becomes PAL /bēt ﺑﯿﺖ generally vary in terms of phonology and lexicon preferences. Commonly identified sub- dialects include urban (which itself varies mostly PAL also elides many short vowels that appear phonologically among the major cities such as in the MSA cognates leading to heavier syllabic jibāl/ ‘mountains’ (and/ ﺟﺒﺎل Jerusalem, Jaffa, Gaza, Nazareth, Nablus and structure, e.g. MSA Hebron), rural, and Bedouin. The Druze EGY /gibāl/) becomes PAL /jbāl/. Additionally community has also some distinctive long vowels in unstressed positions in some PAL phonological features that set it apart. The sub-dialects shorten, a phenomenon shared with زاروا) /variations are a miniature version of the EGY but not MSA: e.g., compare /zāru زاروه) /variations in Levantine Arabic in general. zAr+uwA) ‘they visited’ with /zarū Perhaps the most salient variation is the zAr+uw+h) ‘they visited him’. Finally, PAL has pronunciation of the /q/ phoneme (corresponding commonly inserted epenthetic vowels q1), which realizes as /’/ in most urban (Herzallah, 1990), which are optional in some ق to MSA dialects, /k/ in rural dialects, and /g/ in Bedouin cases leading to multiple pronunciations of the klb ﻛﻠﺐ) /same word, e.g., /kalb/ and /kalib 1Arabic orthographic transliterations are provided in the ‘dog’). This multiplicity is not shared with MSA, Habash-Soudi-Buckwalter (HSB) scheme (Habash et al., which has a simpler syllabic structure and more 2007), except where indicated. HSB extends Buckwalter’s limited epenthesis than PAL. transliteration scheme (Buckwalter, 2004) to increase its readability while maintaining the 1-to-1 correspondence 2.3 Morphology with Arabic orthography as represented in standard encodings of Arabic, i.e., Unicode, etc. The following are PAL, like MSA and its dialects and other the only differences from Buckwalter’s scheme (indicated Semitic languages, makes extensive use of templatic morphology in addition to a large set ة ħ ,({) ئ ŷ ,(>) إ Ǎ ,(&) ؤ ŵ ,(<) أ Â ,(|) آ in parentheses): Ā of affixations and clitics. There are however ى g), ý) غ E), γ) ع Z), ς) ظ Ď ,($) ش š ,(*) ذ v), ð) ث p), θ) K). Orthographic transliterations are) ـ ٍ N), ĩ) ـ ٌ F), ũ) ـ ً Y), ã) presented in italics. For phonological transcriptions, we some important differences between MSA and follow the common practice of using ‘/.../’ to represent PAL in terms of morphology. First, like many phonological sequences and we use HSB choices with some other dialects, PAL lost nominal case and verbal extensions instead of the International Phonetic Alphabet mood, which remain in MSA. Additionally, PAL (IPA) to minimize the number of representations used, as was done by Habash (2010). in most of its sub-dialects collapses the feminine and masculine plurals and duals in verbs and 19 most nouns. Some specific inflections are 3. Related Work Hbyt ﺣﺒﯿﺖ ,.ambiguous in PAL but not MSA, e.g /Habbēt/ ‘I (or you [m.s.]) loved’. 3.1 Corpus Collection and Annotation There have been many contributions aiming to Second, some specific morphemes are slightly or develop annotated Arabic language corpora, with quite different in PAL from their MSA forms, the main objective of facilitating Arabic NLP e.g., the future marker is /sa/ in MSA but /Ha/ or applications. Notable contributions targeting /raH/ in PAL. Another prominent example is the MSA include the work of Maamouri and Cieri, feminine singular suffix morpheme (Ta (2002), Maamouri et al. (2004), Smrž and Hajič Marbuta), which in MSA is pronounced as /at/ (2006), and Habash and Roth (2009). These except at utterance final positions (where it is efforts developed annotation guidelines for /a/). In some PAL urban sub dialects, it has written MSA content producing large-scale multiple allomorphs that are phonologically and Arabic Treebanks. syntactically conditioned: /a/ (after non-front and emphatic consonants), /e/ (after front non- Contributions that are specific to DA include the emphatic consonants), /it/ (nouns in construct development of a pilot Levantine Arabic state such as before possessive pronouns) and /ā/ Treebank (LATB) of Jordanian Arabic, which bTħ contained morphological and syntactic ﺑﻄﺔ .in deverbals before direct objects): e.g) annotations of about 26,000 words (Maamouri et ﺑﻄﺘﻨﺎ ,’Hbħ /Habb+e/ ‘pill ﺣﺒﺔ ,’baTT+a/ ‘duck/ bTnA /baTT+it+na/ ‘our duck’ and /mdars+ā al., 2006). To speed up the process of creating +hum/ ‘she taught them’.