DIWAN: A Dialectal Word Annotation Tool for Arabic Faisal Al-Shargi Owen Rambow Universität Leipzig CCLS, Columbia University Leipzig New York, NY Germany USA [email protected] [email protected] leipzig.de Abstract prehensible to an Arabic speaker from the Levant or the Arabian Peninsula.1 This paper presents DIWAN, an anno- tation interface for Arabic dialectal Within these broad regions, further and consider- texts. While the Arabic dialects differ able geographic distinctions exist – within coun- in many respects from each other and tries, across country borders, and between cities from Modern Standard Arabic, they al- and villages. Some examples include Gulf Ara- so have much in common. To facilitate bic, Bahraini Arabic, Najdi Arabic, Hijazi Ara- annotation and to make it as efficient as bic, Yemeni Arabic, Yemeni Hadhrami Arabic, possible, it is therefore not advisable to Yemeni Sanaani Arabic, Yemeni Ta'izzi-Adeni treat each Arabic dialect as a separate Arabic, Dhofari Arabic, Omani Arabic, Shihhi language, unrelated to the other vari- Arabic, and the Peninsular Arabic dialects. ants of Arabic. Instead, we make anal- yses from other variants available to Despite this diversity, all Arabic dialects share the annotator, who then can choose to certain properties: much of their phonology, use them or not. templatic morphology augmented by affixes and a large set of clitics, large parts of their syntax, 1. Introduction and important (though unpredictable) parts of the lexicon. Arabic is a Central Semitic language, closely related to Aramaic, Hebrew, Ugaritic and Phoe- Current natural language processing (NLP) tools nician. It is spoken by 420 million speakers (na- work well with MSA because they were de- tive and non-native) in the Arab World. Arabic signed specifically for the processing of MSA, also is a liturgical language of 1.6 billion Mus- and because of the abundance of MSA resources. lims around the world. Applying the NLP tools designed for MSA di- rectly to DA yields significantly lower perfor- Modern Standard Arabic (MSA) is the official mance (Chiang et al., 2006; Habash and Ram- Arabic language. It is the educational language bow, 2006; Benajiba et al., 2010; Habash et al., and official language used in news and official 2012). This makes it imperative to direct re- communication across the Arabic-speaking search to building resources and tools for DA world. When Arabs communicate spontaneously processing. in informal settings, they use dialectal Arabic (DA). There are divisions of many dialects of the 1 When Arabic speakers of different dialects meet, they tend Arabic language that occur between the spoken to navigate towards a middle Arabic that encapsulates the languages of different regions. Some varieties of shared aspects they are aware of in order to maximize communication. A better and harder test of comprehension Arabic in North Africa, for example, are incom- is to eavesdrop on a conversation in another dialect. 49 Proceedings of the Second Workshop on Arabic Natural Language Processing, pages 49–58, Beijing, China, July 26-31, 2015. c 2014 Association for Computational Linguistics Figure 1: An example of MSA and DA code switching Arabic dialects lack large amounts of consistent This paper explains the design decisions we have data due to two main factors: the lack of ortho- made in order to meet these goals. DIWAN is graphic standards for the dialects, and the lack of fully implemented for use on Microsoft Win- overall Arabic content on the web (Benajiba et dows and is currently in use for the annotation of al., 2010). While the rise of the internet has in- Palestinian, Yemeni, and Moroccan Arabic. creased the amount of DA being written, some- times Arabic dialects come mixed with the MSA This paper is structured as follows. In Section 2, in various forms of text (see Figure 1, which we review the NLP components we use in DI- shows the code switching in our DIWAN tool). WAN. In Section 3, we describe the workflow Furthermore, language used in social media pos- when using DIWAN. In Section 4, we describe es a challenge for NLP tools in general in any the specific annotation tasks the annotator per- language due to the difference in genre. There- forms. Section 5 gives some technical detail fore, in order to create tools for dialectal Arabic, about the implementation. Section 6 discusses annotated DA corpora are needed in a variety of related work. We conclude in Section 7 with a dialects. discussion of future work. The goal of our Dialectal Word Annotation tool (DIWAN) is to address these gaps on the re- 2. NLP Resources used in DIWAN source creation level. In designing DIWAN, we have determined several important design goals: In order to make the annotation task easier, DI- WAN uses three main existing NLP resources: 1. We want to exploit the similarity between the MSA morphological analyzer SAMA, the dialects as much as possible to facilitate an- Egyptian morphological analyzer CALIMA- notation, which in general is costly and EGY, and the morphological tagger MADAMI- slow. RA which works for both MSA and Egyptian. 2. We want to use a convention for orthogra- We describe them in turn. phy (which the input text does not neces- sarily follow). The first resource is the Standard Arabic Mor- 3. We want to create data which can be used phological Analyzer, SAMA 3.1 (Graff et al. both for creating morphological analyzers 2009), which is based on the BAMA analyzer (which produce all morphological analyses (Buckwalter 2004). This system uses lexical for a word outside of any context) and mor- databases, divided into prefixes, stems, and suf- phological taggers (which determine the fixes, to assign words all possible MSA analyses. correct morphological analysis -- including A sample output is shown in Figure 2 (in Buck- ﻣﺎﺷﻲ the POS tag -- for a word in context). walter transliteration), for the input word mA$i, which is ambiguous between various in- flected forms of a verb meaning `walk’. 50 mA$y ﻲﺷﺎﻣ Figure 2: SAMA result for search on word mA$y ﻲﺷﺎﻣ Figure 3: CALIMA-Egyptian result for search on word The second resource is the Columbia Arabic 3. DIWAN Workflow Language and dialect Morphological Analyzer for Egyptian (CALIMA-EGY) (Habash et al. We designed and built DIWAN as a desktop ap- 2012b). It is an analyzer for Egyptian. A sample plication which can work locally (offline) or output is shown in Figure 3 for the input word online. As an annotation tool, we have designed mA$y. CALIMA returns the MSA readings DIWAN with two types of users: administrators ﻣﺎﺷﻲ shown in Figure 2, and in addition has Egyptian and annotators. The administrator’s responsibil- readings, in particular the interjection `OK’. ity is to create the DIWAN database, specify its settings, and track the annotator’s work. The third resource is MADAMIRA (Pasha et al. 2014). MADAMIRA is a system for morpholog- The administrator has several roles: ical analysis and disambiguation of Arabic that combines some of the best aspects of two previ- 1. She can create, edit and delete tables in ously commonly used systems for Arabic pro- the database. cessing, MADA (Habash and Rambow, 2005; 2. She can create, edit and delete annotator Habash et al., 2009; Habash et al., 2013) and accounts. AMIRA (Diab et al., 2007). MADAMIRA im- 3. She can check the status of the annota- proves upon the two systems with a more stream- tion tasks for each annotator. lined Java implementation that is more robust, 4. She can trace the annotator progress, portable, extensible, and is faster than its ances- work time, errors, etc. tors by more than an order of magnitude. Con- 5. She can generate reports and statistics on trary to SAMA and CALIMA-EGY, which pro- the underlying database (created by the vide all morphological analyses for a word re- annotators). gardless of context, MADAMIRA chooses a sin- 6. She can of course also annotate the data. gle analysis given the context of the word in a -The annotators can only annotate data. The ad ﻣﺎﺷﻲ ﻛﺪه ؟ sentence. For example, in the sentence mA$i kdh? `Is that OK?’, the interjection mean- ministrator assigns tasks to each annotator, and ing will be chosen. the annotations are added to the DIWAN data- base. As the annotator creates annotations, he 51 can reuse the resulting lexical entries in the DI- WAN tool as a new resource for himself or for other annotators. To work with DIWAN, the administrator first prepares the data. We assume that the data is DA written in Arabic script. There are two ways of preparing the data: 1. The administrator can either simply use DI- WAN itself to identify sentences and words in the corpus. DIWAN extracts sentences and words from the prepared file and builds a DIWAN database. 2. Or the administrator can send the corpus to MADAMIRA. MADAMIRA not only iden- tifies sentences and words, it also performs morphological analysis (using MSA and Egyptian resources) and tagging, making a single analysis available for each input word in context. After getting the resulting data Figure 4: DIWAN setup workflow from MADAMIRA, DIWAN will present the analysis for each word to the annotator as a default annotation option. As in the previ- ous case, DIWAN extracts sentences and words from the prepared file and builds a When online, several annotators can work at DIWAN database (which now includes the once, sharing their work immediately through the MADAMIRA analysis). centralized database. These two options are shown in Figure 4. 4.2. Choice of Word to Annotate and The annotator makes the dialect annotations by Using the Resources using the DIWAN GUI.