Linguistic Resources & Topic Models for the Analysis of Persian Poems Ehsaneddin Asgari and Jean-Cédric Chappelier Ecole Polytechnique Fédérale de Lausanne (EPFL) School of Computer and Communication Sciences (IC) CH-1015 Lausanne ; Switzerland [email protected] and [email protected] Abstract This paper describes the usage of Natural Lan- guage Processing tools, mostly probabilistic topic modeling, to study semantics (word cor- relations) in a collection of Persian poems con- sisting of roughly 18k poems from 30 differ- ent poets. For this study, we put a lot of ef- fort in the preprocessing and the development of a large scope lexicon supporting both mod- ern and ancient Persian. In the analysis step, we obtained very interesting and meaningful results regarding the correlation between po- ets and topics, their evolution through time, Figure 1: Elements of a typical Ghazal (by Hafez, as well as the correlation between the topics calligraphed by K. Khoroush). Note that Persian is and the metre used in the poems. This work should thus provide valuable results to litera- right to left in writing. ture researchers, especially for those working on stylistics or comparative literature. about 300 metres in Persian poems, 270 of which are rare, the vast majority of poems composed only from 1 Context and Objectives 30 metres (Mojiry and Minaei-Bidgoli, 2008). The purpose of this work is to use Natural Language Ghazal traditionally deals with just one subject, Processing (NLP) tools, among which probabilis- each couplet focusing on one idea. The words in tic topic models (Buntine, 2002; Blei et al., 2003; a couplet are thus very correlated. However, de- Blei, 2012), to study word correlations in a special pending on the rest of the couplets, the message of a couplet could often be interpreted differently due one of ,(ﻏﺰل) ”type of Persian poems called “Ghazal the most popular Persian poem forms originating in to the many literature techniques that can be found 6th Arabic century. in Ghazals, e.g. metaphors, homonyms, personifica- Ghazal is a poetic form consisting of rhythmic tion, paradox, alliteration. couplets with a rhyming refrain (see Figure 1). Each For this study, we downloaded from the Gan- 1 couplet consists of two phrases, called hemistichs. joor poems website , with free permission to use, a Syllables in all of the hemistichs of a given Ghazal Ghazal collection corresponding to 30 poets, from follow the same pattern of heavy and light syllables. Hakim Sanai (1080) to Rahi Moayyeri (1968), Such a pattern introduces a musical rhythm, called with a total of 17, 939 Ghazals containing about metre. Metre is one of the most important proper- 170, 000 couplets. The metres, as determined by ex- ties of Persian poems and the reason why usual Per- perts (Shamisa, 2004), are also provided for most po- sian grammar rules can be violated in poems, espe- ems. cially the order of the parts of speech. There exist 1http://ganjoor.net/. 23 Proceedings of the Second Workshop on Computational Linguistics for Literature, pages 23–31, Atlanta, Georgia, June 14, 2013. c 2013 Association for Computational Linguistics ârâst) as past root. Its injunctive form) آراﺳﺖ We put a lot of effort into the preprocessing, so and ﺑﺎرا be), leading to) ﺑِـ as to provide more informative input to the mod- requires it to be preceded by eling step. For this, we built a lexicon supporting (beârâ). However, according to phonological rules, -y) is intro) ﯾـ â), a) آ both modern and ancient Persian, as explained in when a consonant attaches to Section 2. In addition, we developed several pre- duced as a mediator. So the correct injunctive form .(”!byârâ,“decorate) ﺑﻴﺎرا processing tools for Persian and adapted them to po- is ems, as detailed in Section 3. In the analysis step, Mediators occur mainly when a consonant comes .(u) و exploiting Probabilistic Topic Models (Blei, 2012), before â or when a syllable comes after â or promising results were obtained as described in Sec- But the problem is slightly more complicated. For -jostan,“seek) ﺟﺴﺘﻦ tion 4: strong correlation between poets and topics instance, the present verb for ,am) ـَﻢ ju). Thus when the pronoun) ﺟﻮ was found by the model, as well as relevant patterns ing”) is ﺟﻮﯾﻢ in the dynamics of the topics over years; good corre- “I”) is attached, the conjugated form should be lation between topics and poem metre was also ob- (juyam,“I seek”), with a mediator. However, the ( ﺟﻮ served. root ju has also a homograph jav (also written -javidan,“chew) ﺟﻮﯾﺪن which is the present root of is pronounced v, not u, there و Modern and Ancient Persian Lexicon ing”). Since here 2 is no need for a mediator and the final form is ﺟﻮم ,This section presents the Persian lexicon we built (javam,“I chew”). Therefore, naively applying the which supports both modern and ancient Persian above mentioned simple rules is wrong and we must words and morphology and provides lemmas for all proceed more specifically. To overcome this kind of forms. This lexicon could thus be useful to many re- problem, we studied the related verbs one by one and search projects related to both traditional and mod- introduced the necessary exceptions. ern Persian text processing. Its total size is about In poems, things are becoming even more compli- 1.8 million terms, including the online version2 cated. Since metre and rhyme are really key parts of of the largest Persian Dictionary today (Dehkhoda, the poem, poets sometimes waives the regular struc- 1963). This is quite large in comparison with e.g. the tures and rules in order to save the rhyme or the morphological lexicon provided by Sagot & Walther metre (Tabib, 2005). For instance, F. Araqi in one (2010), of about 600k terms in total. ﻣﻧﺎﯾ of his Ghazals decided to use the verb form 2.1 Verbs (mi-nâyi,“you are not coming”) which does not fol- low the mediator rules, as it must be (mi- ﻣﻧﯿﺎﯾ -Taking advantage of the verb root collection pro naâyayi). The poet decided to use the above form, vided by Dadegan group (Rasooli et al., 2011), we which still makes sense, to preserve the metre. conjugated all of the regular forms of the Persian The problem of mediators aside, the orders of verbs which exist in modern Persian using grammars parts in the verb structures are also sometimes provided by M. R. Bateni (1970), and added them changed to preserve the metre/rhyme. For instance with their root forms (lemmas) to the lexicon. We in the future tense, the compound part of compound also added ancient grammatical forms, referring to verbs has normally to come first. A concrete exam- ancient grammar books for Persian (Bateni, 1970; ple is given by the verb (jân xâhad ﺟﺎن ﺧﻮاﻫﺪ ﺳﭙﺮد .(P. N. Xanlari,2009 sepord means “(s)he will give up his spirit and will Persian verb conjugation seems to be simple: nor- die”), which is written by Hafez as: ﺧﻮاﻫﺪ ﺳﭙﺮد ﺟﺎن mally each verb has two roots, past and present. In (xâhad sepord jân). To tackle these variations, we each conjugated form, the corresponding root comes included in our lexicon all the alternative forms men- with some prefixes and attached pronouns in a pre- tioned by Tabib (2005). defined order. However, phonological rules intro- As already mentioned, the considered poem col- duce some difficulties through so-called mediators. lection ranges from 1080 to 1968. From a linguis- For instance, the verb (ârâstan, meaning ”to tics point of view some grammatical structures of آراﺳﺘﻦ decorate” or ”to attire”) has (ârâ) as present root the language have changed over this long period of آرا 2http://www.loghatnaameh.org/. time. For instance, in ancient Persian the prefix for 24 hami); today only 3 Preprocessing) ﻫﻤ the continuity of verb was mi) is used. Many kinds of changes could be) ﻣ observed when ancient grammars are compared to Preprocessing is an essential part in NLP which usu- the modern one. The relevant structures to the men- ally plays an important role in the overall perfor- tioned period of time were extracted from a grammar mance of the system. In this work, preprocessing for book of ancient Persian (P. N. Xanlari,2009) and in- Persian Ghazals consists of tokenization, normaliza- cluded in our lexicon. tion, stemming/lemmatization and filtering. Starting from the 4,162 infinitives provided by 3.1 Tokenization Dadegan group (Rasooli et al., 2011) and consid- The purpose of tokenization is to split the poems ering ancient grammars, mediators, and properties into word/token sequences. As an illustration, a of poetic forms, we ended up with about 1.6 mil- hemistich like lion different conjugated verb forms. The underly- ﺷﺎه ﺷﻤﺸﺎد ﻗﺪان ﺧﺴﺮو ﺷﯿﺮﯾﻦ دﻫﻨﺎن ing new structures have exhaustively been tested by is split into the following tokens: a native Persian graduate student in literature and . ﺷﺎه / ﺷﻤﺸﺎد / ﻗﺪان / ﺧﺴﺮو / ﺷﯿﺮﯾﻦ / دﻫﻨﺎن linguistics. This validation took about one hundred The tokenization was done using separator char- hours of work, spot-checking all the conjugations for acters like white spaces, punctuation, etc. However, random selected infinitives. half-spaces made this process quite complicated, as most of them appeared to be ambiguous. 2.2 Other words (than verbs) Half-space is a hidden character which avoids pre- ceding letters to be attached to the following letters; The verbs aside, we also needed a complete list the letters in Persian having different glyphs when of other words. The existing usual Persian elec- attached to the preceding letters or not. mi-raft,“was going”), here) ﻣرﻓﺖ ,tronic lexica were insufficient for our purpose be- For instance cause they are mainly based on newspapers and written with a half-space separating its two parts, mi without ﻣﯿﺮﻓﺖ would be written (رﻓﺖ) and raft (ﻣ) do not necessarily support ancient words.