Corpora and Processing Tools for Non-Standard Contemporary and Diachronic Balkan Slavic

Home , Bulgarian language, Standard Macedonian

Zurich Open Repository and Archive University of Zurich Main Library Strickhofstrasse 39 CH-8057 Zurich www.zora.uzh.ch

Year: 2019

Vukovic, Teodora ; Nora, Muheim ; Winistörfer, Olivier-Andreas ; Anastasia, Makarova ; Ivan, Šimko ; Sanja, Bradjan

Abstract: The paper describes three corpora of different varieties of BS that are currently being developed with the goal of providing data for the analysis of the diatopic and diachronic variation in non-standard Balkan Slavic. The corpora includes spoken materials from Torlak, Macedonian dialects, as well as the manuscripts of pre-standardized Bulgarian. Apart from the texts, tools for PoS annotation and lemmatization for all varieties are being created, as well as syntactic parsing for Torlak and Bulgarian varieties. The corpora are built using a unified methodology, relying on the pest practices and state- of-the-art methods from the field. The uniform methodology allows the contrastive analysis of thedata from different varieties. The corpora under construction can be considered a crucial contribution tothe linguistic research on the languages in the Balkans as they provide the lacking data needed for the studies of linguistic variation in the Balkan Slavic, and enable the comparison of the said varieties with other neighbouring languages.

Posted at the Zurich Open Repository and Archive, University of Zurich ZORA URL: https://doi.org/10.5167/uzh-175260 Conference or Workshop Item Published Version

The following work is licensed under a Creative Commons: Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) License.

Originally published at: Vukovic, Teodora; Nora, Muheim; Winistörfer, Olivier-Andreas; Anastasia, Makarova; Ivan, Šimko; Sanja, Bradjan (2019). Corpora and Processing Tools for Non-Standard Contemporary and Diachronic Balkan Slavic. In: The 12th International Conference on Recent Advances in Natural Language Process- ing (RANLP 2019), Varna, Bulgaria, 2 September 2019 - 4 September 2019, 62-68. Corpora and Processing Tools for Non-Standard Contemporary and Diachronic Balkan Slavic Teodora Vukovic´ Nora Muheim Olivier Winistorfer¨ Ivan Simkoˇ Anastasia Makarova Sanja Bradjan

Slavic Departement, University of Zurich, Plattenstrasse 43, Zurich [email protected], [email protected], [email protected], [email protected], [email protected], [email protected]

Abstract inflection and fully or partially grammaticalized post-positive definite articles (Lindstedt, 2000). The paper describes three corpora of Many other differences lie in the nominal and ff di erent varieties of BS that are currently verbal morphosyntax, adjectival morphology, and being developed with the goal of providing tense and mood system, as well as in the lexical and data for the analysis of the diatopic phraseological domain. Nonetheless, the area itself and diachronic variation in non-standard is not linguistically uniform. On the contrary - the Balkan Slavic. The corpora includes diversification starts from the division into official spoken materials from Torlak, Macedonian standard languages, further separated by various dialects, as well as the manuscripts of phonological and structural isoglosses (Ivic´, 1985; pre-standardized Bulgarian. Apart from Institute for Bulgarian Language, 2018; Stojkov, the texts, tools for PoS annotation 2002; Vidoeski, 1999). Variation occurs even in and lemmatization for all varieties are very small regions, such as the Timok area in being created, as well as syntactic South East Serbia, where substantial differences parsing for Torlak and Bulgarian varieties. occur in the use of post-posed articles between The corpora are built using a unified villages in the mountains and valleys or urban methodology, relying on the pest practices areas (Vukovic´ and Samardziˇ c´, 2018). There is a and state-of-the-art methods from the significant variation not only from an areal, but field. The uniform methodology allows also from a diachronic point of view. Looking at the contrastive analysis of the data from older texts written in pre-standardized Bulgarian, ff di erent varieties. The corpora under phases of change in the language happening over construction can be considered a crucial time are noticeable. contribution to the linguistic research on This variation is best analyzed in non-standard the languages in the Balkans as they varieties, unaffected or minimally affected by the provide the lacking data needed for the prescriptive standard, and ideally in the form of studies of linguistic variation in the Balkan spoken language since it represents arguably a Slavic, and enable the comparison of the more "natural"state. In this paper, we present three said varieties with other neighbouring ongoing projects on South Slavic dialectology and languages. diachronic linguistics, which are currently being 1 Introduction elaborated at the Slavic linguistics department at the University of Zurich. In these projects, special Balkan Slavic (BS) languages are the eastern focus is on places in time and space where the branch of South Slavic languages, which are change happens and the underlying grammatical known for their affiliation to the so-called processes and structures. Apart from linguistic Balkansprachbund. The languages that belong to questions, the focus of our research is on the this group are Bulgarian and Macedonian varieties identification of potential geographical and social as well as the Torlak dialects spoken in South factors influencing changes in Bulgarian, Torlak East Serbia and West Bulgaria. These languages and Macedonian. express typological differences from other Slavic For the purpose of this research, we are languages. Some of the differentiating features are: aiming at three corpora (and also processing complete or almost complete loss of the noun case

62 Proceedings of the Student Research Workshop (RANLPStud 2019) associated with RANLP 2019, pages 62–68 Varna, Bulgaria, 2 – 4 September, 2019. tools) for modern and historical non-standard in size. Texts were transcribed into Latin and BS varieties that reflect diachronic and diatopic Cyrillic script in order to make the data available variation within the Balkan Slavic languages. The to a wider audience. The texts are annotated with varieties covered are pre-standardized Bulgarian, grammatical information, lemmas, English glosses Macedonian dialects, and Torlak varieties from for each token, and English translations for each Serbia and Bulgaria. Apart from the large line (see the annotated sentence in example 1, collection of texts, the resources are equipped presented as it is in the corpus). The MSD are with part-of-speech (PoS) and morphosyntactic easily readable, but do not fin the standards used description (MSD) annotation, while some corpora in the field. The database represents an impressive also include a syntactic tree-bank and a layer achievement, and a particularly valuable one given of normalization. In order to make the corpora that it is the sole resource for dialectal research on mutually comparable, we are developing and BS at the moment. applying a uniform methodology. Methodology, (1) hm kvo´ se corpora and tools are based on the existing disc what.Sg.N.Interr Acc.Refl.Clt standards used in the field as well as resources . какво се for the standard languages of the area, wherever kazva´ seˇ say.3Sg.Impf.I available. This enables inter-comparability and казвам reproducibility of the data. At the same time re- ‘Hmmmm. What is it called?’ using already existing tools makes the process easier for the creators and it offers well-established The corpora described in this paper are methods too, which are discussed below. created with the goal to be comparable with other existing related resources – be it corpora 2 Related Work of dialects or of the standard language. The automatic processing tools developed for the There is currently little data available in digital language varieties included in the collection are form that allows the analysis of the mentioned created based on existing ones whenever possible. variation and change in BS. Bulgarian is the For example, the morphosyntactic tagger and only language supplemented with a corpus of lemmatizer for Torlak is an adaptation of the tagger non-standard spoken varieties apart from the for standard Serbian (Miliceviˇ c´ and Ljubesiˇ c´, standard ones (Alexander, Ronelle and Zhobov, 2016a). Vladimir, 2016; Alexander, 2015). Serbian only An important tool for Serbian is the ReLDI has corpora of written standard and computer- tagger, which assigns morphosyntactic tags and mediated communication (CMC) (CCSL; Utvic´, lemmas to text (Ljubesiˇ c´, 2016). The tagger 2011; Ljubesiˇ c´ and Klubickaˇ , 2014; Miliceviˇ c´ and was developed for standard Serbian, Croatian, Ljubesiˇ c´, 2016b). The only available searchable and Slovene using a character-level machine resource for Macedonian is an unannotated corpus translation method. It assigns tags specified by the of movie subtitles (Steinberger et al., 2014; Lison MULTEXT-East standards (Krstev et al., 2004). and Tiedemann, 2016). The mentioned corpora The python package allows training for any other of standard language (or movie subtitles) are variety with an novel training set as an input extremely valuable resources in itself, but they (Ljubesiˇ c´, 2016). provide little-to-no insight into variation because MULTEXT-East resources represent an they are a sample of only one variety by definition; extremely important milestone in the field furthermore, that standard-variety is by default of computational linguistic for South Slavic controlled by people’s prescribed conceptions of languages (Erjavec, 2010). The collection includes how language should be and how it should be the manually annotated novel 1984 by George written. Orwell that can be used for training and testing of The only dialect corpus of BS, Bulgarian resources and specifications for morphosyntactic Dialectology as Living Tradition (Alexander, descriptions. The PoS labels are formulated as a Ronelle and Zhobov, Vladimir, 2016; Alexander, string of characters, where each character stands 2015) is a database of oral speech comprising for a grammatical category (e.g. the tag Ncfsn for 184 texts from 69 Bulgarian villages, recorded the Serbian noun kuca,´ ‘house’, means ‘Noun, between 1986 and 2013, and is 95,000 tokens common, feminine, singular, nominative’).

63 The recently widely used convention of the interval on the recording. Texts that have been Universal Dependencies (UD) database contains digitized from prints preserve the segmentation tools and specifications for the annotation of into sentences from the original version. Lastly, morphology and syntactic parsing for many texts derived from written manuscripts, which languages, using universal grammatical categories do not always have clear sentence structure or founded in dependency grammar. The repository punctuation, have been segmented into sentences includes tree-banks and MSD taggers for the in post-editing based syntactic structure and closely related South Slavic languages Serbian and meaning. Bulgarian. Each corpus contains several layers. The minimum are the original text with automatically 3 Balkan Slavic Corpora assigned PoS tags and lemmas, while some In order to bridge this gap and supply the materials also contain some form of standardization or necessary for the analysis of the multi-sided normalization. The structure of the corpus allows variation in BS, we are creating several spoken more information to be added over time (e.g. and historical corpora of varieties from the region, English glosses or an English translation). namely the territories in present-day Serbia, When it comes to PoS and MSD annotation, Bulgaria, and Macedonia. The individual corpora the MULTEXT-East tag-set is used, because it is are presently at different stages of development. a widely accepted standard for morphologically Contemporary materials have been collected in the complex languages of Eastern Europe. A further past 10 years, while the historical data comprises advantage is its easy adaptability to new some more recent 19th and 20th century resources grammatical categories, so the grammar of ff that could be classified as micro-diachronical di erent varieties can be matched. We chose (representing a shorter time span), as well as older the MULTEXT-East tag-set over UD because manuscripts dating from 16th-19th century, which they are mutually compatible. Namely, they both ff provide a view on the language change on a larger mark the same categories but in a di erent way scale. The goal is to establish a pipeline and tools (e.g. the MULTEXT-East tag ‘Ncmsn’ would that match the needs for non-standard varieties and be converted to the UD tag ‘UPOS=NOUN, produce comparable resources and would in turn Case=Nom, Gender=Masc, Number=Sing’). They allow the analysis of variation in a set of close can be easily transformed from one to the other, and languages. in fact, this has already been done for the Serbian UD Tree-bank (Samardziˇ c´ et al., 2017). 3.1 Corpus Structure and Methodology All the corpora are provided with relevant In order to make the corpora comparable among metadata containing (where possible) age, gender, themselves but also with other corpora of year of recording and main occupation of the neighboring languages, we are applying some informants as well as geo-spatial information standards from the field as well as creating them to about speaker locations. In the case of the pre- fit a uniform structure. This will also result in ease standardized Bulgarian corpus, the metadata base of access and user-friendliness. consists of approximate dating of the manuscripts For texts which originate from audio recordings, and supposed location of its creation. The metadata transcripts have been made using Exmaralda for the dialectal Macedonian corpus is sometimes ff (Schmidt, 2010) transcription software, developed fragmentary because of the di erent working specifically for linguistic transcription. The standards used and it is not possible to recover optical character recognition (OCR) for the the lacking information. The metadata may be printed texts and manuscripts was performed in later used as a starting point for the analysis of Transkribus (Digitisation and Digital Preservation the correlation of the linguistic data with non- group, 2019), another piece of software created linguistic factors. at the University of Innsbruck for automatic The corpora are stored in files with XML transcriptions of older texts and scripts. markup in accordance with the TEI standards Transcripts of spoken materials are segmented for spoken language and manuscripts (TEI, into utterances based on intonation or syntactic 2019). Aligned audio files are currently not patterns, and each utterance is aligned with an supplemented with the recordings due to the

64 lacking infrastructure. However, recordings can from that sample. The accuracy of the tagger on be accessed on the project’s YouTube channel the data Timok and Luznicaˇ is 92.9% for the (TraCeBa, 2015). The corpora will be made PoS labels and 93.9% for the lemmas. However, available online and freely accessible. the accuracy is lower for the other sections of Each corpus has been tailored to match the this corpus from the 1990s and earlier, and from methodology described above. This way different Bulgaria. We are currently adding more manually samples can be searched at the same time and annotated data from these sources to improve the the results compared. The following subsections results. An example of a sentence from the corpus present individual corpora on various BS varieties. annotated with MULTEXT-East PoS tags in the second line and lemmas in the third line is given in 3.2 Torlak Corpus example 2. The contemporary section of the Torlak corpus is based on fieldwork recordings from the Timok (2) On dosAlˇ sInocˇ iz zAjcarˇ Pp3msn Vmp-sm Rgp Sg Npmsa and Luznicaˇ regions in South-East Serbia, and on doci´ sinoc´ iz Zajecarˇ areas around Belogradcikˇ in Western Bulgaria. ‘He came last night from Zajecar.’ˇ The interviews have been transcribed using Exmaralda (Schmidt, 2010). The micro-diachronic Apart from the morphological annotations, we part of the corpus includes dialect transcripts are presently developing a UD-based tree-bank form East Serbia and West Bulgaria (Sobolev, using the labels from the Serbian UD treebank 1998) collected by Andrey N. Sobolev in the (Samardziˇ c´ et al., 2017). The data has been pre- 1990s, which have been digitized from the printed annotated with the parser for Serbian and is being version using Transkribus (Digitisation and Digital manually corrected in Arborator (Gerdes, 2013). Preservation group, 2019) as well as the data collected in the beginning of the 20th century 3.3 Macedonian Dialect Corpus by Belic(´ Belic´, 1905) and Stanojevic(´ Stanojevic´, The goal in this project is to create the first 1911). Two parts of the collection have been corpus of spoken Macedonian dialects, annotated completed so far. The collection from Timok with PoS tags and with lemmatized tokens. contains around 350,000 tokens and the one from The data is mainly drawn from transcripts of the 1990s has close to 100,000 tokens. The other field-work interviews with older people from data is currently being transcribed and will contain different locations collected by Vidoeski from the in total roughly 200,000 tokens. 1950s until the 1970s. This text-collection also Semi-phonetic transcription of spoken data have includes interviews from other researchers besides been made to reflect the spoken language as well Vidoeski, of which some work is considerably as possible while maintaining a necessary level of older than Vidoeski’s; several interviews are even readability. The transcripts of audio recordings and dating back to 1892. All the texts have been those of the printed interviews contain information published by Vidoeski 1999. The covered period about the accent position encoded in capital letters. of time gives the corpus a certain diachronic They also include information about interruptions depth. The texts have been transcribed using and overlaps, which is not available for the Transkribus (Digitisation and Digital Preservation interviews recovered from print. group, 2019). The modern state of Western We have developed a PoS tagger and a Macedonian dialects is presented by the recent field lemmatizer for the contemporary spoken data from data from multi-ethnic Ohrid, Prespa, Struga and Timok and Luznicaˇ using the ReLDI model. The Debar regions collected in 2013 - 2019 (Makarova, training data and the lexicon combines Serbian and 2019). The contemporary data allows a contrastive dialect material. For the Serbian part we used the analysis of the hypothesized change. SETTimes reference training corpus (Batanovic´, The data comes from diverse origins, so a unified 2018) and the lexicon SrLex 1.2 (Ljubesiˇ c´ and metadata scheme cannot be applied to all the Jazbec, 2016), both freely available. The dialect collections. As these interviews were not planned part consists of the 20,000 tokens, which have as one project, every researcher defined their own been pre-annotated with the ReLDI tagger, and standard. To partially solve this challenge and then manually corrected and the lexicon derived guarantee some uniformity, a standardized frame is used, where potential gaps are clearly stated.

65 This allows the user to decide for themselves which Parallel corpora consisting of editions of the same amount of background information is enough (e.g. text from various stages and dialectal or literary if they accept an unclear year of recording or no backgrounds, enable us to observe the development information about the speaker’s sex) and whether of linguistic features or orthographic influences they want to include such parts of the corpus with independently of the differences caused by contents missing information in their research or not. and genre. The manuscripts have been digitized There are currently practically no automatic from the printed or handwritten versions using tools that could be used to do PoS annotation Transkribus (Digitisation and Digital Preservation for dialectal Macedonian, so they need to be group, 2019). developed specifically for this corpus. The only The MSD labels are based on the MULTEXT- previous attempt to produce an automatic tagger East specifications for Modern Bulgarian. The for Macedonian has been done by Aepli et al.2014, purpose of this corpus is to provide material for where they solely used part-of-speech categories the study of the changes in the morphosyntactic with no morphology at all. In our approach, we will features between Middle and Modern Bulgarian. use the MULTEXT-East tag-set for Macedonian This makes the the standardized Bulgarian tag-set with minor modifications to accommodate the unsuitable. To overcome instances of ambiguity dialectal categories not present in the standard within a text and within the corpus, we adapted it to (such as nominal cases for instance) and the ReLDI reflect both archaic and innovative features. These tagger. To train the initial model we will use include nominal case markers (e.g. dative and the manually annotated corpus and the lexicon genitive-accusative being regularly marked on both provided in the MULTEXT-East collection for masculine nouns and adjectives), verbal infinitives standard Macedonian. After the initial training (e.g. koi mozeˇ iskaza ‘who could retell’) and with this material we will correct the results to multiple options to mark the definiteness (short- take the dialectal forms and variations into account. and long-forms of the adjective, articles tagged The manually annotated sample will then be used as separate tokens). Phonetic ambiguities (e.g. /i/ to train a new model, suitable for the many dialects and /y/) were resolved by conventions based on covered by the corpus. The following example the development of the sound in the approximate shows one manually annotated sentence from the area of origin of the text. In order to avoid any contemporary material (Makarova, 2019): over-interpretation or bias, the tags used for cases refer to morphological and not syntactical (e.g. (3) Pominav mnogu ubo detstvo. verbal voice) or semantic (e.g. difference between Vmia1s-anp Rgp Rgp Ncnsnn pomine mnogu ubavo detstvo common and proper nouns) criteria. ‘I remember a lot from childhood.’ The literature of this period inherited the complex orthography of Church Slavonic. Already 3.4 Pre-Standardized Bulgarian in the Middle Bulgarian period, it was the case that many of its rules were obsolete. The corpus for pre-standardized Bulgarian Both Church Slavonic and vernacular literature contains texts from the period between the 16th attempted to follow these rules, but they weren’t and 19th century, mostly, but not exclusively applied consequently. In the end, the same from present-day Bulgaria. The texts are chosen lemma may appear with different spellings, according to the similarity of their language sometimes even within the same sentence. The and the vernacular. Thus, Church Slavonic texts different manifestations of the same lemma were generally avoided, but some of them were were partly unified by using a diplomatic added for reference. The collection includes texts transcription, eliminating ambiguous signs (e.g. from the Damaskini tradition either as a whole accents, writing of /i/). Furthermore they were (Kotel, Ljubljana, Lovec,ˇ Tixonravov and Svistovˇ unified under the lemmas of the dictionary based damaskini, Pop Punco’sˇ miscellany, perspectively on Tixonravov Damaskin, see (Demina, 2012). also manuscript NBKM 328 of Iosif Bradati), Turkish loanwords, Church Slavonicisms, and or as a parallel corpus of multiple versions of Russian words not included in this dictionary were a recurring story, (e.g. Life of St Petka, Second lemmatized with the Etymological dictionary of Coming of Christ or multiple transcripts of the BAN (Georgiev, 2006; Todorov, 2002) or with Slavobulgarian Chronicle by Paisius of Hilandar).

66 Church Slavonic dictionaries, e.g. (Cejtlin, 1994). Science Foundation IZRPZ0_177557/1), Ill-bred The ﬁrst instance of the PoS tagger and sons project (Swiss National Science Foundation the lemmatizer was trained on a sample of 100015_176378/1, Swiss Excellence Government 6000 manually annotated tokens using the Scholarship awarded to Teodora Vukovic´ and ReLDI framework (Ljubesiˇ c´, 2016). Given the Anastasia Makarova, Language and Space UZH, unsatisfactory accuracy, we are in the process of SyNoDe UZH, Slavisches Seminar UZH, Doctoral adding more manually annotated training data. A Program Linguistics UZH and Ministry of Culture sample of an annotated sentence from the corpus and Information of Serbia. We would like to is given in the example 4. At the same time express our gratitude for their support. we are working on a UD-style tree-bank using syntactic labels taken from Bulgarian and Serbian speciﬁcations (Samardziˇ c´ et al., 2017; Osenova and References Simov, 2004). Aepli, N., von Waldenfels, R., and Samardziˇ c,´ T. (2014). Part-of-speech tag disambiguation by cross- (4) Predade+ˇ syˆ dsˇa+´ ta bu. linguistic majority vote. In Proceedings of the Vmia3s Px—d Nfsnn Pa-fsn Nmsdy First Workshop on Applying NLP Tools to Similar predamˇ se dusaˇ ta bog Languages, Varieties and Dialects, pages 76–84. ‘He surrendered his soul to the God.’ Alexander, R. (2015). Bulgarian dialectology as living tradition: A digital resource of dialectal speech. 4 Conclusion pages 1–13.

In joining our work on different languages with Alexander, Ronelle and Zhobov, Vladimir (2016). similar challenges, we are able to show how to deal Bulgarian dialectology as living tradition. with variation in corpora in a principled way and Batanovic,´ Vuk; Ljubesiˇ c,´ N. S. T. (2018). Setimes.sr therefore contribute to the field of dialectology, — a reference training corpus of serbian. pages 11– on the one hand, and corpus linguistics on –17, Ljubljana, Slovenia. the other. Secondly, our approach demonstrates Belic,´ A. (1905). Dijalekti istocneˇ i juzneˇ Srbije. the fruitfulness of combining methodology for Srpski dijalektoloskiˇ zbornik. Sprska Kraljevska multiple similar projects, by taking advantage of Akademija. the best practices and state-of-the-art methods and tools. The unified methodology in turn guarantees Cejtlin, Ralja Vecerka;ˇ Radoslav Blagova, E. (1994). Staroslavjanskij slovar: po rukopisjam X-XI vekov. comparability of the data, which is required for Russkij jazyk, Moskva. the analysis of change and variation in several different varieties. The corpora under construction Demina, Evgenia Miceva;ˇ Vania Seizova, S. (2012). Recnikˇ na knizovnijaˇ balgarski˘ ezik na narodna in the context of our projects can be considered a osnova ot XVII vek. Valentin Trajanov, Sofia. significant contribution to the linguistic research on the languages in the Balkans as they provide the Digitisation and Digital Preservation group (2019). lacking data needed for linguistic studies of BS, as Transkribus. https://transkribus.eu/ Transkribus/. [Online; accessed 09-July-2019]. well as comparison of the mentioned varieties with other neighbouring languages. Erjavec, T. (2010). Multext-east version 4 The output of these projects will be the : multilingual morphosyntactic specifications, lexicons and corpora. In V: Proceedings, LREC corpora of spoken and written non-standard 2010, 7th International Conference on Language language equipped with with PoS annotation Resources and Evaluations, pages 2544–2547. and lemmatization, as well as UD tree-banks. Additionally, the tools for automatic processing Georgiev, V. (1972–2006). Balgarski˘ etimologicenˇ recnikˇ I-V. Marin Drinov, Sofia. will be available for re-use, as well as training data and lexicons developed based on them. All of the Gerdes, K. (2013). Collaborative dependency resources will be made available online. annotation. In Proceedings of the Second International Conference on Dependency Linguistics Acknowledgments (DepLing 2013), Prague, Czech Republic. Charles University in Prague, Matfyzpress, Prague, Czech The development of the resources described in the Republic. paper has been funded by various sources: TraCeBa project (EraNet Rus Plus grant/Swiss National

67 Institute for Bulgarian Language (2018). Map of the Miliceviˇ c,´ M. and Ljubesiˇ c,´ N. (2016b). Tviterasi, dialectal division of the bulgarian language. http: tviterasiˇ or twitterasi?ˇ Producing and analysing a //ibl.bas.bg/bulgarian_dialects/. [Online; normalised dataset of Croatian and Serbian tweets. accessed 05-July-2018]. Slovensˇcinaˇ 2.0. Ivic,´ P. (1985). Dijalektologija srpskohrvatskog jezika: Osenova, P. and Simov, K. (2004). Btb-tr05: Uvod i stokavskoˇ narecjeˇ . Matica srpska, Novi Sad. Bultreebank stylebook 05. Technical report. Krstev, C., Vitas, D., and Erjavec, T. (2004). Morpho- Samardziˇ c,´ T., Starovic,´ M., Agic,´ v., and Ljubesiˇ c,´ Syntactic Descriptions in MULTEXT-East - the Case N. (2017). Universal dependencies for serbian in of Serbian. Informatica, No. 28. comparison with croatian and other slavic languages. In Proceedings of the 6th Workshop on Balto- Lindstedt, J. (2000). Linguistic balkanization: Contact- Slavic Natural Language Processing, pages 39– induced change by mutual reinforcement. In Gilbers, 44, Valencia, Spain. Association for Computational D., editor, Languages in contact, number 28 in Linguistics. Studies in Slavic and general linguistics. Rodopi, Amsterdam. Schmidt, T. (2010). Exmaralda: un systeme´ pour la constitution et l’exploitation de corpus oraux. Pour Lison, P. and Tiedemann, J. (2016). Opensubtitles2016: une epist´ emologie´ de la sociolinguistique. Actes du Extracting large parallel corpora from movie colloque international de Montpellier, pages 319– and tv subtitles. In Proceedings of the 10th 327. International Conference on Language Resources Sobolev, A. (1998). Sprachatlas Ostserbiens und and Evaluation (LREC 2016). European Language Westbulgariens. Scripta Slavica. Biblion. Resources Association (ELRA). Stanojevic,´ M. (1911). Severno-timackiˇ dijalekat. Ljubesiˇ c,´ Nikola; Klubicka,ˇ F. A. Z. and Jazbec, I.- Dijalektoloskiˇ zbornik. Sprska Akademija Nauka. P. (2016). New inflectional lexicons and training corpora for improved morphosyntactic annotation of Steinberger, R., Ebrahim, M., Poulis, A., Carrasco- croatian and serbian. In Proceedings of the Tenth Benitez, M., Schluter,¨ P., Przybyszewski, M., International Conference on Language Resources and Gilbro, S. (2014). An overview of the and Evaluation (LREC 2016). european union’s highly multilingual parallel corpora. Language resources and evaluation, pages Ljubesiˇ c,´ Nikola; Klubicka,ˇ F. A. Z. J. I.-P. (2016). 679–707. New inflectional lexicons and training corpora for improved morphosyntactic annotation of croatian Stojkov, S. (2002). Balgarskaˇ dialektologija. Akad. and serbian. In Calzolari, N., Choukri, K., izd. Prof. Marin Drinov, Sofia. Declerck, T., Goggi, S., Grobelnik, M., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., TEI, T. E. I. (2019). P5: Guidelines for Electronic Text and Piperidis, S., editors, Proceedings of the Tenth Encoding and Interchange. International Conference on Language Resources CCSL. Corpus of contemporary serbian and Evaluation (LREC’16), Portoroz.ˇ European language - official website (in serbian). Language Resources Association (ELRA). http://www.korpus.matf.bg.ac.rs/ prezentacija/istorija.html. [Online; Ljubesiˇ c,´ Nikola; Klubicka,ˇ F. A. Z. J. I.-P. (2016). accessed 08-March-2018]. New inflectional lexicons and training corpora for improved morphosyntactic annotation of croatian Todorov, T. A., editor (2002). Balgarski˘ etimologicenˇ and serbian. In Chair), N. C. C., Choukri, K., recnikˇ VI-VII. Sofia. Declerck, T., Goggi, S., Grobelnik, M., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, TraCeBa (2015). Terenska istrazivanja. J., and Piperidis, S., editors, Proceedings of https://www.youtube.com/channel/ the Tenth International Conference on Language UC4EpCSAnEb2RIsIRY7pfNdQ/feed. [Online; Resources and Evaluation (LREC 2016), Paris, accessed 20-August-2019]. France. European Language Resources Association Utvic,´ M. (2011). Anotacija korpusa savremenog (ELRA). srpskog jezika. INFOteka 12, br. 2, pages 39–51. Ljubesiˇ c,´ N. and Klubicka,ˇ F. (2014). Wac – web Vidoeski, B. (1999). Dijalektite na Makedonskiot Jazik corpora of bosnian, croatian and serbian. In - tom 2. MANU, Skopje. Proceedings of the 9th Web as Corpus Workshop (WaC-9), pages 29––35. Vukovic,´ T. and Samardziˇ c,´ T. (2018). Prostorna raspodela frekvencije postpozitivnog clanaˇ u Makarova, A. (2019). Unpublished work. timockomˇ govoru. In Cirkovi´ c,´ Svetlana (Ed. in chief) and Andrej N. Sobolev, Barbara Miliceviˇ c,´ M. and Ljubesiˇ c,´ N. (2016a). Tviterasi, Sonnenhauser, Maja Miliceviˇ c,´ Jelenka Pandurevic,´ tviterasiˇ or twitterasi?ˇ producing and analysing a editor, Timok. Terenska istrazivanjaˇ 2015-–2017. normalised dataset of croatian and serbian tweets. Narodna Biblioteka “Njegos”,ˇ Knjazevac.ˇ Slovensˇcinaˇ 2.0, 4(2):156–188.