Quantitative Linguistic Investigations across Universal Dependencies Chiara Alzetta∗ ♠, Felice Dell’Orletta∗, Simonetta Montemagni∗, Petya Osenova† ♣, Kiril Simov†, Giulia Venturi∗ ∗ Istituto di Linguistica Computazionale “Antonio Zampolli”, CNR, Pisa ♠ DIBRIS, Università degli Studi di Genova, ♣ Sofia University † Artificial Intelligence and Language Technologies Department, IICT-BAS [email protected], {petya, kivs}@bultreebank.org, {felice.dellorletta, simonetta.montemagni, giulia.venturi}@ilc.cnr.it Abstract needed (Cotterell and Eisner, 2017; Bjerva and Augenstein, 2018). The paper illustrates a case study aimed Multilingual resources such as the depen- at identifying cross-lingual quantitative dency treebanks developed within the Univer- trends in the distribution of dependency sal Dependencies (UD) project4, thanks to the relations in treebanks for typologically cross-linguistically consistent syntactic annotation different languages. Preliminary results (Nivre, 2015), fostered the development of auto- show interesting differences rooted either matic strategies to extract cross-lingual similari- in language-specific peculiarities or cross- ties and differences in shared constructions from lingual annotation inconsistencies, with a corpora (Murawaki, 2017; Bjerva et al., 2019). potential impact on different application Within this line of research, the paper describes scenarios. 1 a methodology for comparing treebanks of typo- logically different languages with the final aim 1 Introduction and Motivation of detecting and quantifying similarities and dif- ferences in multilingual treebanks analyzed from The identification of cross-lingual quantitative a twofold perspective: language–specific pecu- trends in the distribution of dependency relations liarities vs cross–lingual annotation inconsisten- in “gold” treebanks is increasingly attracting the cies. To this end, we used LISCA (LInguiStically– interest of the computational linguistics commu- driven Selection of Correct Arcs) (Dell’Orletta et nity for different purposes, as testified e.g. by a re- al., 2013), an algorithm which has been success- cently published miscellaneous book on the quan- fully applied in different scenarios, against both titative analysis of dependency structures (Jiang the output of dependency parsers and gold tree- and Liu, 2018) or pilot initiatives such as the banks. In the first case, the score returned by first edition of the workshop “Quantitative Syn- LISCA was meant to identify unreliable automati- 2 tax 2019” . Among possible applications, it is cally produced dependency relations (Dell’Orletta worth mentioning studies aimed at acquiring ty- et al., 2013). When used against gold annota- pological evidence to be integrated in multilin- tions, LISCA was used to detect shades of syntac- gual NLP algorithms (see Ponti et al. (2018) for a tic markedness of syntactic constructions in man- survey and the workshop “Typology for Polyglot ually annotated corpora from a monolingual per- 3 NLP” ), or at detecting annotation inconsistencies spective (Tusa et al., 2016), or to acquire quantita- to improve the quality of treebanks (see (Dickin- tive typological evidence from a multilingual per- son, 2015; de Marneffe et al., 2017) to mention spective (Alzetta et al., 2018b). Last but not least, only a few). While the latter is a well-established it was also exploited to identify anomalous annota- research topic, although with still many open is- tions (going from annotation inconsistencies to er- sues, automatically acquiring typological informa- rors) from a monolingual perspective in gold tree- tion is still at its beginning, so automatic strate- banks (Alzetta et al., 2018a). gies to extract such information from corpora are The methodology exploited for the present work 1Copyright c 2020 for this paper by its authors. Use per- (described in Section 2) was tested in a case study mitted under Creative Commons License Attribution 4.0 In- carried out on four Indo-European languages ternational (CC BY 4.0). belonging to three different genera (according 2https://www.aclweb.org/anthology/W19-79.pdf 3https://typology-and-nlp.github.io/ 4https://universaldependencies.org/ to WALS classification, Dryer and Haspelmath d and h in the sentence. Global features, instead, (2013)): Bulgarian (Slavic, BUL), English (Ger- are aimed at locating each DR within the overall manic, ENG), Italian and Spanish (Romance, ITA sentence structure, and include e.g. the distance and SPA). UD treebanks constitute an ideal test of d from the root of the dependency tree or from bed for our analysis since, sharing the same an- the closest or most distant leaf node, and the num- notation scheme, allow the investigation of cross- ber of “brother” and “children” nodes of d, occur- lingual similarities and differences in shared con- ring respectively to its right or left in the linear structions. Besides similarities connected with sequence of of the sentence. In this case the UD annotation strategy aimed at maximising study, LISCA has been used in its delexicalized parallelism across languages, results in Section version in order to abstract away from variations 4 reflect shared possibly “universal” features of resulting from lexical effects, thus guaranteeing languages. Differences, in turn, can either re- cross-lingual comparability of results. The output flect typologically relevant language peculiarities of LISCA consists of the list of all DRs in the Tar- or highlight inconsistencies in the application of get Corpus ranked by decreasing score. the shared annotation scheme. The paper focuses The LISCA score is a context-sensitive and on both aspects. Section 5 concludes the paper frequency-based measure reflecting the degree discussing our findings and future directions of re- of similarity of the “linguistic environments” in search. which a given DR occurs in the Reference and Target corpora: it encodes the probability to ob- Contribution. The present contribution has two serve a DR instance occurring in a specific con- main goals: we aim to show how the methodol- text on the basis of the Statistical Model con- ogy can be used 1) to acquire quantitative evidence structed starting from the Reference Corpus. In of cross-linguistically shared properties, and 2) to more abstract terms, the LISCA score can be seen highlight divergences due either to language id- as reflecting the prototypicality degree of a spe- iosyncrasies or annotation inconsistencies across cific linguistic structure: whereas higher LISCA treebanks. scores identify DR instances appearing in “typi- 2 Method cal” (more frequent and likely) contexts with re- spect to the statistics acquired from the Refer- As shown in Figure 1, our methodology for ex- ence Corpus, lower scores identify less common or ploring multilingual treebanks is articulated in the even atypical DR instances of the Target Corpus. following two steps. From a multilingual perspective, the comparison I) LISCA Analysis. The LISCA algorithm op- of the ranked DRs lists obtained from corpora of erates in two steps: 1) it collects statistics about different languages can shed light on similarities a set of linguistically motivated features extracted and differences at linguistic and/or annotation lev- from an automatically dependency parsed corpus els. To carry out this comparative analysis, in this (referred to as Reference Corpus) to build a sta- study the ranked list of DRs has been split into 20 tistical model (SM) of the language; 2) it uses intervals of equal size, henceforth “bins” (plus a the obtained SM to assign a score to each depen- further bin for the remaining ones): the first bins dency relation (DR) instance, defined as a triple contain DRs presenting a high LISCA score and, d(ependent), h(ead), t(ype) of dependency linking conversely, the last bins contain DRs associated d to h, in a Target Corpus. Borrowing a metaphor with low LISCA scores. from Jakobson (1973), we can look at the SM as II) Ranking Exploration. We exploited CLaRK encoding the DNA of the language being analysed. system (Simov et al., 2004) to identify and com- Note, in fact, that the features considered by the pare quantitative trends from LISCA rankings. LISCA algorithm to build the SM cover, for each CLaRK system work–flow is the following: firstly, DR instance, a wide variety of factors, both local each Target Corpus is converted from the CoNLL- and global. Local features include e.g. the dis- U format5 into XML format, then the XPath lan- tance in terms of tokens between d and h, the asso- guage is used to select the nodes (sentences or to- ciative strength linking the grammatical categories kens) with the required properties. In this way we involved in the relation (i.e. POSd and POSh), the POS of the head governor, the type of dependency 5http://universaldependencies.org/ connecting d to h, and the relative linear order of format.html Figure 1: Method work–flow. can define different configurations and check the 4.1 Ranking of Dependencies distribution of the node characteristics along the As pointed out above, higher LISCA scores are as- DR rankings. signed to DRs that show a linguistic context highly typical for the language, whereas low scores are 3 Data associated with atypical (or simply less typical) For each language taken into account, two linguis- syntactic structures; (un)typicality is assessed here tically annotated corpora have been used: a large with respect to the statistics acquired from the Ref- Reference Corpus and a Target Corpus. erence Corpus. Each Reference Corpus consists of a monolin- As a first step of our comparative analysis, for gual corpus of texts from the news and Wikipedia each language we focused on the distribution of domains of around 40 million tokens, constitut- individual DRs across the 20 LISCA bins. Fig- ing a set of examples large enough to reflect ure 2 reports the median bin of occurrence for all the actual distribution of phenomena in the spe- 29 shared DRs in the ranking of each language. cific language. Reference corpora were morpho- The median bin was selected by sorting all in- syntactically annotated and dependency parsed by stances of a given DR on the basis of the asso- the UDPipe pipeline (Straka et al., 2016) trained ciated LISCA score and by identifying the me- on the Universal Dependency treebanks, version dian element of the ranked list: its bin of occur- 2.2 (Nivre et al., 2017). rence was taken as representative of the relation. Target corpora correspond here to manually Top and bottom relations (respectively at the ex- validated (“gold”) Universal Dependencies tree- treme left and right in Fig.2 graph) in language- banks (v2.2). Specifically, we considered the specific rankings show interesting similarities: if following UD treebanks: on the one hand DRs involving function words case det aux(:pass) i) English Web (254,830 tokens and (e.g. , , ) are associated 16,622 sentences) (Silveira et al., 2014); with higher LISCA scores for all languages, on ii) Italian Stanford Dependency Treebank the other hand special or “loose” DRs such as orphan parataxis (278,429 tokens and 14,167 sentences) (Bosco et and or clausal subjects and csubj(:pass) advcl al., 2013); adverbial clauses ( , ) all iii) Spanish UD treebank (547,680 tokens and occur in the last bins, representing relations with 17,680 sentences) (Alonso and Zeman, 2016); more variable contexts across all languages. An- iv) UD_Bulgarian-BTB (156,149 tokens and other cross-language parallelism concerns the rel- 11,138 sentences) (Simov et al., 2005). ative rankings of subsets of DRs: clausal com- plements with obligatory control (xcomp) are as- 4 Results signed a higher score with respect to the wider class of clausal complements without it (ccomp); Results are analysed from a twofold perspective, the direct object relation (obj) precedes in the focusing on the distribution across the bins of dif- ranking the oblique argument/modifier (obl); and ferent DR types and structures. the nominal subject (nsubj) always precedes its Figure 2: Median occurrence in LISCA bins of shared DRs across languages.

Bins 1 to 10 Bins 11 to 20 DR length BUL ENG ITA SPA BUL ENG ITA SPA 1 66.42 55.80 65.79 69.16 38.96 37.62 36.97 37.34 2 20.26 23.31 24.93 21.62 16.69 17.93 17.05 17.47 3-4 6.45 12.24 5.16 5.21 19.45 21.11 15.85 16.31 5-10 4.75 4.72 2.71 2.01 14.34 14.09 13.05 13.34 ≥10 2.39 3.93 1.41 1.98 10.56 9.25 17.08 15.54 # DRs 64,885 110,184 141,389 248,794 22,510 42,475 44,310 96,781

Table 1: Percentage distribution by length of DRs involving leaves in the first and in last 10 LISCA bins. For each group of bins the number of all DRs involving leaves is given. clausal counterpart (csubj). It is interesting to terminers, e.g. demonstratives. Here are two ex- report that the frequency of a DR seems to plays a amples for Bulgarian where the first one shows minor role in determining the position of a given the usage of the morphologically expressed post- DR in the LISCA ranking: consider, for instance, positioned definite article (thus no explicit (det) the punct relation which is a highly frequent relation) while the second shows the usage of a DR (covering around 11% of DRs in all four lan- demonstrative pronoun (marked with (det) re- guages), but nevertheless it was placed in the mid- lation)): (1) (‘Жената влезе в стаята’) (lit. dle part of the ranking for all languages. Looked Woman-the entered room-the) and (2) (‘Тази же- at from this perspective, the LISCA ranking of re- на влезе в стаята’) (lit. This woman entered lations - which is heavily influenced by the prin- room-the). The frequency of the examples type ciples underlying the UD annotation schema - (1) in the treebank is about 10 times bigger than seems to reflect the complexity of rela- the frequency of the examples of type (2). Thus, tions (Alzetta et al., 2020), where more complex the nsubj nodes modified by explicit determiner to parse DRs are characterised by a higher vari- is a rare case in Bulgarian treebank. ability in their contexts of occurrence. With respect to b), there are interesting exam- Some interesting differences can also be re- ples, even among core UD DRs: this is the case of ported, originating either in a) language-specific indirect objects (iobj), whose annotation criteria peculiarities or b) possibly inconsistent annota- highly diverge across languages. The sources of tions across languages. Concerning a), ENG nom- dissimilarities might come partially from the an- inal subjects (nsubj, nsubj:pass) are ranked notation specifications per language about what a significantly higher with respect to the other three second argument (iobj) vs an adjunct (obl) is. languages, all sharing the pro-drop and free word If a closer look is taken into the data, it turns out order properties; or determiners (det) show the that in ITA and ENG the iobj is typically ex- same distribution for SPA, ENG and ITA in con- pressed by a PRON(oun), as in these two exam- trast to BUL, where the definite article is post- ples: ITA: ‘ti (PRON) ho dato’ (lit. ‘I gave you’); positioned and expressed morphologically, with ENG: ‘causing us (PRON) truble’. In ITA this the exception of some pronouns functioning as de- represents 100% of the cases, while in ENG 84%, whereas in SPA and BUL this relation is expressed 91.76% for Spanish. Interestingly, the first 6, 6, by a pronoun in only 46.7% and 19% of the cases 8 and 4 bins respectively for Bulgarian, English, respectively. In Spanish, for example, the iobj Italian and Spanish contain exclusively leaves. In relation is used also for NOUNs: in the Span- other words, leaves are typically associated with ish example ‘Obligaron al Gobierno (NOUN) a higher LISCA scores: due to their smaller con- comprar creditos’ (lit. Forced the Government to text, they are characterised by higher processing buy credits) the noun is annotated as indirect ob- reliability. This is in line with the fact that DRs in- ject of obligaron, whereas in Italian the construc- volving functional words, e.g. case, det, aux, tion ‘Non ho dato soldi al presidente (NOUN)’ etc. typically occur in the first bins (see Figure (lit. I didn’t give money to the president) the 2). On the contrary, the last 10 bins of all lan- noun is marked as obl relation. In Bulgarian guages mostly contain DRs not involving leaves the iobj relation is used not only for marking (68.28% BUL, 63.54% EN, 69.33% ITA, 64.54% the dative pronouns, but also for marking head SP). For what concerns the leaves in the second NOUNs in PPs. The prevalence of this relation half of the bins, they turned out to be typically in- on NOUNs is due to the following factors: (1) volved in particularly complex syntactic contexts, the existence of long dative counterparts to short such as long distance dependencies or occurring in dative pronouns that consist of a preposition and constructions that are not typical for that relation. a noun (‘Майката даде играчка на детето’) (prep NOUN) (lit. Mother-the gave toy to child- 5 Conclusion the-DAT); and (2) the marking of indirect com- In this paper we presented method for studying the plements as indirect objects, while the obl rela- distribution of DRs in gold treebanks which was tion has been reserved for adjuncts (‘Те продъл- tested in a case study carried out on four languages жават да участват в лотарията’) (non-dative belonging to three different genera. The cross- prep NOUN) (lit. They continue to participate in lingual comparison of the LISCA-based ranking lottery-the). This suggests that different annota- of UD relations across the bins shows: on the one tion criteria guide the assignment of the iobj DR, hand, shared (possibly universal) trends, concern- possibly not all of them originating in peculiarities ing e.g. the similar distribution of dependencies of the language. involving leaves or of long distance dependencies, Other interesting examples concern the annota- which are respectively concentrated at the top and tion of multi-word expressions and proper names at the bottom of the LISCA ranking for each lan- (fixed and flat), which are treated differ- guage; on the other hand, recorded differences in ently across languages. For example, in BUL all the ranking of relations can be explained in terms grammatically fixed multi-words, such as complex of either language peculiarities (e.g. the pro-drop prepositions (like с оглед на ‘with regard to’) or property of BUL-ITA-SPA vs ENG, or the sur- conjunctions (like за да ‘in order to’), are treated face realisation of definite determiners in BUL vs as fixed while in Italian the annotation reflects ENG-ITA-SPA) or potential inconsistencies in the the underlying syntactic structure, as in the case application of the UD annotation scheme (see the of, e.g., ‘a base di’ (lit. made of ) and ‘in relazione case of the indirect object relation). Both types of a’ (lit. in relation to). results play a potentially key role in different sce- narios, going from typology-driven multilingual 4.2 Distribution of Leaves NLP to the improvement of the cross-lingual con- sistency of treebanks. For each language, we investigated the distribu- tion of DRs across the LISCA bins focusing on Acknowledgements DRs involving leaves as dependants (henceforth leaves), as opposed to DRs without leave nodes Thanks to the anonymous reviewers for their help- (henceforth non-leaves). Results of this analy- ful comments. This work was partially supported sis are reported in Table 1. Despite minor differ- by the Bulgarian National Interdisciplinary Re- ences, all languages share a similar trend: leaves search e-Infrastructure for Resources and Tech- are mostly ranked in the first 10 bins represent- nologies in favor of the Bulgarian Language and ing for Bulgarian 91.52% of the DRs occurring in Cultural Heritage, part of the EU infrastructures them, 95.56% for English, 98,27% for Italian and CLARIN and DARIAH – CLaDA-BG, Grant num- ber DO01-272/16.12.2019. F. Dell’Orletta, G. Venturi, and S. Montemagni. 2013. Linguistically-driven selection of correct arcs for de- pendency parsing. Computaciòn y Sistemas, 2:125– References 136. H. M. Alonso and D. Zeman. 2016. Universal depen- M. Dickinson. 2015. Detection of Annotation Errors dencies for the ancora treebanks. Procesamiento del in Corpora. Language and Linguistics Compass, Lenguaje Natural, 57:91–98. 9(3):119–138. C. Alzetta, F. Dell’Orletta, S. Montemagni, and G. Ven- Matthew S. Dryer and Martin Haspelmath, editors. turi. 2018a. Dangerous relations in dependency 2013. WALS Online. Max Planck Institute for Evo- treebanks. In Proceedings of the 16th Interna- lutionary Anthropology, Leipzig. tional Workshop on Treebanks and Linguistic The- ories (TLT16), pages 201–210, Prague, Czech Re- Roman Jakobson. 1973. Essais de linguistique public, January. générale t. 2: rapports internes et externes du lan- gage. Les éditions de Minuit. C. Alzetta, F. Dell’Orletta, S. Montemagni, and G. Ven- turi. 2018b. Universal dependencies and quanti- J. Jiang and H. Liu. 2018. Quantitative Analysis of De- tative typological trends. a case study on word or- pendency Structures. De Gruyter Mouton, Berlin, der. In Proceedings of the 11th Edition of Interna- Boston. tional Conference on Language Resources and Eval- uation (LREC 2018), pages 4540–4549. Association Yugo Murawaki. 2017. Diachrony-aware induction of for Computational Linguistics. binary latent representations from typological fea- tures. In Proceedings of the Eighth International C. Alzetta, F. Dell’Orletta, S. Montemagni, and G. Ven- Joint Conference on Natural Language Processing turi. 2020. Uncovering typological context- (Volume 1: Long Papers), pages 451–461. sensitive features. In Proceedings of the Second Workshop on Typology for Polyglot Natural Lan- J. Nivre, A. Željko, A. Lars, and et alii. 2017. Univer- guage Processing. sal dependencies 2.0. In LINDAT/CLARIN digital library at the Institute of Formal and Applied Lin- Johannes Bjerva and Isabelle Augenstein. 2018. From guistics, Charles University. phonology to syntax: Unsupervised linguistic typol- ogy at different levels with language embeddings. J. Nivre. 2015. Towards a universal grammar for natu- In Proceedings of the 2018 Conference of the North ral language processing. In Computational Linguis- American Chapter of the Association for Computa- tics and Intelligent - Proceedings of tional Linguistics: Human Language Technologies, the 16th International Conference, CICLing 2015, Volume 1 (Long Papers), pages 907–916. Part I, pages 3–16, Cairo, Egypt, April. Johannes Bjerva, Yova Kementchedjhieva, Ryan Cot- E.M. Ponti, H. O’Horan, Y. Berzak, I. Vulic,´ R. Re- terell, and Isabelle Augenstein. 2019. A proba- ichart, T. Poibeau, E. Shutova, and A. Korho- bilistic generative model of linguistic typology. In nen. 2018. Modeling language variation and Proceedings of the 2019 Conference of the North universals: A survey on typological linguistics American Chapter of the Association for Computa- for natural language processing. arXiv preprint tional Linguistics: Human Language Technologies, arXiv:1807.00914. Volume 1 (Long and Short Papers), pages 1529– 1540, Minneapolis, Minnesota, June. Association N. Silveira, T. Dozat, M.C. de Marneffe, S. Bowman, for Computational Linguistics. M. Connor, J. Bauer, and C.D. Manning. 2014. A gold standard dependency corpus for english. In C. Bosco, S. Montemagni, and M. Simi. 2013. Con- Proceedings of the Ninth International Conference verting italian treebanks: Towards an italian stanford on Language Resources and Evaluation (LREC). dependency treebank. In Proceedings of the ACL Linguistic Annotation Workshop & Interoperability K. Simov, A. Simov, H. Ganev, K. Ivanova, and I. Grig- with Discourse, Sofia, Bulgaria, August. orov. 2004. The CLaRK System: XML-based Cor- pora Development System for Rapid Prototyping. Ryan Cotterell and Jason Eisner. 2017. Probabilistic Proceedings of LREC 2004, pages 235–238. typology: Deep generative models of vowel invento- ries. In Proceedings of the 55th Annual Meeting of K. Simov, P. Osenova, A. Simov, and M. Kouylekov. the Association for Computational Linguistics (Vol- 2005. Design and Implementation of the Bulgar- ume 1: Long Papers), pages 1182–1192. ian HPSG-based Treebank. Journal of Research on Language and Computation. Special Issue, pages M.C. de Marneffe, M. Grioni, J. Kanerva, and F. Gin- 495–522. ter. 2017. Assessing the Annotation Consistency of the Universal Dependencies Corpora. In Proceed- M. Straka, J. Hajic, and J. Strakova. 2016. UD- ings of the 4th International Conference on Depen- Pipe: Trainable pipeline for processing CoNLL-U dency Linguistics (Depling 2007), pages 108–115, files performing tokenization, morphological anal- Pisa, Italy, September. ysis, pos tagging and parsing. In Proceedings of the Tenth International Conference on Language Re- sources and Evaluation (LREC). E. Tusa, F. Dell’Orletta, S. Montemagni, and G. Ven- turi. 2016. Dieci sfumature di marcatezza sintat- tica: verso una nozione computazionale di comp- lessitá. In Proceedings of the Third Italian Confer- ence on Computational Linguistics (CLiC-it), pages 3–16, Napoli, Italy, December.