Emerging Topics at the Third Italian Conference on Computational Linguistics and EVALITA 2016

IJCoL Italian Journal of Computational Linguistics

3-1 | 2017 Emerging Topics at the Third Italian Conference on Computational Linguistics and EVALITA 2016

Electronic version URL: http://journals.openedition.org/ijcol/411 DOI: 10.4000/ijcol.411 ISSN: 2499-4553

Publisher Accademia University Press

Electronic reference IJCoL, 3-1 | 2017, “Emerging Topics at the Third Italian Conference on Computational Linguistics and EVALITA 2016” [Online], Online since 01 June 2017, connection on 28 January 2021. URL: http:// journals.openedition.org/ijcol/411; DOI: https://doi.org/10.4000/ijcol.411

IJCoL is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License editors in chief

Roberto Basili Università degli Studi di Roma Tor Vergata Simonetta Montemagni Istituto di Linguistica Computazionale “Antonio Zampolli” - CNR

advisory board

Giuseppe Attardi Università degli Studi di Pisa (Italy) Nicoletta Calzolari Istituto di Linguistica Computazionale “Antonio Zampolli” - CNR (Italy) Nick Campbell Trinity College Dublin (Ireland) Piero Cosi Istituto di Scienze e Tecnologie della Cognizione - CNR (Italy) Giacomo Ferrari Università degli Studi del Piemonte Orientale (Italy) Eduard Hovy Carnegie Mellon University (USA) Paola Merlo Université de Genève (Switzerland) John Nerbonne University of Groningen (The Netherlands) Joakim Nivre Uppsala University (Sweden) Maria Teresa Pazienza Università degli Studi di Roma Tor Vergata (Italy) Hinrich Schütze University of Munich (Germany) Marc Steedman University of Edinburgh (United Kingdom) Oliviero Stock Fondazione Bruno Kessler, Trento (Italy) Jun-ichi Tsujii Artificial Intelligence Research Center, Tokyo (Japan) editorial board

Cristina Bosco Università degli Studi di Torino (Italy) Franco Cutugno Università degli Studi di Napoli (Italy) Felice Dell’Orletta Istituto di Linguistica Computazionale “Antonio Zampolli” - CNR (Italy) Rodolfo Delmonte Università degli Studi di Venezia (Italy) Marcello Federico Fondazione Bruno Kessler, Trento (Italy) Alessandro Lenci Università degli Studi di Pisa (Italy) Bernardo Magnini Fondazione Bruno Kessler, Trento (Italy) Johanna Monti Università degli Studi di Sassari (Italy) Alessandro Moschitti Università degli Studi di Trento (Italy) Roberto Navigli Università degli Studi di Roma “La Sapienza” (Italy) Malvina Nissim University of Groningen (The Netherlands) Roberto Pieraccini Jibo, Inc., Redwood City, CA, and Boston, MA (USA) Vito Pirrelli Istituto di Linguistica Computazionale “Antonio Zampolli” - CNR (Italy) Giorgio Satta Università degli Studi di Padova (Italy) Gianni Semeraro Università degli Studi di Bari (Italy) Carlo Strapparava Fondazione Bruno Kessler, Trento (Italy) Fabio Tamburini Università degli Studi di Bologna (Italy) Paola Velardi Università degli Studi di Roma “La Sapienza” (Italy) Guido Vetere Centro Studi Avanzati IBM Italia (Italy) Fabio Massimo Zanzotto Università degli Studi di Roma Tor Vergata (Italy)

editorial office Danilo Croce Università degli Studi di Roma Tor Vergata Sara Goggi Istituto di Linguistica Computazionale “Antonio Zampolli” - CNR Manuela Speranza Fondazione Bruno Kessler, Trento AILC IDENTITY - CMYK Registrazione presso il Tribunale di Trento n. 14/16 del 6 luglio 2016

C:100 M:0 Y:100 K:0

direttore responsabile One-color version Red Michele Arnese

Pubblicazione resa disponibile nei termini della licenza Creative Commons Attribuzione – Non commerciale – Non opere derivate 4.0

C:0 M:100 Y:100 K:0

Dark background version

(pick only the design elements)

www.sarabarcena.com isbn 978-88-99982-64-5

Accademia University Press via Carlo Alberto 55 I-10123 Torino [email protected] www.aAccademia.it/IJCoL_3_1

aAccademia university press

Accademia University Press è un marchio registrato di proprietà di LEXIS Compagnia Editoriale in Torino srl IJCoL Volume 3, Number 1 june 2017

Emerging Topics at the Third Italian Conference on Computational Linguistics and EVALITA 2016

CONTENTS Nota editoriale Roberto Basili, Simonetta Montemagni 7

Panta rei: Tracking Semantic Change with Distributional Semantics in Ancient Greek Martina A. Rodda, Marco S. G. Senaldi, Alessandro Lenci 11

Distributed Representations of Lexical Sets and Prototypes in Causal Alternation Verbs Edoardo Maria Ponti, Elisabetta Jezek, Bernardo Magnini 25

Determining the Compositionality of Noun-Adjective Pairs with Lexical Variants and Distributional Semantics Marco S. G. Senaldi, Gianluca E. Lebani, Alessandro Lenci 43

LU4R: adaptive spoken Language Understanding For Robots Andrea Vanzo, Danilo Croce, Roberto Basili, Daniele Nardi 59

For a performance-oriented notion of regularity in inflection: the case of Modern Greek conjugation Stavros Bompolas, Marcello Ferro, Claudia Marzi, Franco Alberto Cardillo, Vito Pirrelli 77

EVALITA Goes Social: Tasks, Data, and Community at the 2016 Edition Pierpaolo Basile, Francesco Cutugno, Malvina Nissim, Viviana Patti, Rachele Sprugnoli 93

Nota Editoriale

Roberto Basili∗ Simonetta Montemagni∗∗ Università di Roma, Tor Vergata ILC–CNR, Pisa

Eccoci al quarto numero dell’Italian Journal of Computational Linguistics (IJCoL), la Ri- vista Italiana di Linguistica Computazionale edita dall’ “Associazione Italiana di Lingui- stica Computazionale” (AILC - www.ai-lc.it). La rivista, al suo terzo anno di pubblicazione, si sta affermando come importante occasione per la promozione e la dif- fusione della ricerca in linguistica computazionale condotta all’interno della comunità nazionale da prospettive diverse e complementari, che integrano in un rapporto dialet- tico i punti di vista umanistico e matematico–formale, teorico e applicato. Il ruolo sempre più centrale che il linguaggio svolge nei processi di comunicazione odierna è quotidianamente certificato da riflessioni e annunci che anche i media tradizionali hanno fatto propri. La natura essenzialmente linguistica della comunicazione nelle reti sociali on–line, così come il vantaggio competitivo che oggi l’industria associa alla capacità computazionale di trattare il linguaggio in volumi crescenti di dati confermano l’importanza strategica della ricerca in questa area che costituisce in prospettiva un fecondo terreno di innovazione sociale, industriale ed economica. I temi che ruotano attorno a linguaggio e computazione forniscono opportunità uniche per comprendere da un lato i modi con cui le macchine possono elaborare le produzioni linguistiche (scritte e orali) e definire processi avanzati di tipo applicativo, e dall’altro contribuire a una sempre migliore comprensione dei modi con cui il linguaggio opera e cambia nel tempo, nello spazio e attraverso diversi canali e mezzi di comunicazione. I contributi di questo volume ben si collegano a queste due dimensioni di ricerca, in linea con lo spirito della rivista che intende proporsi come forum in cui le diverse anime della linguistica computazionale dialogano e si confrontano. Seguendo la tradizione dei primi due numeri, questo è un volume miscellaneo che raccoglie lavori di ricerca ispirati da giovani ricercatori che sono emersi nell’ambito della Conferenza CLiC-it 2016, tenutasi a Napoli il 5 e 6 dicembre 2016, come parti- colarmente promettenti nel panorama della linguistica computazionale italiana. Questi contributi, selezionati tra le diverse aree tematiche della conferenza, testimoniano linee di ricerca originali e innovative della comunità italiana, e in modo particolare dei suoi più giovani protagonisti. Gli articoli sono stati selezionati attraverso un processo iterativo di peer–review. Ogni articolo è stato sottoposto a tre valutazioni da parte di comitati diversi: come contributo alla conferenza; come candidato ai premi di “Best Young Paper” e “Distinguished Young Paper” di CLiC-it 2016; infine, nella versione estesa, come articolo di rivista scientifica. A questi articoli si aggiunge un contributo invitato che propone una rassegna e una riflessione critica sull’esperienza di EVALITA 2016, la campagna di valutazione delle tecnologie del linguaggio per la lingua italiana

Dept. of Enterprise Engineering - Via del Politecnico 1, 00133 Rome. ∗ E-mail: [email protected] Istituto di Linguistica Computazionale “Antonio Zampolli” (ILC-CNR) - Via Moruzzi 1, 56124, Pisa. ∗∗ E-mail: [email protected]

© 2017 Associazione Italiana di Linguistica Computazionale Italian Journal of Computational Linguistics Volume 3, Number 1 scritta e parlata che, come da tradizione, contribuisce a fare il punto sull’efficacia e la qualità dei metodi di analisi della lingua italiana all’interno di un ricco e vario repertorio di task applicativi. Anche nell’edizione del 2016, EVALITA ha costituito un raccordo essenziale tra la fase empirica e applicativa della ricerca e la riflessione metodologica e teorica. I contributi del volume si articolano in due macro–sezioni: la prima raccoglie contributi di ricerca originali e innovativi, la seconda fornisce un resoconto della campagna di valutazione EVALITA 2016 delineandone al contempo linee di sviluppo per il futuro. All’interno della prima macro–sezione, i primi quattro contributi sono accomunati dall’utilizzo di modelli di semantica distribuzionale: è interessante notare che nel 2016 Distributional Semantics è risultata essere la parola chiave usata più frequentemente dagli autori nella caratterizzazione dei propri contributi a CLiC-it. Non meraviglia quindi che 4 dei 5 contributi selezionati siano riconducibili a questo paradigma di ricerca. Essi differiscono a vari livelli, che vanno dalle finalità della ricerca e le lingue trattate, ai modelli delle distribuzioni statistiche delle parole nei corpora, alle proprietà semantiche considerate e alla caratterizzazione dei contesti linguistici usati nella determinazione degli spazi semantici. In questa sede ci limitiamo a segnalare i diversi scenari – sia teorici sia applicativi – all’interno dei quali tali tecniche sono state utilizzate. Il lavoro di Rodda, Senaldi e Lenci propone un metodo per lo studio del cam- biamento semantico a partire dalla variazione degli spazi semantici distribuzionali, dimostrando il potenziale contributo della semantica distribuzionale in una prospettiva diacronica. Il metodo è stato sperimentato all’interno di uno studio finalizzato all’identificazione delle aree coinvolte da cambiamenti semantici nel lessico del greco antico tra l’epoca precristiana e cristiana. L’articolo di Ponti, Jezec e Magnini utilizza tecniche di semantica distribuzionale per esplorare questioni aperte in ambito linguistico sull’interfaccia tra sintassi e semantica, riguardanti le proprietà lessico–semantiche degli argomenti verbali. Lo studio, condotto sulla classe dei verbi ad alternanza causativa–incoativa, ha portato alla luce importanti differenze nelle proprietà degli argomenti che rappresentano categorie non uniformi, la cui distribuzione attorno a un prototipo varia in misura significativa tra diverse posizioni argomentali, suggerendo l’esistenza di diverse restrizioni di selezione. Nel contributo di Senaldi, Lebani e Lenci, la similarità distribuzionale tra il vettore di una data espressione e il vettore delle sue varianti lessicali è alla base di una serie di indici finalizzati a discriminare tra espressioni idiomatiche e composizionali: viene dimostrato che gli indici proposti presentano una maggiore efficacia rispetto a quanto proposto nella letteratura distribuzionale sulla composizionalità. Diversamente dai precedenti contributi che mostrano come modelli di semantica distribuzionale contribuiscano significativamente a fare luce su questioni aperte della linguistica teorica, Vanzo, Croce, Basili e Nardi utilizzano la semantica distribuzionale all’interno di uno scenario applicativo di Human-Robot Interaction. In particolare, dimostrano come essa svolga un ruolo rilevante nell’apprendimento strutturato di un processo di Semantic Role Labeling per interfacce robotiche. Chiude la macro–sezione il contributo di Bompolas, Ferro, Marzi, Cardillo e Pirrelli che mostra ruolo e impatto derivante dall’utilizzo di approcci basati sulla nozione di paradigma sull’analisi e sull’apprendimento delle parole. Con esperimenti condotti su tre lingue (greco moderno, italiano e tedesco) gli autori dimostrano che diverse classi verbali sono apprese in funzione del loro grado di trasparenza e predicibilità. Tali risultati, in linea con evidenza psicolinguistica, contribuiscono significativamente a rafforzare l’ipotesi del lessico mentale come sistema emergente integrato.

8 Basili et al. Nota Editoriale

Il volume si chiude con il contributo invitato degli organizzatori della quinta edizione della campagna di valutazione EVALITA. Nel 2016 sono stati organizzati 6 shared tasks, tra cui alcuni nuovi, e una competizione sponsorizzata dall’IBM, che hanno attratto globalmente 34 partecipanti. Una delle novità di questa edizione è rappresentata dal focus sulla lingua dei social media, che ha portato alla creazione di una risorsa con annotazioni multi–livello (PoS tags, sentiment information, named entities and linking, e factuality information) che è stata condivisa tra diversi tasks, permettendone anche una maggiore compenetrazione. Sul versante dei metodi e delle tecniche alla base dei sistemi che hanno partecipato a EVALITA 2016, è interessante segnalare che Deep Learning è risultata la parola chiave usata più frequentemente dagli autori nella caratterizzazione dei propri contributi, mostrando anche la complementarità, per il 2016, di EVALITA rispetto a CLiC-it. A partire dall’analisi dell’edizione 2016, gli organizzatori di EVALITA concludono delineando interessanti prospettive di sviluppo della campagna di valutazione per gli anni a venire. Al lettore dunque il piacere di approfondire i temi e gli stimoli che la rivista – anche in questo numero – continua a raccogliere.

Editorial Note Summary

We are pleased to announce the fourth issue of the Italian Journal of Computational Linguistics (IJCoL), published by the Italian Association of Computational Linguistics (AILC, www.ai-lc.it). The journal, in its third year of publication, is becoming an important opportunity for promoting and disseminating research results achieved by the Italian computational linguistics community from different and complementary – e.g. humanistic vs. computational or theoretical vs. applied – perspectives. The increasingly central role that language plays in today’s communication pro- cesses is daily acknowledged. The essentially linguistic nature of communication in social networks, as well as the competitive advantage that industry today associates with the computational capacity to handle language in big data confirm – together – the strategic importance of the research in this area, creating a fertile ground for social, industrial and economic innovation. The topics revolving around language and computation provide unique opportunities, on the one hand, to understand the ways in which machines can process language productions (both written and oral) and to define advanced applications, and on the other hand to contribute to a deeper understanding of the ways in which language works and changes over time, space and through different channels and means of communication. Contributions to this volume are well linked to these two dimensions of research, in line with the spirit of the journal that presents itself as a forum where the different souls of computational linguistics confront each other and are combined. As in the case of the first two IJCoL issues, this is a miscellaneous volume that collects research work inspired by young researchers who emerged as particularly promising within the CLiC-it 2016 Conference, held in Naples on 5–6 December 2016. These contributions testify original and innovative research lines of the Italian computational linguistics community, and in particular of its youngest protagonists. The articles were evaluated through an iterative peer–review process carried out by different committees: as a contribution to the CLiC-it conference; as a candidate for the CLiC- it 2016 “Best Young Paper” and “Distinguished Young Paper” awards; finally, in its extended version, as a scientific journal article. This issue also includes an invited contribution by the organizers of the EVALITA 2016 evaluation campaign.

9 Italian Journal of Computational Linguistics Volume 3, Number 1

The contributions in this issue are organized into two macro–sections, with the first one collecting original and innovative research contributions, and the second one providing an overview of EVALITA 2016 and outlining further developments. The first four papers illustrate different applications of distributional semantic models: it is interesting to note that in 2016 Distributional Semantics was the keyword most frequently used by CLiC-it authors for characterizing their contribution. They differ at different levels, ranging from the research goal and the language(s) dealt with to the computational techniques used to model word co-occurrence statistics, the semantic properties taken into account and the contexts used in determining semantic spaces. The work by Rodda, Senaldi and Lenci proposes an innovative method for studying semantic change starting from the variation of distributional semantic spaces, thus demonstrating the potential contribution of distributional semantics to diachronic studies. The method was tested in a case study aimed at identifying the areas of semantic change in the ancient Greek lexicon between the pre-Christian and Christian era. The paper by Ponti, Jezec and Magnini uses distributional semantics to investigate open linguistic issues on the syntax–semantics interface, with particular emphasis on the lexical-semantic properties of verbal arguments. The study, carried out on the class of causative–inchoative verbs, has brought to light important differences in the properties of arguments. In the contribution of Senaldi, Lebani and Lenci, distributional similarity between the vector of a given expression and the vector of its lexical variants is used as the basis of a set of indices aimed at discriminating between idiomatic vs. compositional expressions, which turned out to be more effective than those proposed so far in the distributional literature on compositionality. Unlike the previous contributions showing how distributional semantics models can significantly contribute to shed light on open theoretical linguistics issues, Vanzo, Croce, Basili and Nardi use distributional semantic models within an application scenario: Human–Robot Interaction. In particular, they show the beneficial impact of distributional semantic lexicons on the structured learning of a Semantic Role Labeling component in robotic interfaces. This section is closed by the paper by Bompolas, Ferro, Marzi, Cardillo and Pirrelli, who demonstrate role and impact of paradigm–based approaches for word processing and learning. The results of a case study simulating the acquisition of Modern Greek conjugation, compared with evidence from German and Italian, support a view of the mental lexicon as an emergent integrative system. The volume closes with the invited contribution of the organizers of the fifth edition of the EVALITA evaluation campaign. In 2016, 6 shared tasks were organized together with a challenge sponsored by IBM, which globally attracted 34 participants. One of the novelties of this edition is the focus of social media language, which led to the creation of a resource with multi-level annotations (PoS tags, sentiment information, named entities and linking, and factuality information) that has been used across different tasks. For what concerns methods and techniques underlying EVALITA 2016 participant systems, it is worth reporting that Deep Learning was the keyword most frequently used by the authors in characterizing their system; this also demonstrates the complementar- ity of EVALITA with respect to CLiC–it in 2016. EVALITA’s organizers conclude their paper by outlining interesting lines of development for future EVALITA campaigns. The synthetic view provided above does not exhaust the wide range of topics touched by the papers in this issue; this leaves the reader the pleasure to discover the themes and stimuli that the journal continues to collect – even in this issue.

10 Panta rei: Tracking Semantic Change with Distributional Semantics in Ancient Greek

Martina A. Rodda ∗ Marco S. G. Senaldi ∗∗ Scuola Normale Superiore di Pisa Scuola Normale Superiore di Pisa

Alessandro Lenci † Università di Pisa

We present a method to explore semantic change as a function of variation in distributional semantic spaces. In this paper, we apply this approach to automatically identify the areas of semantic change in the lexicon of Ancient Greek between the pre-Christian and Christian era. Distributional Semantic Models are used to identify meaningful clusters and patterns of semantic shift within a set of target words, deﬁned through a purely data-driven approach. The results emphasize the role played by the diffusion of Christianity and by technical languages in determining semantic change in Ancient Greek and show the potentialities of distributional models in diachronic semantics.

1. Introduction

Distributional Semantics is grounded on the assumption that the meaning of a word can be described as a function of its collocates in a corpus. This suggests that diachronic meaning shifts can be traced through changes in the distribution of these collocates over time (Sagi, Kaufmann, and Clark 2011). While some studies focused on testing the explanatory power of this method over frequency- and syntax-based approaches (Wijaya and Yeniterzi 2011; Kulkarni et al. 2015), more advanced contributions to the ﬁeld explored how distributional models can be used to test competing hypotheses about semantic change (Xu and Kemp 2015), or to investigate the productivity of constructions in diachrony (Perek 2016). The results attest the explanatory power of distributional methods in modeling diachronic shifts in meaning. In this paper, we propose a method to identify semantic change through the Representational Similarity Analysis (RSA) (Kriegeskorte and Kievit 2013) of distributional vector spaces built from diachronic corpora. RSA is a method extensively used in neuroscience to test cognitive and computational models by comparing the geometry of their representation spaces (Edelman 1998). Stimuli are represented with a representational dissimilarity matrix that contains a measure of the dissimilarity relations of the stimuli with each other. Different matrices are compared to evaluate the correspondence of the representational spaces built from different sources (e.g., behavioral and neuroimaging data). We argue that this method can be applied to compare distributional representations of the lexicon at different temporal stages. The hypothesis is that the elements in the lexical spaces

Scuola Normale Superiore - Piazza dei Cavalieri 7, 56126 Pisa, Italy. E-mail: [email protected] ∗ Scuola Normale Superiore - Piazza dei Cavalieri 7, 56126 Pisa, Italy. E-mail: [email protected] ∗∗ CoLing Lab, University of Pisa - Via S. Maria 36, 56126 Pisa, Italy . † E-mail: [email protected]

© 2017 Associazione Italiana di Linguistica Computazionale Italian Journal of Computational Linguistics Volume 3, Number 1 showing larger geometrical variations in time correspond to the lexical areas that underwent major semantic changes. To the best of our knowledge, this is the first time RSA is used in diachronic distributional semantics. Here we present a case study that applies RSA to track patterns of semantic change within the lexicon of Ancient Greek. We focus on the first few centuries AD, when the rise of Christianity caused a deep and widespread cultural shift within the Hellenic world. We predict that this shift will be reflected in the Greek lexicon of the time. In addition to past studies (Boschetti 2009; O‘Donnell 2005), we apply a bottom-up approach to the detection of semantic change, with no prior definition of a list of lemmas to be analyzed. The goal is to develop a quantitative “discovery procedure” to detect lexical semantic changes, enabling the researcher to discuss and interpret any meaningful patterns that may arise. From a methodological standpoint, this study aims to show how Distributional Semantics can be applied fruitfully to such a small and literary corpus as the collection of Ancient Greek texts. The results will also highlight the ways in which Distributional Semantics can complement the intuition of the researcher in analyzing semantic change in Ancient Greek, providing a useful tool for future studies in Classics. A distributional approach seems particularly suited to philological research, as it is already common and intuitive for researchers in this field to determine the exact meaning, usage restrictions, and stylistic connotations of a word by analyzing the context in which it occurs, especially when no other sources (such as ancient lexica) are available. Distributional Semantics provides the tools to perform a similar task not just on a much larger scale, but drawing information from the whole corpus; as such, it has the potential to highlight patterns in semantic change that would not otherwise be noticeable.

2. Related Work

The past few years have seen the rise of a series of studies tackling diachronic semantic change via computational methods. As pointed out by Sagi, Kaufmann, and Clark (2011), the increasing availability of computational tools for analyzing and manipulat- ing large data sets and corpora allows for testing hypotheses and detecting statistical trends in a large-scale perspective that does not hinge on the intuitions of the linguist or the philologist. Crucially, most of this research has relied on a diachronic application of the distributional hypothesis (Harris 1954) by modeling semantic shift as a change of the co-occurrence patterns of a given lemma over time. In Sagi, Kaufmann, and Clark (2011)’s proposal, the semantic narrowing or broad- ening of English words in the 1150-1710 period is modeled as an increase or decrease in density of the vector space populated by all the token occurrences of a given word in the various decades. The mean cosine similarity between all the token vectors of dog, for instance, decreases over time since it shifts from denoting a speciﬁc breed of dog to indicating Canis familiaris exemplars in general. Contrariwise, the mean cosine similarity between the token vectors of hound increases through the decades, since it originally meant ‘dog’ in general and ended up referring to dogs bred for hunting. Gulordava and Baroni (2011) resort to the American English section of the Google Books Ngram corpus, a collection of more than 5 millions of digitized books that were published between the sixteenth century and today (Michel et al. 2011), to build vector representations for words at two different time spans (the 60s and the 90s). The cosine similarity between the vector of a given word in the 60s space and the vector of the same word in the 90s space is then used as a measure of semantic shift for that term. These two time spans are taken into consideration in light of the major technological innovations that occurred in

12 Rodda et al. Tracking Semantic Change in Ancient Greek the 90s and presumably affected the English lexicon. Such a distributional approach is shown by the authors to complement the results of a simpler frequency-based one, already proposed by Michel et al. (2011), which for instance interprets the increase in relative frequency of a given term over time as a signal of its acquired popularity and therefore of its semantic shift. As Wijaya and Yeniterzi (2011) highlight, such a method falls short of describing the nature of the investigated changes and of spotting more gradual shifts that are not reﬂected in frequency variations. In their work (Wijaya and Yeniterzi 2011), k-means clustering and Topics-Over-Time (Wang and McCallum 2006), a time-dependent topic model, are exploited to observe how and when the topics surrounding a given word change in diachrony. Results clearly bring to light words that change their semantic meaning over time (e.g. gay from ‘frolicsome’ to ‘homosexual’ around the 70s) and words getting additional meanings (e.g. mouse from ‘long-tailed animal’ to ‘computer device’ around the 80s-90s). Kulkarni et al. (2015) compare the frequency-based approach with a syntactic one, which tracks variations in the probability distribution of part of speech tags given a target word in the different time snapshots of a corpus, and a best-performing word embeddings-based one (Mikolov et al. 2013), which learns word vectors for different time periods, warps the vector spaces into a unique coordinate system and creates a distributional time series for every word to assess its semantic displacement across time. With respect to Wijaya and Yeniterzi (2011), they also propose an algorithm for detecting the exact semantic change point in the time series built for each word with each of the three methods presented above. Their approach is also shown to be scalable and applicable to spotting shifts in different time spans, namely a century of written books with Google Book Ngrams, in years of Twitter blogging and in a decade of Amazon movie reviews. Diachronic distributional semantics is instead employed by Xu and Kemp (2015) to corroborate the parallel change law with respect to the differentiation one as for the semantic behavior of synonyms in time. Synonymic pairs like impending and imminent therefore tend to semantically evolve in parallel rathen than going different routes, maybe by virtue of analogical forces that aim at maintaining relationship patterns between words. Another application of the distributional approach to a diachronic corpus is carried out by Perek (2016), who investigates the productivity of the “V the hell out of NP” construction from 1930 to 2009. The vectors of the verbs occurring in this construction are analyzed with multidimensional scaling and clustering to pinpoint the preferred semantic domains of the construction in its diachronic evolution, while a mixed effects logistic regression analysis shows the density of the semantic space of the construction around a given word in a certain period to be predictive of that word joining the construction in the subsequent period. Hamilton, Leskovec, and Jurafsky (2016b) evaluate the performance of different kinds of word embeddings (PPMI, SVD, word2vec) in detecting attested historical semantic shifts (e.g. broadcast from ‘scatter’ to ‘transmit’) on cross-linguistic data by measuring changes in pair-wise similarities and the semantic displacement of a given lemma across time and run a series of regression analyses that reveal two general statistical laws of semantic change, namely that frequent words evolve at a slower rate and polysemous ones mutate faster. In a second study (Hamilton, Leskovec, and Jurafsky 2016a), they make use of both a global and a local neighborhood measure of semantic change to disentangle shifts due to cultural changes from purely linguistic ones. While the ﬁrst index, which measures the cosine similarity between the vectors of the same word in consecutive decades, fares better in spotting purely linguistic changes for verbs, the second one, which keeps track of the changes in the nearest neighbors of

13 Italian Journal of Computational Linguistics Volume 3, Number 1 a word over time, performs better in detecting culturally motivated changes on nouns (e.g. virus from ‘infectious disease’ to ‘unauthorized and harmful computer program’).

3. Materials and Methods

3.1 The Corpus

The corpus used for this study is based on the TLG-E (Thesaurus Linguae Graecae) collection of Ancient Greek literary texts. This corpus does not include inscriptions or private letters and/or non-literary papyri, but it does include several fragmentary texts in both poetry and prose genres. Texts were divided into two sub-corpora, the former spanning from the 7th to the 1st century BC (pre-Christian era), while the latter spans from the 1st to the 5th century AD (early Christian era). The pre-Christian sub-corpus contains 6,795,253 tokens, while the Christian sub-corpus totalizes 29,051,269 tokens.

Table 1 Percentage distribution of the main textual genres in the BC era and the AD era subcorpora (please keep in mind that a given text may belong to more than one genre at once). Genre BC era AD era Epic poetry 2.3% 0.3% Historiography 13.79% 15.43% Iambus and lyric 13% 6.24% Tragedy 6.7% 2.6% Comedy 12.88% 0% Philosophy 14.86% 47.87% Astronomy 2.54% 7.10% Medicine 3.84% 19.17% Mathematics 5.71% 2.09%

As Table 1 clearly shows, the two subsections are rather heterogeneous as regards the distribution of the main textual genres that compose them. When inspecting percentage values, please keep in mind that a given text may partake of more than one genre at once. As we can see, while the percentage of poetical texts (epic, iambic and lyric poetry) and theatrical texts (tragedy and comedy) diminishes from the BC to the AD era, the AD centuries are characterized by a greater diffusion of philosophical and technical (e.g. astronomical and medical) writings, with the exception of mathematical writings, that decrease from 5.71% to 2.09%. The percentage of historiographical works, finally, does not appear to vary considerably across the centuries. Texts were lemmatized using Morpheus (Crane 1991). This parser is estimated to reach approximately 80% accuracy in lemmatizing Ancient Greek (Boschetti 2009, p. 60). Minor issues with the lemmatization are therefore to be expected, and will be mentioned and discussed in the Results section. Generally speaking, they seem to fall into two categories. The most basic issue arises when some inflected forms of a lemma are erroneously lemmatized separately (examples are visible in Table 5, where the comparative and superlative of the adjective ταχύς “takhýs; swift”, e.g., appear as distinct lemmas); this kind of mis-lemmatization, however, should not have a significant impact on the semantic analysis, since said redundant lemmas will effectively have the

14 Rodda et al. Tracking Semantic Change in Ancient Greek same meaning, and can be expected to behave in similar ways. Cases where forms of a word are lemmatized under an entirely unrelated lemma could, on the other hand, affect the results in a more signiﬁcant way, but they appear to be very rare (the main example that can be detected in our data concerns forms of ψυχή “psykhé; soul” being erroneously lemmatized under ψῦχος “psykhos;ˆ cold”: see section 4.3 below).

3.2 Building the Distributional Vector Spaces

Distributional Semantic Models (Lenci 2008; Turney and Pantel 2010) implement the distributional hypothesis advanced by Harris (1954), whereby linguistic expressions that are similar in meaning tend to occur in similar contexts. In these models, target linguistic expressions are represented as vectors in a high-dimensionality space, while each dimension of the vectors records the co-occurrence statistics of the target elements with some contextual features, e.g. the content words occurring in a ﬁxed contextual window on the left and on the right of the target. By virtue of their representation with distributional vectors, words are encoded as points in a semantic space (Sahlgren 2006), and geometric measures of vector similarity or distance, like cosine (Turney and Pantel 2010), are exploited to model their semantic similarity. Like previous applications of distributional semantics to Ancient Greek (Boschetti 2009), we built two vector spaces from the TLG corpus, one from the pre-Christian subsection (BC-Space henceforth) and one from the Christian subsection (AD-Space henceforth). After ﬁltering out stop-words (mainly particles, pronouns and connectives) and lemmas occurring with a frequency below 100 tokens, the pre-Christian and Christian sub-corpus contain, respectively, 4,109 and 10,052 lemmas, which were used both as targets and dimensions in our vector spaces. A vector space model was then built for each sub-corpus using the DISSECT toolkit (Dinu, Pham, and Baroni 2013). Co- occurrences were computed within a window of 11 words (5 content words to the right and to the left of each target word). Association scores were weighted using positive point-wise mutual information (PPMI) (Turney and Pantel 2010), a statistical association measure that computes if two words x and y co-occur more often than expected by chance and sets to zero the negative results:

P (x, y) PPMI(x, y)=max(0, log ) (1) P (x)P (y)

The resulting matrices were reduced to 300 latent dimensions with Singular Value Decomposition (SVD) (Deerwester et al. 1990).

3.3 RSA of the Distributional Vector Spaces

We have adapted the RSA method to discover semantic changes between the two vector spaces:

1. we identiﬁed the words occurring in both sub-corpora with a frequency higher than 100 tokens, obtaining 3,977 lemmas; 2. we built a representational similarity matrix (RSM) from the BC-Space (RSMBC) and one from the AD-Space (RSMAD). Each RSM is a square matrix indexed horizontally and vertically by the 3,977 lemmas and containing in each cell the cosine similarity of a lemma with the other

15 Italian Journal of Computational Linguistics Volume 3, Number 1

lemmas in a vector space (this is a minor variation with respect to the original RSA method, which instead uses dissimilarity matrices). A RSM is a global representation of the semantic space geometry in a given period: vectors represent lemmas in terms of their position relative to the other lemmas in the semantic space; 3. for each lemma, we computed the Pearson correlation coefﬁcient between its vector in RSMBC and the corresponding vector in RSMAD.

The Pearson coefﬁcient measures the degree of semantic shift across the two temporal slices. The lower the correlation, the more a word changed its meaning.

4. Discussion of Results

The following section focuses on the words that underwent the biggest changes, i.e. those with the lowest correlation scores. The primary goal is to establish whether these words can be clustered into meaningful groups. This would allow us to pinpoint the areas within the lexicon of Ancient Greek that underwent a signiﬁcant semantic shift during the earliest centuries of Christianity.

4.1 Qualitative Analysis

The 50 lemmas with the lowest correlation coefﬁcients were scrutinized by hand, in order to establish whether meaningful subgroups emerge. (This list of words is not reproduced here due to space constraints. They are a subset of the 200 words used to build the plot in section 4.3) The ﬁndings in this section, while inevitably limited by the intuition of the researcher, will provide the starting point for a more sophisticated analysis to be performed in the following sections. The lemmas under consideration form a somewhat heterogeneous collection, including some adverbs and relatively common verbs such as ἕπομαι “hépomai; follow”, as well as some proper nouns. This notwithstanding, two promising subsets of words emerge even at this preliminary stage (see the examples in Table 2).

Table 2 Some examples of lemmas undergoing the most substantial semantic change Lemma BC era meaning AD era meaning CHRISTIAN TERMS παραβολή parabolé ‘comparison’ ‘parable’ λαός laós ‘people’ ‘the Christians’ κτίσις ktísis ‘founding’ ‘creation’ TECHNICAL TERMS ὑπόστασις hypóstasis ‘foundation’ ‘substance’ δύναμις dýnamis ‘power’ ‘property (of beings)’ ῥητός rhetós ‘stated’ ‘literal (vs. allegorical)’

The ﬁrst group comprises several nouns designating eminently Christian concepts, such as παραβολή (“parabolé; parable”, previously “comparison”), λαός (“laós”; used for the Christian community as opposed to non-Christians, previously “people”), κτίσις

16 Rodda et al. Tracking Semantic Change in Ancient Greek

(“ktísis; creation”, previously “founding, settling”). These findings are in line with the idea that the diffusion of Christianity played a substantial role to drive semantic change in the first centuries AD (cf. Boschetti (2009)). Other Christian terms, such as θεός (“theós; God”), ἄγγελος (“ángelos; angel”, previously “messenger”), πατήρ (“patér; father”), υἱός (“hyiós; son”), also occur among the 100 words with the lowest correlation coefficients. The shift undergone by words such as τόκος (“tókos; childbirth”) is also likely to be connected to their occurrence in Christian contexts, even though it is hard to define this as a “meaning shift” stricto sensu. Such cases, and the theoretical issues they bring about, will be discussed separately in section 4.2, in light of the results of the nearest neighbor analysis. Another group of lemmas comprises technical terms whose usage seems to have undergone a specialization or a shift from one domain of knowledge to another. These include words such as ὑπόστασις (“hypóstasis; substance”, previously “sediment, foundation”), δύναμις (“dýnamis; property (of beings)”, previously “power”), or ῥητός (“rhetós; literal” as opposed to “allegorical”, previously “stated”). When the lemmas in this group refer to metaphysical concepts or exegetical terms, the influence of Christian thought may also be present. Within this category as well, one finds cases such as ἐνιαύσιος (“eniáusios; annual”), where the meaning of the word can hardly be assumed to have changed in the strictest sense, but its context of usage (as will be made clear by the nearest neighbor analysis in the next section) has shifted towards technical literature. Together, the most clear-cut examples of these two groups (including those for which a semantic shift will be recognizable thanks to the nearest neighbor analysis performed in the following section) account for about half of the 50 words that underwent the most substantial semantic change. There is, of course, a measure of subjectivity in judging which words shifted towards a Christian or technical meaning; the findings in this section, however, can be supported through a more refined analysis.

4.2 Analysis of Nearest Neighbors

Nearest neighbor analysis proves especially useful when it comes to detecting shifts in meaning that would not be predictable through simple observation. Thus, for instance, the neighbors for μοῖρα (“môira”, another highly polysemous lemma, with meanings spanning from “part” to “destiny”) in the AD-Space come exclusively from the domain of astronomy and geometry (see Table 4; note that διάμετρον “diámetron; daily ration” is likely to be a lemmatization error for διάμετρος “diámetros; diameter”), showing a strong specialization towards a technical usage (“degree” or “division” of the Zo- diac). Similarly, among the neighbors for the apparently anodyne noun ζυγόν (“zygón; yoke”) one ﬁnds the constellations Λέων (“Léon; Leo”), Σκορπίον (“Skorpíon; Scor- pius”), Παρθένος (“Parthénos; Virgo”), and Τοξότης (“Toxótes; Sagittarius”), revealing a shift in usage towards the astronomical sense, where Ζυγόν is the name of the constellation and Zodiac sign “Libra”. This word, however, is the only name of a constellation that appears among the last 50 lemmas according to the correlation coefﬁcient; in any case, the presence of words such as ὑποτάσσω (“hypotásso; to set, to submit”), δούλειος (“dóuleios; slavish”), and φορτίον (“fortíon; load”) among the nearest neighbors in the AD-space shows that the astronomical meaning did not become as predominant as in the case of μοῖρα. A similar surprising result comes from the geographical adjective, Ποντικός (“Pon- tikós; coming from Pontus”), whose nearest neighbors shift from proper names and philosophical terms in the pre-Christian age (an association due, without doubt, to the usage of “Ponticus” as an epithet for authors, e.g. Heraclides) to names of currency and

17 Italian Journal of Computational Linguistics Volume 3, Number 1

Table 3 Examples of nearest neighbors in the BC- and AD-space πνεῦμα ‘breath’ ‘spirit’ → BC-space NNs AD-space NNs ἀήρ aér ‘air’ θεάομαι theáomai ‘to contemplate’ ὑγρός hygrós ‘moist’ ἀληθινός alethinós ‘true’ θερμός thermós ‘hot’ αἰών aión ‘aevum’ ψυχρός psykhrós ‘cold’ κτίσις ktísis ‘creation’ ὑγράζω hygrázo ‘to be wet’ υἱός hyiós ‘son’ θερμαίνω thermáino ‘to heat’ θεός theós ‘God’ πυκνός pyknós ‘compact’ πατήρ patér ‘God the Father’ ἀναπνοή anapnoé ‘breathing’ δοξάζω doxázo ‘magnify’ ψυχρόομαι psykhróomai ‘to be chilly’ οἰκονομία oikonomía ‘administration’ θερμότης thermótes ‘heat’ πληρόω pleróo ‘to ﬁll’ δύναμις ‘power’ ‘property (of beings)’ → BC-space NNs AD-space NNs προάγω proágo ‘to lead forward’ ἐνέργεια enérgeia ‘activity’ πολιορκία poliorkía ‘siege’ μετέχω metékho ‘to partake of’ ἀθροίζω athróizo ‘to gather’ ἐνεργέω energéo ‘to be in action’ στρατόπεδον stratópedon ‘encampment’ κινητικός kinetikós ‘related to motion’ στρατιώτης stratiótes ‘soldier’ φύς phýs ‘son’ παράταξις parátaxis ‘line of battle’ οὐσία ousía ‘substance’ ἀναζεύγνυμι anazéugnymi ‘to yoke’ ἰδιότης idiótes ‘speciﬁc property’ καταπλήσσω kataplésso ‘to strike down’ φύσις phýsis ‘nature’ Καρχηδόνιος Karkhedónios ‘Carthaginian’ ποιότης poiótes ‘quality’ ἀναλαμβάνω analambáno ‘to take up’ δισσός dissós ‘twofold’

trade wares, probably as a reﬂection of the integration of Pontus as a Roman province (with the obvious repercussions on trade) in the 1st century AD. This is not, strictly speaking, a shift in meaning, but in real-word reference and usage; as such, it is parallel to cases such as θεός, where the most relevant change is in the cultural context. Specialization towards a narrower usage is not, however, the only possible route of semantic change for technical terms: some of these appear to have moved from one domain to another. The case of πνεῦμα, whose semantic domain shifts from physics to metaphysics and philosophy (see Table 2 above), has already been discussed. Another example is σύμπτωμα (“sýmptoma” with the generic meaning of “chance occurrence”), whose top three neighbors in the BC-space are λογισμός (“logismós; calculation, reasoning”), θεωρέω (“theoréo; to contemplate”), and προερέω (“proeréo; to predict”); in the AD-space, in their place we ﬁnd πυρετέω (“pyretéo; to be feverish”), νόσημα (“nósema; disease”), and πυρετός (“pyretós; fever”), revealing a shift from the philosophical to the medical domain (i.e. from “property” to “symptom”). Another example, this time spanning the technical and Christian domains, is παραβολή (“parabolé; parabola, parable”, among other possible meanings), whose neighbors in the BC-space mostly have to do with geometry, while in the AD-space they pertain to the domain of biblical and literary exegesis. The nearest neighbors of ῥητός, one of the lemmas that had already

18 Rodda et al. Tracking Semantic Change in Ancient Greek

Table 4 Examples of nearest neighbors for astronomical terms μοῖρα ‘part, portion’ ‘degree, division (of the Zodiac)’ → BC-space NNs AD-space NNs ἕπομαι hépomai ‘to follow’ ἔγγιστος éngistos ‘nearest, next’ δύω dýo ‘to plunge in, to enter’ ζῳδιακός zodiakós ‘Zodiac’ μένος ménos ‘might, spirit’ ἰσημερινός isemerinós ‘equinoctial’ κέω kéo ‘to lie down, to rest’ πάροδος párodos ‘passage, entrance’ γαῖα gâia ‘earth’ διάμετρον diámetron [‘diameter’*] ἀστήρ astér ‘star’ τμῆμα tmêma ‘section, sector’ ἦμαρ êmar ‘day’ Κριός Kriós ‘Aries’ τόσος tósos ‘so much (as)’ μεσουρανέω mesouranéo ‘to culminate’ λείπω léipo ‘to leave’ κέντρον kéntron ‘center’ αὐτίκα autíka ‘at once’ μεσημβρινός mesembrinós ‘of noon, southern’ * see in-text discussion. ζυγόν ‘yoke’ ‘Libra’ → BC-space NNs AD-space NNs κέω kéo ‘to lie down, to rest’ ὑποτάσσω hypotásso ‘to set; to submit’ ὦμος ômos ‘shoulder’ δούλειος dóuleios ‘slavish’ ἕπομαι hépomai ‘to follow’ Λέων Léon ‘Leo’ πούς póus ‘foot’ κυριεύω kyriéuo ‘to be lord’ μέση mése ‘middle string’ φορτίον fortíon ‘load’ μέσος mésos ‘middle’ δουλεύω douléuo ‘to be slave’ δόρυ dóry ‘shaft, spear’ Σκορπίον Skorpíon ‘Scorpius’ μοῖρα môira ‘part, portion’ Παρθένος Parthénos ‘Virgo’ λαιός laiós ‘left’ Τοξότης Toxótes ‘Sagittarius’ γόνυ góny ‘knee’ ἐλεύθερος eléutheros ‘free’

been singled out as promising examples of a shift towards a technical meaning through qualitative analysis, show a similar evolution from the mathematical to the exegetical domain. There are also sporadic cases where the shift in meaning seems to be from a more technical usage in the BC-space to a more generalized meaning in the AD-space. A representative example is the verb δίειμι (“díeimi; to go through”). Its nearest neighbors in the BC-space all come from the domain of physics, and are indeed strongly specialized towards adjectives indicating properties of matter (see Table 5; some more minor issues with lemmatization make an appearance here, with the same adjective being categorized as two different lemmas, but since these lemmas seem to behave in a similar fashion, the impact on the results can be supposed to be minimal). In the AD-space, the physical domain seems to have disappeared entirely, with the synonym διέρχομαι (“diérkhomai; to go through”) now taking pride of place among the nearest neighbors. Of course, it is also possible that the appearance of this kind of pattern for a limited number of lemmas might be due to the different size of the two sub-corpora. Finally, like in the qualitative analysis, we ﬁnd examples of lemmas where the shift seems to have to do with a different context of usage rather than thorough meaning

19 Italian Journal of Computational Linguistics Volume 3, Number 1

Table 5 Nearest neighbors for δίειμι δίειμι ‘to go through’ BC-space NNs AD-space NNs πυκνός pyknós ‘compact’ διέρχομαι diérkhomai ‘to go through’ λεπτόν leptón ‘thin’* θάσσων thásson ‘swifter’* λεπτός leptós ‘thin’* διεξέρχομαι diexérkhomai ‘to pass through’ ξηρά xerá ‘dry’* ἀξιόλογος axiólogos ‘remarkable’ παχύς pakhýs ‘thick’ ὁπόσος hopósos ‘as much (as)’ ψυχρός psykhrós ‘cold’ τάχιστος tákhistos ‘very swift’* ξηρός xerós ‘dry’* χωρίον khoríon ‘place’ ὑγρός hygrós ‘moist’ διέξειμι diéxeimi ‘to pass through’ μανός manós ‘sparse’ ἀποχωρέω apokhoréo ‘to go away’ ὑγρότης hygrótes ‘moisture’ πλεῖστος pléistos ‘(the) most, greatest, largest’ * see in-text discussion.

change. Perhaps the most clear-cut case is the locative adverb αὐτόθεν (“autóthen; from this very spot, immediately”), whose nearest neighbors in the BC-space are entirely generic (including words such as ἄγνυμι “ágnymi; to break” and ναῦς “náus; ship”), while in the AD-space they seem to pertain mostly to the domain of logical and mathematical reasoning (with words such as ὑπόθεσις “hypóthesis; hypothesis”, ἀκόλουθος “akólouthos; following, consequent”, and ἀποδείκνυμι “apodéiknymi; to prove, to demonstrate”). In this case, just as for τόκος in section 4.1, it is hard to posit a “meaning shift” of any sort, but we can envisage a technical context of usage becoming predominant. Cases in which the change in context does not seem to straightforwardly translate to a shift in meaning, draw attention to one of the subtlest implications of the results presented here. Given the small dimensions of the corpus, it is sometimes difficult to rule out an influence of the genre of the texts analyzed on the distribution of results — for instance, the impact of technical usage on the meaning of many of the terms that underwent the most significant semantic change might be connected to the presence of a higher number of philosophical and technical treatises in the AD-space. As we showed in Section 3.1, the percentage of works classified as “philosophical” in the TLG catego- rization system does indeed rise steeply in the AD-corpus (47.87%, as opposed to 14.86% in the BC-corpus), but the increase is less noticeable for other technical genres (e.g. astronomical writings, 2.54% to 7.10%, and medical writings, 3.84% to 19.17%), while the percentage of mathematical writings is actually lower in the AD-corpus (2.09%) than in the BC-corpus (5.71%). Note that, since the same work can be categorized as belonging to more than one genre in the TLG, percentages for different genres need to be kept apart. Further research should undoubtedly highlight the effect of corpus composition; a focus on shorter periods of time might be of interest for future studies, since, for instance, the rise of technical prose writing is widely recognized as being a characteristic of the Hellenistic Age (cf. e.g. Gutzwiller (2007, p. 154-167). Note that, for the aims of this study, texts from this period are included in the BC-space, not the AD- space). A documented change in the proportion of different possible usages of a word, however, is in itself a very informative result, especially in a field such as Classics, where

20 Rodda et al. Tracking Semantic Change in Ancient Greek the analysis of (literary) texts is paramount. Indeed, the shift towards Christian usage for several terms can in itself be described as the introduction of an entirely new genre of Christian writings, but this would sidestep the issue that there has been a noticeable change in the usage of these words (and, by deﬁnition, their meaning, according to the Distributional Hypothesis).

4.3 t-SNE Plot

As a final analysis, we embedded the RSMAD vectors for the 200 words with the lowest correlation coefficient with the corresponding RSMBC vectors in a two-dimensional space using t-SNE (Figure 1), a technique for dimensionality reduction and data visu- alization that overcomes some of the limitations of standard multidimensional scaling (Van der Maaten and Hinton 2008). This procedure allows for easy identification of clusters, thus revealing the semantic relation between the most recent meanings of the words that underwent the greatest semantic change. While the analysis in the previous sections was aimed at detecting patterns of semantic shift between the BC-space and AD-space, the purpose of the t-SNE plot is to investigate whether there is any significant relationship between the meanings of the words that underwent such a shift; because of this difference in purpose, the information contained in the plot is limited to one semantic space. For the same reasons, the potential issues about the composition of the corpus and the impact of genre, as sketched at the end of section 4.2 above, are not relevant for the discussion here. A number of small clusters can be observed in the plot. Near the left periphery, the most relevant group (in blue) is composed of terms pertaining to (Christian) theology, from κύριος (“kýrios; Lord”), λαός and θεός, to παρουσία (“parousía; Advent”), ποιμήν (“poimén; shepherd”), τέρας (“téras; sign, portent”), and οὐρανός (“ouranós; heaven”). The position of ψῦχος (“psykhos;ˆ cold”) near this cluster is due to the mis-lemmatization of some inflected forms of ψυχή (“psykhé; soul”) under this lemma, as revealed by nearest neighbor analysis (see section 3.1 above). To the left of this group, a small cluster of terms (in light blue) pertaining to Christian exegesis (ῥητός, παραβολή, διασαφέω “diasaphéo; to illustrate”) can be recognized. At the far right of the plot, diametrically opposed to the previous clusters, another small group of Christian terms can be recognized; this includes πατήρ, ὑιός, πνεῦμα, and potentially καρδία (“kardía; heart”) and σάρξ (“sárx; flesh”). The upper portion of the plot (in green) houses technical terms from the domains of medicine (the upper-most group, spanning the personal name ῾Ιπποκράτης “Hip- pokrátes; Hippocrates”, the nouns διάθεσις “diáthesis; condition” and σύμπτωμα, the verb καταπλάσσω “kataplásso; to apply a plaster/poultice”, and the adjective πρόσφατος “prósphatos; fresh”), astronomy and geometry (difficult to distinguish, from μοῖρα and πάροδος “párodos; passage” to ἄκρος “ákros; top-most” and δισσός “dissós; two-fold”). Philosophical terminology (in red) can be found in the lower right area (δύναμις, ὑπόστασις, etc.), while a separate cluster of terms pertaining to moral philosophy (ἐπιτήδειος “epitédeios; suitable”, ἱκανός “hikanós; sufficient”, ἐπιμελής “epimelés; care- ful”, all clustering around the crucial term ἄλυπος “álypos; without pain, painless”) is visible nearer to the center of the plot (in brown). Some smaller groups are also noticeable, such as μνᾶ (“mnâ; mina”) and δραχμή (“drakhmé; drachma”), both units of currency, on the left (in orange), and πρώτιστος (“prótistos; the very first”) and Τίμαιος (the proper name Tímaios, Latin Timaeus), both connected to (Neo-)Platonic philosophy, on the right (in red). All in all, despite the inevitable amount of noise, the plot in Figure 1 supports the findings detailed so far. We can see how the main

21 Italian Journal of Computational Linguistics Volume 3, Number 1

καταπλάσσω πρόσφατος χρηστέον ἁρµόζω

σύµπτωδιάθεσιςµα Ἱπποκράτη

100 Ἱπποκράτης µοῖρα πάροδος

ἐπιβολεύς ἕποµαι µῦς ἐπιβολή δωδέκατος ἤτοι γόνιµος ὁλοσχερήςἐνιαύσιος ἄκρος δισσόςἔµπαλιν πρώτιστος σύνεγγυς προερέωχωρίς Τίµαιος συνάγωπροστίθηµι µνᾶ ἐκφαίνω ἀποκαθίστηµι φωνήεις λήγω δραχµή θεός κτίσις πρόειµι προσλαµβάνω 50 τουτέστι θεάοµαιαἰών θέωψῦχος Ἀρίσταρχος παράγω τρίς τελειόω δυάω χρῄζω σωτήρ οὐδέτερος ἔσχατοςἄνω πρόδηλος χρέος κύριος ἐξαιρέω ἐνδείκνυµι πρόσκειµαικαίνωἐγκλίνω παρουσία ἐξαίρω πλεονάκιςἐγχωρέω πρόθεσις πληρόω Ἀπίς συντελέωὑγιής ὁµός ὑπεναντίος δήπου λαόςἰδού τύπον προσαρµόζω ἑτοιµάζω ὧδε τύχος ἐξαποστέλλω καταγράφω ὑπόλοιπος ἄγγελος ποιµτέραςήν διπλάζω σύµπας ἀξιόλογος παραβολή ἀπόστασις πνεῦµα πατήρ Dim 2 Dim ῥητός θρόνονἀνέρχοµαι ὕψος παµπολύςἐπικρατέω υἱός διασαφέω οὐρανός ἐγγράφω θλίβωκαρδίασάρξ 0 Στέφανος ἀποτέµνω περιφέρω ὀνοµάζω σκότος λαµπρόςἀνώτερος δίειµι ἀνώνυµος φωνέωπρόσωπον ἡλίκοςπρόσωθενὑποστρέφω πολλαχῆ ἐντολή ἄλλοθι ἐντεῦθεν ἠρεὑποχωρέωµέω ὁπόσος ἅτεροςπροσεῖπον ἔστε καταναλίσκωἀπωθέω κυρέωφοιτάω ταπεινόωχήραταλαιπωρία τοσόσδε Διογένης δύναµις ῥύοἀποστρέφωµαι ἐπιµένω καταπίπτω εὐδία ὑπόστασις φυσάω ἐπίσχωὑπέρχοµαι προυπάρχωὑφίστηµι ἐπιτήδειος ἐπικίνδυνος σκώληξ ἄλυπος Ἄρατος θιγγάνω ἱκανός κυριεύω ἐπιβάλλω προγίγνοµαι ὕπαιθρος µεταφορά ἁδρόςεὔθετος -50 εἰσβαίνω ζυγόν ἡµίονοςµύρµηξ Ἀσκληπιάδαιἐπιµελής παρέξ Κρόνος ἀποτίθηἀναλαµµιβάνω ἐπιτίθηµι ἀνθέω ὄρθιος Ἰώ γονή ἄρδω µόγις ἄκαρποςβλαστάνω ὀλισθάνω ἕωθεν ἐξαπίνηςφρίσσω Σάτυρος ὄψιος πονέοµαι Χῖος πάµπαν Ποντικός κάµατος ῥόδονἰός τόκος -100 ἀνά

-100 -50 0 50 100

Dim 1 Figure 1 Relative positions within the AD-Space of the 200 words with the lowest correlation scores. Dimensionality reduction was performed using t-SNE.

semantic changes in the Greek lexicon between the pre-Christian and Christian era affected the domains of religion (in a broader sense) and/or technical language. Within these domains, some more ﬁne-grained relations between words that went through a signiﬁcant semantic shift can be observed.

5. Conclusions and future work

This paper shows how Distributional Semantics can be used as an exploratory tool to detect semantic change. In this case study on Ancient Greek, the proposed method based on distributional RSA not only conﬁrms the hypothesis that the diffusion of Christianity was a crucial cause of semantic change in the Greek lexicon, but also allows for the identiﬁcation of unexpected patterns of evolution, such as the specialization in the usage of technical terms. From a methodological standpoint, the fact that the results obtained from such a small corpus of purely literary texts are both meaningful and informative is of great relevance. The nearest neighbor analysis performed in section 4.2 brought to light several patterns of change, which proved informative both as concerns the evolution of some semantic domains between the BC- and AD-space, and

22 Rodda et al. Tracking Semantic Change in Ancient Greek the potential effects of the composition of the corpus (in itself a potentially interesting source of information for Ancient Greek). The t-SNE plot, by showing how the words that underwent the most relevant meaning shifts tend to form semantically-motivated clusters, provided a further opportunity to detect areas of the lexicon that underwent significant semantic change. As far as broader methodological issues are concerned, the choice to adopt a data- driven approach proved fruitful, in that it brought to light directions of change that were not expected a priori. For traditional research in Classics, a computational approach to the lexicon of Ancient Greek is compelling because it provides new information about a language for which the judgments of native speakers are unavailable (cf. Perek (2016)). The results of this study show how Distributional Semantics can complement the findings of the philologist, as well as help discover patterns of lexical change that would otherwise be impossible to grasp beyond an intuitive level. Nonetheless, a few issues remain open and could benefit from a more fine-grained investigation in future studies. First and foremost, it could be interesting to observe which parts of speech tend to change first, e.g. whether nouns or verbs (Dubossarsky, Weinshall, and Grossman 2016), and whether specific genres are more prone to change than others. Secondly, a targeted study of a more restricted period right after or right before the advent of Christianity (rather than the twelve-century time span considered here) could help confirm that the shifts we detected were primarily due to the spread of Christianity itself, which would have then represented a major breaking point, and rule out the possibility that a more natural and broad-spectrum change was already taking place.

References Boschetti, Federico. 2009. A Corpus-based Approach to Philological Issues. Ph.D. thesis, University of Trento. Crane, Gregory. 1991. Generating and parsing classical greek. Literary and Linguistic Computing, 6(4):243–245. Deerwester, Scott, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391. Dinu, Georgiana, Nghia The Pham, and Marco Baroni. 2013. DISSECT — DIStributional SEmantics Composition Toolkit. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 31–36, Soﬁa,Bulgaria, August, 4-9. Dubossarsky, Haim, Daphna Weinshall, and Eitan Grossman. 2016. Verbs change more than nouns: A bottom-up computational approach to semantic change. Lingue e linguaggio, 15(1):7–28. Edelman, Shimon. 1998. Representation is representation of similarities. Behavioral and Brain Sciences, 21:449–467. Gulordava, Kristina and Marco Baroni. 2011. A distributional similarity approach to the detection of semantic change in the google books ngram corpus. In Proceedings of the GEMS 2011 Workshop on GEometrical Models of Natural Language Semantics, pages 67–71, Edinburgh, Scotland, July 31. Gutzwiller, Kathryn J. 2007. A Guide to Hellenistic Literature. Blackwell Publishing. Hamilton, William L., Jure Leskovec, and Dan Jurafsky. 2016a. Cultural shift or linguistic drift? Comparing two computational measures of semantic change. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2116–2122, Austin, Texas, USA, November 1-5. Hamilton, William L., Jure Leskovec, and Dan Jurafsky. 2016b. Diachronic word embeddings reveal statistical laws of semantic change. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1489–1501, Berlin, Germany, August 7-12. Harris, Zellig S. 1954. Distributional structure. Word, 10(2-3):146–162. Kriegeskorte, Nikolaus and Roger A. Kievit. 2013. Representational geometry: Integrating cognition, computation, and the brain. Trends in Cognitive Sciences, 17(8):401–412.

23 Italian Journal of Computational Linguistics Volume 3, Number 1

Kulkarni, Vivek, Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2015. Statistically significant detection of linguistic change. In Proceedings of the 24th International World Wide Web Conference, pages 625–635, Florence, Italy, May 18-22. Lenci, Alessandro. 2008. Distributional semantics in linguistic and cognitive research. Italian Journal of Linguistics, 20(1):1–31. Michel, Jean-Baptiste, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, et al. 2011. Quantitative analysis of culture using millions of digitized books. Science, 331(6014):176–182. Mikolov, Tomas, Ilya Sutskever, Kai Chen, Greg S. Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26tth International Conference on Neural Information Processing System, pages 3111–3119, Lake Tahoe, Nevada, USA, December 5-10. O‘Donnell, Matthew Brook. 2005. Corpus Linguistics and the Greek of the New Testament. Number 6. Sheffield Phoenix Press. Perek, Florent. 2016. Using distributional semantics to study syntactic productivity in diachrony: A case study. Linguistics, 54(1):149–188. Sagi, Eyal, Stefan Kaufmann, and Brady Clark. 2011. Tracing semantic change with latent semantic analysis. In Kathryin Allan and Justyna A. Robinson, editors, Current Methods in Historical Semantics. Mouton de Gruyter, pages 161–183. Sahlgren, Magnus. 2006. The Word-Space Model: Using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces. Ph.D. thesis. Turney, Peter D. and Patrick Pantel. 2010. From Frequency to Meaning: Vector Space Models of Semantics. Journal of Artificial Intelligence Research, 37:141–188. Van der Maaten, Laurens and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9:2579–2605. Wang, Xuerui and Andrew McCallum. 2006. Topics over time: A non-markov continuous-time model of topical trends. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 424–433, Philadelphia, Pennsylvania, USA, August 20-23. Wijaya, Derry Tanti and Reyyan Yeniterzi. 2011. Understanding semantic change of words over centuries. In Proceedings of the 2011 International Workshop on DETecting and Exploiting Cultural diversiTy on the Social Web, pages 35–40, Glasgow, United Kingdom, October 24-28. Xu, Yang and Charles Kemp. 2015. A computational evaluation of two laws of semantic change. In Proceedings of the 37th Annual Meeting of the Cognitive Science Society, pages 2703–2708, Pasadena, California, July 22-25.

24 Distributed Representations of Lexical Sets and Prototypes in Causal Alternation Verbs

Edoardo Maria Ponti∗ Elisabetta Jezek∗∗ Universitá di Cambridge Università degli Studi di Pavia

Bernardo Magnini† Fondazione Bruno Kessler

Lexical sets contain the words filling an argument slot of a verb, and are in part determined by selectional preferences. The purpose of this paper is to unravel the properties of lexical sets through distributional semantics. We investigate 1) whether lexical set behave as prototypical categories with a centre and a periphery; 2) whether they are polymorphic, i.e. composed by sub- categories; 3) whether the distance between lexical sets of different arguments is explanatory of verb properties. In particular, our case study are lexical sets of causative-inchoative verbs in Italian. Having studied several vector models, we find that 1) based on spatial distance from the centroid, object fillers are scattered uniformly across the category, whereas intransitive subject fillers lie on its edge; 2) a correlation exists between the amount of verb senses and that of clusters discovered automatically, especially for intransitive subjects; 3) the distance between the centroids of object and intransitive subject is correlated with other properties of verbs, such as their cross-lingual tendency to appear in the intransitive pattern rather than transitive one. This paper is noncommittal with respect to the hypothesis that this connection is underpinned by a semantic reason, namely the spontaneity of the event denoted by the verb.

1. Introduction

The arguments of a verb are the “slots” that have to be filled to satisfy its valency (subject, object, etc.). Hence, verbs display so-called selectional restrictions over the possible fillers occupying these slots (Séaghdha and Korhonen 2014), which play a major role in determining the verb meaning. Moreover, the selection of the fillers happens in accordance with the different senses of a verb. Lists of the possible fillers occurring with a certain pattern of a verb can be collected in a corpus-driven fashion: these lists are named “lexical sets” (Hanks 1996). Several approaches in Computational Linguistics managed to inspect lexical sets and their patterns of variation (Montemagni, Ruimy, and Pirrelli 1995; McKoon and Macfarland 2000), as well as to use them as features for verb classification (McCarthy 2000; Joanis, Stevenson, and James 2008). On the other hand, selectional preferences were employed for many tasks in Natural Language Processing, including Word Sense Disambiguation (Resnik 1997; McCarthy and Carroll 2003, inter alia), Metaphor Pro- cessing (Shutova, Teufel, and Korhonen 2013), Information Extraction (Pantel et al.

E-mail: [email protected] ∗ E-mail: [email protected] ∗∗ E-mail: [email protected] †

2007), Discourse Relation Classification (Ponti and Korhonen 2017), Dependency Pars- ing (Mirroshandel and Nasr 2016), and Semantic Role labeling (Gildea and Jurafsky 2002; Zapirain et al. 2013). We aim at establishing a new method of analysis for lexical sets. In particular, we address the following key questions: are lexical sets Aristotelian categories (yes-no membership) or prototypical categories (graded membership)? If the latter holds true, are the fillers arranged homogeneously or do they cluster around some sub-categories? Finally, is the relation between lexical sets from different patterns of a same verb informative about this verb’s meaning? We address these questions under a distributional semantics perspective. In fact, distributional semantics provides a mapping between each filler and a vector lying in a continuous multi-dimensional space. By virtue of this, lexical sets can be treated as continuous categories, where members can be either central or peripheral. Moreover, this allows to quantify the distance between vectors with spatial measures. In order to test the relevance for linguistic theory of this approach, we focus on a case study, namely verbs undergoing the causative-inchoative alternation. Solving the above-mentioned issues may help clarifying some of the vexed questions about this class of verbs. These show both transitive and intransitive patterns: the object of the former and the subject of the latter play the same semantic role of patient. Based on the cross-lingual frequency of each of these patterns and the direction of morphological derivation, it is possible to sort these verbs onto a scale (Haspelmath 1993; Samardzic and Merlo 2012; Haspelmath et al. 2014), which can be possibly interpreted semantically as the “spontaneity” of the corresponding event (see § 2.2). We investigate the existence of any asymmetry between the lexical sets of the transitive object and the intransitive subject, and if so the connection of this asymmetry with the spontaneity scale. The structure of the paper is as follows. In § 2, we define the core notions of this study, including lexical sets, causative-inchoative verbs, and distributional semantics. § 3 presents the method and the data, whereas § 4 reports the results of the experiments. Finally, § 5 draws the conclusions and § 6 proposes possible future lines of research.

2. Deﬁnitions and Previous Work

In this section, we describe in detail the notions underlying the subject matter of the research (lexical sets, § 2.1), the case study (causative-inchoative verbs, § 2.2), and the method (distributional semantics, § 2.3). At the same time, we present the previous work concerning each of these aspects.

2.1 Lexical Sets

A lexical set can be defined as the set of words that occupy a specific argument position within a single verb sense, such as {gun, bullet, shot, projectile, rifle...} for the sense ‘to shoot’ of to fire, or {employer, teacher, attorney, manager...} for its sense ‘to dismiss’. The notion of lexical set was firstly introduced by Hanks (1996). Its purpose is explaining how the semantics of verbs is affected by the patterns of complements they are found with. Hanks’ approach is justified by the pervasiveness of patterns in corpora: these patterns are instantiated by specific lexical items typically occurring in the argument positions. These items, called fillers, form sets belonging to different patterns of meaning. Hanks and Pustejovsky (2005) and Hanks and Jezek (2008) propose an ontology where fillers are clustered into semantic types, i.e. categories such as [[Location]], [[Event]], [[Vehicle]], [[Emotion]]. These form a hierarchy that branches into more specific types.

26 Ponti et al. Distributed Representations of Lexical Sets and Prototypes

However, these categories are problematic, as lexical sets tend to “shimmer” (Jezek and Hanks 2010): their membership tends to change according to the verb they associate with. The shimmering nature of lexical sets is not an accidental phenomenon. Rather, it stems from the fact that verb selectional restrictions may cut across conceptual categories due to the speciﬁc predication introduced by the verb. For example, both wash and amputate typically select [[Body Part]] as their direct object. Nevertheless, they select different prototypical members of the set, as can be seen in the examples below from Jezek and Hanks (2010) where only shared members are underlined:

(1) [[Human]] wash [[Body Part]] where [[Body Part]]: {hand, hair, face, foot, mouth...}

(2) [[Human]] amputate [[Body Part]] where [[Body Part]]: {leg, limb, arm, hand, ﬁnger...}

Lexical sets can be extracted from corpora automatically. This operation hinges upon traditional techniques for the acquisition of subcategorisation frames and selectional restrictions. The former allow to capture the syntactic pattern (Brent 1991; Schulte Im Walde 2000) or semantic frame pattern (Baker, Fillmore, and Lowe 1998) in which each verb is found. The latter create probabilistic models of preferences over ﬁllers that are evaluated intrinsically through human judgments (Brockmann and Lapata 2003) and extrinsically through disambiguation tasks (Van de Cruys 2014).

2.2 Causative-Inchoative Verbs

In principle, lexical sets can be constructed for every verb. In this work, however, we limit our inquiry to a specific subset of verbs, namely causative-inchoative verbs in Italian. This provides a testbed for our methods of analysis, which can be easily extended to other classes of verbs and alternations. The choice of this specific subset is due to the fact that understanding the internal structure of lexical sets and their relations seems to be crucial to solve the problems surrounding this class of verbs, including asymmetries between transitive objects and intransitive subjects and their relation with the spontaneity scale (see below). Causative-inchoative verbs alternate. In other terms, they appear in two patterns, either as transitive or intransitive. In the former, an agent brings about a change of state affecting a patient; in the latter, the change of the same patient is presented as spontaneous (e.g. break, as in “Mary broke the key” vs. “the key broke”). The verbs in the two patterns can be expressed by either a same form or two distinct forms cross-lingually. In the second case, the forms can be morphologically asymmetrical: one has a derivative affix and the other does not. Otherwise, the forms are suppletive, being completely unrelated (e.g. kill/die). The members of causative-inchoative verbs that retain a same form or are morphologically related in both patterns vary cross-lingually (Montemagni, Ruimy, and Pirrelli 1995). Alternations regarding physical change-of-state and manner- of-motion are found in English, whereas they are limited to psychological and physical changes-of-state in Italian. In Japanese and Salish languages, also verbs like arrive and appear do alternate (Alexiadou 2010). From a semantic point of view, Italian causative-inchoative verbs are required to be telic and have an inanimate patient (Cennamo 1995). Morpho-syntactically, they are generated from an asymmetrical derivation, called “anti-causativisation.” The intransitive form is sometimes marked with the pronominal clitic si: its presence is mandatory,

27 Italian Journal of Computational Linguistics Volume 3, Number 1 optional or forbidden according to verb-specific rules (Cennamo and Jezek 2011). Be- cause of this, many different categorisations of Italian causative-inchoative verbs were attempted (Folli 2002; Jezek 2003). Causative-inchoative verbs in general are endowed with peculiar properties. Haspelmath (1993) claims that verbs with a cross-lingual preference for a marked causative form denote a more “spontaneous” situation. Spontaneity is intended by the author as the likelihood that the occurrence of the event denoted by the verb does not require the intervention of an agent. In this way, a correlation between the form and the meaning of these verbs was borne out. Moreover, Samardzic and Merlo (2012) and Haspelmath et al. (2014) demonstrated that verbs appearing more frequently (intra- and cross-lingually) in the inchoative form tend to morphologically derive the causative form. Here, the correlation bridges between form and frequency. Vice versa, situations entailing an agentive participation prefer to mark the inchoative form and occur more frequently in the causative form. However, what remains uncertain is whether spontaneous and agentive variants of the same verbs differ in their lexical sets. Atkins and Levin (1995) and Levin and Hovav (1995) argued that selectional restrictions for spontaneous verbs (named internal causation verbs) are stricter because their event unfolds due to some inherent property of the patient: referents without this capability are excluded, contrary to what happens with agentive verbs (named external causation verbs). This capability is defined “teleological” by (Copley and Wolff 2014). However, McKoon and Macfarland (2000) reported contradicting results from corpus-based analyses that did not find any significant difference in the content of lexical sets of spontaneous and agentive verbs (although from a sample of less than 100 sentences and only 5 categories). Instead, they reported a difference between transitive objects and intransitive objects in that the former contained a larger amount of abstract nouns compared to the latter.

2.3 Distributional Semantics

Once established the domain, we need to provide a reliable method of inquiry. Previous works based on set theory treated lexical sets as Aristotelian categories, of which a filler is either a member or not. For instance, Montemagni, Ruimy, and Pirrelli (1995) collected lexical sets manually and employed set intersection as a measure of similarity between two sets. Research in psychology, however, has long since demonstrated that the members of a linguistic set are found in a radial continuum where the most central element is the prototype for its category, and those at the periphery are less representative (Rosch 1973; Lakoff 1987).1 The full exploitation of the semantic information inherent to the argument fillers of verbs can take advantage of some recent developments in distributional semantics. Efficient algorithms have been devised to map each word of a vocabulary into a corresponding vector of n real numbers, which can be thought as a sequence of coordinates in a n-dimensional space (Mikolov et al. 2013a, inter alia). This mapping is yielded by unsupervised machine learning, based on the assumption that the meaning of a word can be inferred by its context, i.e. its neighbouring words in texts. This is known as Dis- tributional Hypothesis (Harris 1954; Firth 1957). Distributed models have some relevant properties: first, the geometric closeness of two vectors corresponds to the similarity

1 For previous work on lexical sets considering prototypicality in the context of the notion of shimmering, see Jezek and Hanks (2010).

28 Ponti et al. Distributed Representations of Lexical Sets and Prototypes in meaning of the corresponding words. Moreover, its dimensions possibly retain a semantic interpretation such that non-trivial analogies can be established among words. Word vectors allow to capture the spatial continuum implied by the notion of prototype. Previous works showed that word vectors can be clustered to imitate linguistic categories, because each cluster captures the ‘semantic landscape’ of a word (Hilpert and Perek 2015). The center of these clusters can be interpreted as the prototype of the corresponding category, and the proximity of the cluster members to the center as the degree of their prototypicality. In fact, the cluster members are not scattered randomly, but rather are arranged according to the internal structure of the cluster (Dubossarsky, Weinshall, and Grossman 2016). The prototypicality of the cluster members provides an explanation about linguistic phenomena, such as the resistance to the diachronic change of meaning (Geeraerts 1997; Dubossarsky et al. 2015). In this work, we extend the usage of the notion of prototypicality from meaning- based categories derived through vector quantization to grammatically deﬁned categories (i.e. lexical sets) derived through dependency parsing. Moreover, we go beyond the estimation of the distance of each word from the centroid of its category: in particular, we propose new methods to assess the internal diversity in terms of sub-categories and the distance between lexical sets themselves.

Table 1 List of 20 causative-inchoative verbs and count of their fillers for each argument slot. Lemma Translation S O chiuder(si) to close 289 606 aprir(si) to open 195 1337 aumentare to improve 534 998 romper(si) to break 83 419 riempir(si) to fill 58 166 raccoglier(si) to gather 85 505 connetter(si) to connect 39 134 divider(si) to split 129 246 finire to stop 1092 721 uscire/portare fuori to go/put out 325 638 alzar(si) to arise/raise 75 304 scuoter(si) to rock 10 69 bruciare to burn 75 174 congelare to freeze 10 30 girare to spin 155 243 seccare to dry 15 14 svegliar(si) to awake/wake 68 89 scioglier(si) to melt 94 143 (far) bollire to boil 2 2 affondare to sink 18 74

29 Italian Journal of Computational Linguistics Volume 3, Number 1

3. Data and Method

We sourced the lexical sets from a sample of ItWac, a wide corpus gathered by crawling texts from the Italian domain in the web using medium frequency vocabulary as seeds (Baroni et al. 2009). This sample was further enriched with morpho-syntactic information through the graph-based MATE-tools parser (Bohnet 2010). We trained and evaluated this parser on the Italian treebank inside the collection of Universal Dependencies v1.3 (Nivre et al. 2016). The evaluation on gold standard data suggests how many errors we expect to affect the predictions on the new data, i.e. the ItWac corpus: these errors are then propagated to the following steps for the extraction of lexical sets. According to the LAS metric, the relevant dependency relations scored 0.751 for dobj (direct object), 0.719 for nsubj (subject), and 0.691 for nsubjpass (subject of a passive verb). A target group of 20 causative-inchoative verbs was taken from Haspelmath et al. (2014): they are listed in Table 1, together with the count of the extracted lexical sets for the relevant semantic macro-roles (see below). Once the sentences were parsed, the target verbs were identified inside the dependency trees. The lemmas of these verbs and the forms of their arguments were stored in a database. Argument fillers were grouped according to the semantic macro-roles defined by Dixon (1994), rather than their dependency relations: subjects of transitive verbs (A), subjects of intransitive verbs (S) and objects (O). In particular, the subjects of verb forms accompanied by the si-clitic and those without an object depending on the same verb were treated as S.2 These operations resulted in a database structured as follows: in each row, a verb is alongside of the fillers it occurs with in a specific sentence. For example, compare an original sentence and its corresponding entry in Example 3:

(3) Plinio il Vecchio non cita più il Po di Adria Pliny the Elder doesn’t mention anymore the Po of Adria perché l’ argine dell’ Adige si era rotto ed era because the bank of the Adige had broken and had conﬂuito nella Filistina. merged with the Filistina.

Verb A S O citare Plinio _ Po rompere argine conﬂuire _ _ _

Because of the nature of vector models, we made the following design choices to deal with special linguistic phenomena. We discarded everything but the head of multi-word nouns, such as Plinio of Plinio il Vecchio, to preserve the one-to-one mapping between words and vectors. We did not distinguish proper and common nouns, such as Po and argine, since their representations lie in the same multi-dimensional space. Subjects in ellipsis or co-reference were neglected, since no pragmatic annotation of the sentences was available: for instance, Adige should appear as S in the entry for conﬂuire, which is left blank instead. Finally, polysemous words and homonyms were

2 Subjects of verbs inﬂected in the passive voice were treated as O, instead.

30 Ponti et al. Distributed Representations of Lexical Sets and Prototypes represented by a single form, hence a single vector, since their representations conflate all the relevant meanings. For instance, Adige is both a river and a location, but these are not distinguished in the vector model. The database was later collapsed by verb lemma so that each of them became associated to three sets of fillers (one per macro-role). Each of these sets is a corpus- based lexical set. Compared to manually picked lexical sets, they are more noisy but less sparse: the vastness of the data mitigates the errors in the parsing step. Moreover, the automation in lexical set extraction allows to access the fillers of virtually every verb: resources based on manual selection like T-PAS (Jezek et al. 2014), on the other hand, are limited to a small amount of verbs. Afterwards, each of the argument fillers was mapped to a vector according to three different pre-trained models. The vectors are generated unsupervisedly back- propagating the gradient from the loss of a task to update randomly initialized embeddings. Each model, however, relies on different tasks:

CBOW stands for Continuous Bag of Words and is part of the Word2Vec r suite (Mikolov et al. 2013b). The Italian model was developed by Dinu, Lazaridou, and Baroni (2015) through negative sampling. 300-dimensional representations were obtained by training a binary classiﬁer that discrimi- nates whether a pair of a target word w and a context c belongs to an actual text or not. True contexts are obtained from a window of 5 words on either side of the target word, false contexts are drawn randomly from a noise distribution. Moreover, infrequent words are pruned. On the other hand, a preprocessing step called subsampling deletes words from windows whose frequency f is higher than an threshold with a probability p =1 t . The − f text from which word-context pairs are sampled is the full ItWac corpus, consisting in 1.6 billion tokens. The representations resulting from this algorithm are organised by topical similarity or relatedness. fastText embeddings were trained on the dump of the Italian Wikipedia r through a character-level skip-gram (Bojanowski et al. 2017). The skip-gram algorithm predicts the following element given a sequence of previous elements. The elements of Bojanowski’s version are characters: each word is represented by a bag of character n-grams and its representation consists in the sum of the representations of each of them. The vector dimensionality is 300; random sequences are drawn with a ratio of 5 to 1 compared to true sequences, with a probability proportional to the square root of the uni-gram frequency. The size of the context window is uniformly sampled from values between 1 and 5. The rejection threshold for subsampling is 4 10− . The usage of character n-grams makes this vector model sensitive to morphological similarity. Polyglot vectors result from a classiﬁer distinguishing between sequences r taken from the dump of the Italian Wikipedia and corrupted sequences (Al-Rfou, Perozzi, and Skiena 2013). The window of phrases was 5, the dimensionality of vectors 64. Subsampling discarded words not appearing in the raking of the 100 thousand most frequent ones.

In order to measure spatial distances inside these vector models, many different metrics are available, including pure geometrical (Euclidean) distance. In this work, we

31 Italian Journal of Computational Linguistics Volume 3, Number 1 rely on the popular metrics of cosine distance (Cha 2007). Assume that a and b are th vectors, and that ai and bi are their i components, respectively. The cosine similarity between a and b is then deﬁned as follows:

n a b cos(θ)= i=1 i i (1) n 2 n 2 i=1 ai i=1 bi The opposite, namely cosine distance d, is simply deﬁned as 1 cos(θ). As for the − values that d(a,b) can assume, the minimum is at 0 (angles completely overlap) and the maximum is at 1 (orthogonal vectors).

4. Experiments

In order to answer the questions outlined in the introduction, we devised three experiments. In all of them, we assume that the S and O lexical sets might not overlap, as borne out by Montemagni, Ruimy, and Pirrelli (1995) and McCarthy (2000). The ﬁrst (§ 4.1) investigates the internal structure of lexical sets and how their members aggregate around a prototype. Based on psycholinguistic theories (see § 2.3), we expect vectors to lie on a continuum. As for causative-inchoative verbs in particular, objects were shown to be more homogeneous than subjects (McKoon and Macfarland 2000), hence they should form denser clusters. The second experiment (§ 4.2) deals with polymorphism of lexical sets, i.e. the amount of distinct sub-categories they contain. We expect the number of sub-clusters to be proportional to the number of verb senses. Finally, the third experiment (§ 4.3) is aimed at studying how lexical sets of different arguments of the same verb are related. In particular, we predict that the distance between intransitive subject and transitive object varies depending on the spontaneity of the causative-inchoative verb (see § 2.2). In fact, intransitive subjects and transitive objects of spontaneous verbs should be limited to referents with “teleological capability” (Atkins and Levin 1995; Levin and Hovav 1995), hence creating dense groups possibly located far from each other. Fillers of agentive verbs instead should show a higher degree of overlap because of their looser constraints.

4.1 Prototypicality: Distance from Centroid

Once the fillers have been mapped to their corresponding vectors, a lexical set appears as a group of points in a multi-dimensional model. The centre of this group is the Euclidean mean among the vectors, which is a vector itself and is called centroid. In the first experiment, we measure the cosine distance of every vector member of a lexical set from the centroid estimated from all the other members.3 This leave-one-out setting aims at avoiding biases due to outliers (such as phraseological usages or misspellings). In semantic terms, this measure should correspond to assessing how far a filler is from its prototype. We obtained two sets (S and O) of cosine distance values for each verb: these can be plotted as boxes and whiskers, like in Figure 1. The cosine distances are represented as a scatter plot: on its side, a box informs about the average (horizontal line), the median4

3 Every ﬁller was weighted proportionally to its absolute frequency. 4 The median is the value separating the higher half of the ordered values from the lower half.

32 Ponti et al. Distributed Representations of Lexical Sets and Prototypes

chiudere aprire aumentare rompere 1.00 1.00 1.00 1.00

0.75 0.75 0.75 0.75