Multi-Sense Language Modelling
Andrea Lekkas1,2,3, Peter Schneider-Kamp1, Isabelle Augenstein2 1Dept. of Mathematics and Computer Science, University of Southern Denmark 2Ordbogen A/S, Odense, Denmark 3Dept. of Computer Science, University of Copenhagen, Denmark {lekkas|petersk}@imada.sdu.dk, [email protected]
Abstract Sentence Meaning The effectiveness of a language model is in- “John sat on the bank.n.01: sloping land, espe- fluenced by its token representations, which bank of the river cially the slope beside a body of and watched the water must encode contextual information and han- currents” dle the same word form having a plurality of “Jane went to bank.n.02: a financial institu- meanings (polysemy). Currently, none of the the bank to dis- tion that accepts deposits and common language modelling architectures ex- cuss the mort- channels the money into lending plicitly model polysemy. We propose a lan- gage” activities guage model which not only predicts the next word, but also its sense in context. We argue Table 1: Example of polysemy. Senses taken from that this higher prediction granularity may be WordNet 3.0 useful for end tasks such as assistive writing, and allow for more a precise linking of lan- Ruder, 2018), ELMo (Peters et al., 2018), and all guage models with knowledge bases. We find that multi-sense language modelling requires Transformer architectures. However, even for con- architectures that go beyond standard language textual embeddings, polysemy is handled in an models, and here propose a structured predic- implicit, non-discrete way: the sense that a word tion framework that decomposes the task into assumes in a particular occurrence is unspecified. a word followed by a sense prediction task. To Here, we propose the task of multi-sense lan- aid sense prediction, we utilise a Graph Atten- guage modelling, consisting of not only word, but tion Network, which encodes definitions and also sense prediction. We conjecture that multi- example uses of word senses. Overall, we find that multi-sense language modelling is a sense language modelling would: highly challenging task, and suggest that fu- 1) improve the precision of linking a language ture work focus on the creation of more anno- tated training datasets. model to a knowledge base, as done in Logan et al.(2019), to help generate factually cor- 1 Introduction rect language. For instance: “The explorers Any variant of language model, whether standard descended in the cave and encountered a bat” left-to-right, masked (Devlin et al., 2019) or bidi- refers to the entity ‘bat (animal)’ and not to rectional (Peters et al., 2018) has to address the ‘bat (baseball implement)’. 2) be useful in applications such as assistive writ-
arXiv:2012.05776v2 [cs.CL] 13 Sep 2021 problem of polysemy: the same word form having multiple meanings, as seen in Tab.1. The meaning ing (Chen et al., 2012), where it is desirable of a particular occurrence depends on the context, to display more information about a word to a and all modern language modelling architectures user, such as its sense, definition or usage. from simple RNNs (Mikolov et al., 2010) to Trans- Another potential use would be to explore if formers (Vaswani et al., 2017) use context-based such dictionary information could improve stan- representations. However, token representations in dard language modelling, and reduce the number language models are not explicitly disambiguated. of training data needed, relevant for e.g. language Single-prototype embeddings, i.e., traditional word modelling for low-resource languages. vectors, have a 1-to-1 correspondence with word Consequently, our research objectives are to: forms. Contextual embeddings change depending on the tokens in their context window, and are em- • model next-sense prediction as a task along- ployed in recent models like ULMFit (Howard and side standard next-word prediction in lan- guage modelling, and examine the perfor- multi-sense embeddings in downstream tasks such mance of different model architectures. as Semantic Relatedness and PoS tagging. • encode sense background knowledge from sense definitions and examples, and examine Supervised multi-sense embeddings Other how it can aid sense prediction. models rely on sense-label supervision in a text corpus. context2vec (Melamud et al., 2016) As a sense inventory, we use WordNet 3.0 builds contextual word embeddings by applying (Miller, 1995). The sense background knowledge a biLSTM on text, and provides the option to is encoded in a dictionary graph, as shown in Fig. create sense embeddings by using a sense-labelled 3. When reading a word w, the model can rely on corpus. Raganato et al.(2017) frames WSD as a an additional input signal: the state of the node that sequence learning problem, with the aim of finding represents w in the dictionary graph (the “global sense labels for an input sequence. The training node”). Node vectors in the graph are updated by a corpus is the same used in our work, SemCor Graph Attention Network (Velickoviˇ c´ et al., 2018). (Miller et al., 1993), and the core architecture Findings: We find that sense prediction is a sig- is a biLSTM that reads both the preceding and nificantly more difficult task than standard word subsequent context. A biLSTM is also employed prediction. A way to tackle it is to use a structured in LSTMEmbed (Iacobacci and Navigli, 2019), prediction framework, where the next sense de- that obtains a sense-labelled training corpus by pends on the prediction of the next word. The most applying the BabelFly (Moro et al., 2014) sense successful model we identified for this uses a hard tagger on the English Wikipedia and other texts. cut-off for the number of words considered (Se- Most recently, SenseBERT (Levine et al., 2020) lectK, see § 4.4). The additional input signal from uses WordNet supersenses (categories like food, the dictionary graph provides only marginal im- artifact, person, etc.) to add a form of sense- provements. For future work, we argue that larger prediction to BERT: an additional objective re- sense-labelled datasets would go a long way to- quires the model to predict one of the supersenses wards improving the overall performance of multi- that the word w can assume. sense language models. Sense representations can leverage WordNet glosses, as done in Chen et al.(2014) after single- 2 Related Work prototype vectors are trained with a skip-gram We here briefly discuss relevant works that disam- model. Likewise, pre-trained word embeddings biguate between word senses to address polysemy. are the starting point for AutoExtend (Rothe and They can be grouped in three categories: 1) multi- Schütze, 2015), an autoencoder architecture where prototype embeddings not connected to a knowl- word embeddings constitute the input and the target edge base; 2) supervised multi-sense embeddings whereas the embeddings for WordNet synsets are based on a text corpus that only utilise a KB tan- the intermediate encoded representation. Kumar gentially as the sense inventory; and 3) models that et al.(2019) use a BiLSTM and self-attention to get rely more substantially on features from a KB, like contextual embeddings, then disambiguate them glosses or semantic relations. via a dot product with sense embeddings based on WordNet definitions. Multi-prototype embeddings Huang et al. (2012) learn multi-prototype vectors by clustering KB-based methods In the last couple of years, word context representations. Single-prototype efforts have been made to enable BERT to disam- embeddings are determined by 2 FF-NNs with biguate between senses. GlossBERT (Huang et al., a margin objective on predicting the next word, 2019) takes in context-gloss pairs as input: for ev- a quasi-language modelling setting even if it ery target word in a context sentence, N=4 WordNet utilises both the preceding and subsequent context. glosses are found; a classification layer determines Multi-sense skip-gram (Neelakantan et al., 2014) which lemma the word assumes. SenseEmBERT also defines senses as cluster centroids, measuring (Scarlini et al., 2020a) relies on the BabelNet map- the cosine distance of the surrounding context of ping between WordNet and Wikipedia pages to ±5 words. Li and Jurafsky(2015) use Chinese collect relevant text for synsets, and computes the Restaurant Processes to decide whether to create a sense embeddings as a rank-weighted average of new cluster, and also investigate the usefulness of relevant synsets. Scarlini et al.(2020b) applies the same pipeline of of context extraction, synset em- beddings and sense embeddings construction: it computes lemma representations via BERT, and uses UKB (Agirre et al., 2014) to create a set of contexts for the synset. Sense embeddings are ob- tained by concatenating the BERT representation of the sense contexts found in SemCor and the sense definition in WordNet. Our model We use pre-trained embeddings and WordNet glosses and relations to create a dictionary graph. The vectors of sense nodes can be viewed as sense embeddings, although we are not primarily interested in their quality since our main objective Figure 1: Overview of the SelectK version of the model is multi-sense language modelling. Future work that uses a Transformer-XL for the StandardLM task. may rely on them to improve sense disambiguation: for model variants that have to choose the correct sense among a limited number of candidates, there is an opening for the application of more complex multi-sense models than the ones explored here.
3 Multi-Sense Language Model 3.1 Architecture A language modelling task decomposes the proba- bility of predicting an entire text of length N as the product of each word prediction, where the proba- bility of the next word p(wi) is influenced by the preceding context [w1, ..., wi−1]: Figure 2: The input signals for graph-informed sense N prediction: the standard word embedding and the vec- Y tor of the global node from the dictionary graph p(w1, ..., wN ) = p(wi|w1, ..., wi−1) (1) i=1
Our model aims to carry out two language mod- to examine the effectiveness of sense architectures elling tasks: independently from the accuracy of the standard 1. Standard language modelling: next-token pre- language modelling task. diction at the granularity of words. The input signal for the sense prediction 2. Sense prediction: next-token prediction at the architecture consists of FastText pre-trained granularity of WordNet senses. embeddings (Bojanowski et al., 2017), possibly augmented by concatenating the graph node Therefore, the objective is to produce, at each corresponding to the current word, as shown in Fig. position in the corpus, two probability distributions: 2. Moreover, a model variant that uses structured one over the vocabulary of words and one over the prediction also relies on the K most likely next vocabulary of senses. words predicted by the standard language modeling The architecture for the standard language mod- task: w1 , . . . , wK . An example of a structured elling task is either a 3-layer GRU followed by a t+1 t+1 prediction variant, SelectK (§ 4.4), is shown in Fig. FF-NN, or an 8-layer Transformer-XL (Dai et al., 1. 2019). Both are pre-trained on WikiText-2 (Merity et al., 2016), and then the whole model is trained on the SemCor sense-labeled corpus (Miller et al., The correctness of the prediction is evaluated 1993). Some experiments use a Gold LM, that al- using two measures: perplexity and accuracy (see ways predicts the correct next word, with the aim Appendix, §A.4). 3.2 Dictionary Graph The purpose of the dictionary graph is to provide a way for the input signal to be informed by the sense representations, which constitute the state of sense nodes, and possibly to leverage WordNet’s graph structure and glosses. First, we read in a text corpus and create the vocabulary. We register the lemmatised version of inflected forms, later connected to their parent Figure 3: Part of the dictionary graph for the word “bank”. Global nodes are in yellow, sense nodes in forms. Then, for each sense of a word, we retrieve green, definitions and examples in light blue and blue. the text of its definitions and examples, and regis- ter the connections with synonyms and antonyms. t+1 In WordNet, senses are specified as the synsets of state hi of node i in a graph with m nodes. First, a word; example: w=’bank’ → [(’bank.n.01’), ... a linear transformation W is applied over all the t t ,(’bank.v.07)] . nodes: Wh1, ..., Whm. Then, for each node j The next step is to compute the sentence embed- in the neighbourhood of i, we compute the non- dings for definitions and examples. It is possible normalised attention coefficient eij, via a 1-layer to use any one of several methods, ranging from FF-NN with LeakyReLU activation: LSTMs to BERT’s last layer. For the sake of com- T putation speed, sentence embeddings are obtained eij = LeakyReLU(A [Whi, Whj]) (2) as the average of the FastText word vectors. The normalised attention coefficients α are ob- Finally, the nodes are initialised and stored in a ij tained by applying a softmax over the neighbour- graph with their edges. The graph object is created hood of i, N(i). Finally, the new state of node i with PyTorch-Geometric (Fey and Lenssen, 2019); is given by applying a non-linear function ρ to the before training, it holds the initial values for the weighted sum of the neighbours’ states: sense and global nodes. The sense node starts as the average of the embeddings of definitions and examples for that sense. As shown in Fig.3, every t+1 X t hi = ρ αijWhj (3) sense node is directly connected to its definition j∈N(i) and example nodes. Those determine its starting po- sition; ideally, during training, it would be moved Velickoviˇ c´ et al.(2018) report that using multiple towards the more significant glosses. The global attention heads, by averaging or by concatenating, node is initialised as the FastText embedding for is beneficial to stabilise the model; we therefore w. use 2 concatenated heads.
3.3 Graph Attention Network 3.4 Dealing with Missing Sense Labels We employ a Graph Attention Network (Velickoviˇ c´ Even in a sense-labelled corpus, some of the words, et al., 2018) to update the nodes of the dictionary such as stopwords, will not have a sense label. This graph. Unlike Graph Convolutional Networks may cause a problem because in some versions of (Kipf and Welling, 2017), a GAT does not require a the model (§ 4.4, §4.5, §4.6), the GRU used for fixed graph structure, and can operate on different the sense prediction task should be able to read the local graph-batches like the neighbourhood of input text as an uninterrupted sequence of sense the current word. Unlike other methods like tokens. In particular, two types of words may have graphSAGE (Hamilton et al., 2017) it can handle no sense specification: a variable number of neighbours. The underlying 1. stopwords: ’for’, ’and’, ’of’, etc. idea of GATs is to compute the representation 2. inflected forms: ’is’, ’said’, ’sports’, etc. vector hi of node i based on its neighbouring nodes j ∈ N (i), which have different attention In order to provide stopwords with a weights (i.e. importance). sense label, we add a dummySense (e.g. ‘for.dummySense.01’) for all the words without a We here describe how a GAT obtains the new sense. The corresponding graph node is initialised Words Senses distribution over the vocabulary, where the most PPL Accuracy PPL Accuracy likely K words are w1, ..., wK . 2 GRUs 160.94 0.209 562.46 0.053 Every word wi has a set of senses: S(wi) = 2 Transformer-XL 128.99 0.241 186.72 0.217 {si1, ..., siN }. The next sense at location t is cho- Table 2: Baselines: two separate GRUs, and two sep- sen among the senses of the K most likely words: arate Transformer-XL models. Results of word and K sense prediction on SemCor’s test set. [ s(t) ∈ S(wi) (4) i=1 as the single-prototype FastText vector for that A softmax function is applied over the logits of the word. Inflected forms are lemmatised using selected senses, while all other senses are assigned NLTK’s WordNetLemmatizer, to be able to a probability = 10−8. The senses’ logits are read and predict the senses of their parent form computed by a dedicated GRU, as described in (’is’→’be’, ’said’→’say’ etc.) Figure1. Alternatively, a Transformer-XL could 4 Architectures for Sense Prediction be used, but it would presumably require more training data than the SemCor corpus alone. As previously described, multi-sense language K is a hyperparameter; K=1 means that the modelling consists of the two tasks of standard lan- model always chooses among the senses of the guage modelling and sense prediction, and we aim most likely word. In general, the sense prediction to output two probability distributions - for the next performance depends on the performance of the word token and the next sense token, respectively. standard language model: if all the K most likely globals are incorrect, the correct sense cannot be 4.1 GRU retrieved. We verify what happens for K={1,5,10}. A 3-layer GRU followed by a 1-layer FF-NN. There are no shared layers with the standard lan- 4.5 Sense Context Similarity guage modelling architecture, whether that be a Another structured prediction method is to select Transformer-XL or another GRU. In the latter case, the senses of the most likely K globals as candi- the 2 GRUs share the input signal, i.e. the FastText dates; then, rank them based on the cosine simi- embeddings, possibly concatenated with the word larity between the local context and each sense’s node state. average context. Since language modelling is per- formed from left to right, the context is based only 4.2 Transformer-XL on the preceding tokens [wt−1, ..., wt−c] without Transformer-XL is a left-to-right transformer en- considering subsequent tokens. coder that operates at word-level granularity. It For each occurrence s1, ...sN of a sense s, the was chosen instead of a BERT model because the occurrence context OC(si) is computed as the av- latter uses masked language modelling with Word- erage of the word embeddings of the preceding c Piece sub-word tokenisation, and thus cannot eas- tokens. Afterwards, the Sense Context SC(s) is ily be adapted to a setting where the the next word computed as the average of the occurrences’ con- and its sense should be predicted. In our setting, texts: the Transformer-XL learns word embeddings from scratch and does not rely on the FastText vectors. OC(si) = avg(wt−1, ..., wt−c) (5) SC(s) = avg(OC(s ), ..., OC(s )) 4.3 Most Frequent Sense 1 N This heuristic baseline chooses the most frequent We also experiment with obtaining a representa- sense found in the training set for the most likely tion of the local context with a 3-layer GRU. word predicted by the standard language model. 4.6 Self-Attention Coefficients 4.4 SelectK Another way to choose among the candidate senses SelectK is a structured prediction approach: choos- of the most likely K globals is to use the softmax ing the next sense depends on the prediction of the scores from the self-attention mechanism. Every next word. As the text is read, for every location t, sense s has an average context SC(s) it appears the standard language model outputs a probability in, as seen in Eq.5. The contexts of the candidate Methods & Parameters SemCor test SensEval dataset Senses Standard Senses Polysem Globals Senses Polysem Globals architecture LM K context ACC ACC ACC ACC ACC ACC MFS Gold - - 0.83 0.62 1 0.82 0.62 1 SelectK Gold 1 - 0.90 0.80 1 0.82 0.61 1 SenseContext Gold 1 average 0.73 0.45 1 0.71 0.41 1 Self-attention Gold 1 average 0.52 0.24 1 0.51 0.25 1 MFS TXL - - 0.24 0.02 0.24 0.23 0.04 0.23 TXL TXL - 0.22 0.02 0.24 0.20 0.03 0.22 SelectK TXL 1 - 0.24 0.02 0.24 0.23 0.03 0.23 SelectK TXL 5 - 0.13 0.01 0.24 0.11 0.01 0.22 SenseContext TXL 1 average 0.22 0.01 0.24 0.22 0.02 0.23 SenseContext TXL 1 GRU 0.19 0.00 0.24 0.17 0.00 0.23 SenseContext TXL 5 average 0.13 0.01 0.24 0.12 0.01 0.22 Self-attention TXL 1 average 0.17 0.01 0.24 0.16 0.02 0.23 Self-attention TXL 5 average 0.08 0.01 0.24 0.07 0.01 0.23 MFS GRU -- 0.22 0.02 0.22 0.22 0.03 0.22 SelectK GRU 1 - 0.22 0.02 0.22 0.22 0.02 0.21 SenseContext GRU 1 average 0.20 0.00 0.22 0.20 0.00 0.22 Self-attention GRU 1 average 0.16 0.01 0.22 0.16 0.02 0.22 SelectK GRU 5 - 0.13 0.00 0.21 0.12 0.01 0.21 GRU GRU - - 0.05 0.00 0.21 0.06 0.00 0.21
Table 3: The most relevant results of each method. Task: sense prediction. Datasets: SemCor’s test split (10%) and the aggregated SensEval dataset by Raganato et al.(2017). senses are collected in the matrix C. Then, a prob- The graph area is defined expanding outwards from ability distribution over the senses is obtained by the global node of the current word w. In our ex- computing the self-attention coefficients: periments, a graph area contains a maximum of 32 nodes and extends for only 1 hop, thus coincid- Q · C softmax √ (6) ing with the neighbourhood of the current word’s dk global node.
All the rows of the query matrix Q are represen- 5.2 Model variants tations of the current context. As previously, the local context can be constructed as a simple aver- In Tab.3 and4, we show the results of sense age of the last c word embeddings, or as the output predictions with different architectures and of a 3-layer GRU. The sense contexts in C take parameters. up the role of keys in the formula of self-attention scores. Sense prediction architectures:
5 Evaluation • GRU: a 3-layer GRU followed by FF-NN • TXL: an 8-layer Transformer-XL 5.1 Dataset and Graph Settings • MFS: Given the most likely word predicted To train a multi-sense language model, we need a by the standard LM, choose its most frequent sense-labelled text corpus. We use SemCor (Miller sense in the training set (see § 4.3). et al., 1993), a subset of the Brown Corpus labelled • SelectK: A GRU computes the logits over with senses from WordNet 3.0. Training, valida- the senses’ vocabulary, but the softmax is tion and test sets are obtained with a 80/10/10 split. restricted to the candidate senses only: the Since standard LM architectures are pre-trained senses of the most likely K words from the on WikiText-2, the vocabulary is obtained from standard LM task (see § 4.4). WikiText-2 and SemCor’s training split (the latter • SenseContext: Choosing among the candi- with min. frequency=2). The dictionary graph has date senses based on the cosine similarity of ≈169K nodes and ≈254K edges. Consequently, local context and average sense context (see § due to memory constraints, we apply mini-batching 4.5). on the graph: for each input instance the Graph At- • Self-Attention: Choosing among the candi- tention Network operates on a local graph area. date senses by computing the self-attention Methods & Parameters SemCor test set SensEval dataset Senses Standard Senses Senses architecture LM K context ACC DG ∆ PPL DG ∆ ACC DG ∆ PPL DG ∆ SelectK TXL 1 - 0.24 0 119.6 -0.1 0.23 0 172.54 -1.34 SelectK TXL 5 - 0.13 0 121.0 0 0.11 0 162.2 +0.3 SenseContext TXL 1 average 0.22 0 118.9 +1.2 0.22 0 172.4 +8.0 SenseContext TXL 5 average 0.13 0 135.8 +0.5 0.12 0 213.2 +2.1 Self-attention TXL 1 average 0.17 +0.01 111.9 -7.3 0.17 +0.01 155.7 -16.8 Self-attention TXL 5 average 0.08 0 120.6 +1.3 0.08 +0.01 161.7 -4.6 GRU GRU - - 0.05 0 148.5 -12.5 0.06 0 157.7 -5.8 SelectK GRU 1 - 0.22 0 131.7 -1.6 0.22 +0.01 153.1 -3.6 SelectK GRU 5 - 0.13 -0.01 148.5 -12.1 0.11 -0.01 157.7 -5.6 SenseContext GRU 1 average 0.21 +0.01 129.5 -3.7 0.21 +0.01 153.3 -3.4 SenseContext GRU 5 average 0.09 +0.01 141.9 -18.5 0.10 +0.02 182.0 +18.7 Self-attention GRU 1 average 0.16 +0.01 128.1 -10.3 0.17 +0.01 154.0 -2.7 Self-attention GRU 5 average 0.08 0.00 139.8 -20.7 0.07 0.00 154.8 -9.5
Table 4: Word and sense prediction. DG ∆ = change caused by including the input from the dictionary graph.
coefficients of local context and average sense is extremely reliant on the correctness of the word contexts (see § 4.6). prediction task. On the aggregated dataset of the SemEval-SensEval tasks (Raganato et al., 2017), Tab.2 compares the simplest methods available: picking the most frequent sense is a difficult base- using either two GRUs or two Transformer-XL for line to beat (with Senses ACC=0.82 and Polysem the two tasks of word and sense prediction. While ACC=0.62), but SelectK with K=1 is extremely the Transformer-XL performs better than the GRU, close. these are not the best results: further experiments We found that increasing K to 5 or 10 leads to a shed light on the sense prediction model variants. worse performance, due to the increased difficulty of choosing among a greater number of candidate 5.3 Results senses. Using a Transformer-XL as Standard LM Overall Results In Tab.3, we compare the re- gives a small improvement over the GRU. The con- sults of the different methods for sense prediction. text representation made from the average of the We report the Senses ACC, accuracy on the senses last 20 tokens is better than the one created by a of all words, and the Polysem ACC: accuracy on dedicated GRU trained on SemCor. the senses of polysemous words; words that have more than 1 sense are an important part of any Predictions for Polysemous Words All non- multi-sense task, and are expected to be more dif- Gold model variants have very low accuracy when ficult. We include the accuracy for standard word it comes to predicting the right sense of a polyse- prediction, Globals ACC, because in our structural mous word. This is due to polysemous words being prediction setting, if the next word is incorrect, the most difficult ones to predict correctly in the none of its candidate senses will be correct. standard language modelling task, and therefore in We evaluate sense prediction using accuracy a structured prediction framework. instead of perplexity, since in structured predic- This is corroborated by observing the word pre- tion methods, perplexity values are non-significant dictions of the Transformer-XL standard language (≈ 108), due to having assigned = 10−8 as the model. The first batch of the SensEval dataset probability of the non-candidate senses. has 130 correct predictions out of 512 samples, Experiments with the Gold standard language where the correct words and their frequency are: model allow us to examine the methods’ capability {‘of’:10, ‘the’:27, ‘
Thomas N. Kipf and Max Welling. 2017. Semi- References Supervised Classification with Graph Convolutional Networks. In Proceedings of the 5th International Eneko Agirre, Oier López de Lacalle, and Aitor Soroa. Conference on Learning Representations, ICLR ’17. 2014. Random walks for knowledge-based word sense disambiguation. Computational Linguistics, Sawan Kumar, Sharmistha Jat, Karan Saxena, and 40(1):57–84. Partha Talukdar. 2019. Zero-shot word sense dis- ambiguation using sense definition embeddings. In Piotr Bojanowski, Edouard Grave, Armand Joulin, and ACL, pages 5670–5681, Florence, Italy. Tomas Mikolov. 2017. Enriching word vectors with subword information. TACL, 5:135–146. Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Mei-Hua Chen, Shih-Ting Huang, Hung-Ting Hsieh, Shashua, and Yoav Shoham. 2020. SenseBERT: Ting-Hui Kao, and Jason S. Chang. 2012. FLOW: A Driving some sense into BERT. In ACL, pages first-language-oriented writing assistant system. In 4656–4667, Online. Proceedings of the ACL 2012 System Demonstra- tions, pages 157–162, Jeju Island, Korea. Associa- Jiwei Li and Dan Jurafsky. 2015. Do multi-sense em- tion for Computational Linguistics. beddings improve natural language understanding? In EMNLP, pages 1722–1732, Lisbon, Portugal. Xinxiong Chen, Zhiyuan Liu, and Maosong Sun. 2014. A unified model for word sense representation and Robert Logan, Nelson F. Liu, Matthew E. Peters, Matt disambiguation. In EMNLP, pages 1025–1035, Gardner, and Sameer Singh. 2019. Barack’s wife Doha, Qatar. hillary: Using knowledge graphs for fact-aware lan- guage modeling. In ACL, pages 5962–5971, Flo- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Car- rence, Italy. bonell, Quoc V. Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond Oren Melamud, Jacob Goldberger, and Ido Dagan. a fixed-length context. CoRR, abs/1901.02860. 2016. context2vec: Learning generic context em- bedding with bidirectional LSTM. In Proceedings Jacob Devlin, Ming-Wei Chang, Kenton Lee, and of The 20th SIGNLL Conference on Computational Kristina Toutanova. 2019. BERT: Pre-training of Natural Language Learning, pages 51–61, Berlin, deep bidirectional transformers for language under- Germany. standing. In NAACL, pages 4171–4186, Minneapo- lis, Minnesota. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- Matthias Fey and Jan E. Lenssen. 2019. Fast graph els. CoRR, abs/1609.07843. representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Graphs and Manifolds. Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTER- Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. SPEECH, pages 1045–1048. ISCA. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, George A. Miller. 1995. Wordnet: A lexical database volume 30, pages 1024–1034. Curran Associates, for english. Communications of the ACM, 38:39–41. Inc. George A. Miller, Claudia Leacock, Ee Tengi, and Jeremy Howard and Sebastian Ruder. 2018. Universal Ross T. Bunker. 1993. A semantic concordance. language model fine-tuning for text classification. In In Proceedings ARPA Human Language Technology ACL, pages 328–339, Melbourne, Australia. Workshop, pages 303–308.
Eric Huang, Richard Socher, Christopher Manning, Andrea Moro, Alessandro Raganato, and Roberto Nav- and Andrew Ng. 2012. Improving word represen- igli. 2014. Entity linking meets word sense disam- tations via global context and multiple word proto- biguation: a unified approach. TACL, 2:231–244. types. In ACL, pages 873–882, Jeju Island, Korea. Arvind Neelakantan, Jeevan Shankar, Alexandre Pas- Luyao Huang, Chi Sun, Xipeng Qiu, and Xuanjing sos, and Andrew McCallum. 2014. Efficient non- Huang. 2019. GlossBERT: BERT for word sense parametric estimation of multiple embeddings per disambiguation with gloss knowledge. In EMNLP- word in vector space. In EMNLP, pages 1059–1069, IJCNLP, pages 3509–3514, Hong Kong, China. Doha, Qatar. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Module name Version used Gardner, Christopher Clark, Kenton Lee, and Luke gensim 4.0.1 Zettlemoyer. 2018. Deep contextualized word repre- h5py 3.1.0 sentations. In NAACL, pages 2227–2237, New Or- langid 1.1.6 leans, Louisiana. lxml 4.6.3 nltk 3.6.2 Alessandro Raganato, Claudio Delli Bovi, and Roberto numpy 1.19.5 Navigli. 2017. Neural sequence learning models pandas 1.1.5 for word sense disambiguation. In EMNLP, pages scikit-image 0.17.2 1156–1167, Copenhagen, Denmark. scikit-learn 0.24.2 scipy 1.5.4 Sascha Rothe and Hinrich Schütze. 2015. AutoEx- tables 3.6.1 tend: Extending word embeddings to embeddings torch 1.5.0+cu101 for synsets and lexemes. In ACL-IJCNLP, pages torch-cluster 1.5.4 torch-geometric 1.4.3 1793–1803, Beijing, China. torch-scatter 2.0.4 torch-sparse 0.6.2 Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. torch-spline-conv 1.2.0 2020a. Sensembert: Context-enhanced sense em- transformers 4.5.1 beddings for multilingual word sense disambigua- Proceedings of the AAAI Conference on Arti- tion. Table 5: Software dependencies ficial Intelligence, 34(05):8758–8765.
Bianca Scarlini, Tommaso Pasini, and Roberto Navigli. 2020b. With More Contexts Comes Better Perfor- For the SemCor corpus, we read all the tokens with mance: Contextualized Sense Embeddings for All- frequency=1 as
A Reproducibility Checklist A.2 Computing Infrastructure A.1 Dataset and Pre-Processing GPU model: Each experiment was executed on 1 NVidia GeForce RTX 2080 Ti. This work makes use of two corpora: GPU memory: 10.76 GB. • WikiText-2: text corpus for pre-training CPU(s): 20 x IntelCore i9-9820X CPU @ the standard language modelling architecture, 3.30GHz. whether GRU or Transformer-XL. Tokenized RAM: 125 GB via NLTK’s sentence tokenizer OS: Ubuntu 18.04 LTS (Bionic) # training tokens= 2,052,960 A.3 Software Dependencies # validation tokens= 213,920 Python version: 3.6.13 • SemCor: sense-labeled corpus for training CUDA Version: 10.1 the Multi-sense language models. Stored in The code is included in the attached ZIP folder an .xml file in UFSAC notation, with sense Multisense-LM_code.zip. The necessary steps and labels from WordNet 3.0 scripts to run experiments are specified in the # training tokens = 646,038 README file. To be able to execute the code, # validation tokens = 80,760 it is necessary to install several packages, shown in # test tokens = 78,453 Tab.5 A.4 Evaluation Measures The hyperparameters were chosen to optimize the A.4.1 Perplexity validation PPL on WikiText-2, compatibly with the memory capacity of the GPUs (10.76GB). Given a test set T = (w1, w2, w3, ...wN ), and a language model L that assigns a probability P (wi) • batch size={1, 2, 4} to each word wi in a given position, the perplexity of L on T is defined as the inverse probability of • sequence length={128, 256, 512} encountering the words in a test set, normalised by • learning rate={1e-6, 1e-5, 2e-5, 1e-4} the number of words in the set: s B Further Experiments 1 1 1 PP (T ) = = N 1/N Tables6,7,9,8 can be found at the end of the P (w1w2...wn) P (w1w2...wn) appendix. (7)
This measure coincides with elevating 2 to the Shannon entropy value H(p) for the probability distribution:
H(p) − P p(w )·log (p(w )) PP (T ) = 2 = 2 i i 2 i (8)
A.4.2 Accuracy The number of correct predictions divided by the correct total number of tokens to predict, i.e. total A.5 Experimental Settings A.5.1 GRU The GRUs employed for standard language mod- elling and for sense prediction have the same archi- tecture: 3 layers [1024→1024→512], followed by a Linear FF-NN to produce the logits. A grid-search is used to find reasonable hyper- parameters:
• batch size={20, 32, 40}
• sequence length={35,70}
• learning rate={1e-5, 5e-5, 1e-4}
A.5.2 Transformer-XL The configuration of the Transformer-XL tool is obtained by applying several modifications to the version found in the transformers library, that be- longs to the model pre-trained on WikiText-103. The architecture must be scaled down, to prevent overfitting on a corpus of limited size like WikiText- 2 + SemCor. Consequently:
• dimensionality of embeddings = {300, 400, 512} (down from 512)
• d_model={512, 768} (down from 1024)
• d_inner={512, 768, 1024} (down from 1024) Parameters SemCor test set SenseEval Dataset Standard Senses Polysem Globals Globals Senses Polysem Globals Globals K Graph LM ACC ACC ACC PPL ACC ACC ACC PPL Gold 1 0.898 0.801 1 1 0.817 0.607 1 1 Gold 1 w/ graph 0.891 0.784 1 1 0.816 0.606 1 1 Gold 5 0.437 0.315 1 1 0.411 0.256 1 1 Gold 5 w/ graph 0.436 0.315 1 1 0.411 0.256 1 1 TXL 1 0.236 0.023 0.244 119.7 0.227 0.034 0.226 173.9 TXL 1 w/ graph 0.237 0.024 0.245 119.6 0.226 0.035 0.226 172.5 TXL 5 0.128 0.006 0.238 121 0.112 0.005 0.223 161.9 TXL 5 w/ graph 0.129 0.007 0.239 121.0 0.111 0.005 0.229 162.2 GRU 1 0.22 0.018 0.224 133.2 0.217 0.023 0.213 156.7 GRU 1 w/ graph 0.223 0.017 0.227 131.7 0.222 0.027 0.221 153.1 GRU 5 0.132 0.004 0.209 160.6 0.122 0.006 0.21 163.3 GRU 5 w/ graph 0.125 0.007 0.218 148.5 0.114 0.005 0.216 157.7
Table 6: Results of the SelectK method.
Methods & Parameters SemCor test set SensEval dataset Senses Standard Senses Polysem Globals Globals Senses Polysem Globals Globals Graph architecture LM ACC ACC ACC PPL ACC ACC ACC PPL MFS Gold 0.828 0.624 1 1 0.821 0.619 1 1 TXL Gold 0.219 0.02 1 1 0.203 0.025 1 1 GRU Gold 0.053 0 1 1 0.058 0 1 1 MFS TXL 0.236 0.023 0.244 118.7 0.225 0.037 0.225 172.2 TXL TXL 0.217 0.021 0.241 129 0.201 0.027 0.221 197.5 MFS GRU 0.22 0.018 0.224 133.2 0.217 0.028 0.217 156.7 GRU GRU 0.053 0 0.209 160.9 0.058 0 0.21 163.5 GRU GRU w/ graph 0.053 0 0.218 148.5 0.058 0 0.217 157.7
Table 7: Results of the methods that do not use structured prediction (MFS, TXL, GRU). Parameters SemCor test set SenseEval Dataset Standard Senses Polysem Globals Globals Senses Polysem Globals Globals K Graph Context LM ACC ACC ACC PPL ACC ACC ACC PPL Gold 1 average 0.727 0.446 1 1 0.708 0.405 1 1 Gold 1 w/ graph average 0.729 0.448 1 1 0.709 0.404 1 1 Gold 5 average 0.413 0.188 1 1 0.435 0.178 1 1 TXL 1 average 0.223 0.013 0.243 117.7 0.220 0.022 0.226 164.4 TXL 1 w/ graph average 0.223 0.013 0.244 118.9 0.218 0.22 0.225 172.4 TXL 1 GRU 0.187 0.002 0.242 118.9 0.171 0.002 0.225 166.4 TXL 5 average 0.127 0.008 0.240 135.3 0.121 0.011 0.219 211.1 TXL 5 w/ graph average 0.13 0.008 0.239 135.76 0.123 0.011 0.22 213.19 GRU 1 average 0.204 0.004 0.224 133.2 0.196 0.004 0.217 156.7 GRU 1 w/ graph average 0.211 0.007 0.228 129.5 0.211 0.013 0.222 153.3 GRU 1 GRU 0.041 0 0.21 160.4 0.038 0 0.21 163.3 GRU 5 average 0.082 0.004 0.21 160.4 0.077 0.003 0.21 163.3 GRU 5 w/ graph average 0.13 0.009 0.226 140.9 0.127 0.014 0.214 183.5
Table 8: Results of the SenseContext method.
Parameters SemCor test set SenseEval Dataset Standard Senses Polysem Globals Globals Senses Polysem Globals Globals K Graph Context LM ACC ACC ACC PPL ACC ACC ACC PPL Gold 1 average 0.515 0.235 1 1 0.512 0.245 1 1 Gold 1 w/ graph average 0.509 0.235 1 1 0.493 0.25 1 1 Gold 5 average 0.207 0.088 1 1 0.23 0.101 1 1 Gold 5 w/ graph average 0.209 0.094 1 1 0.229 0.104 1 1 TXL 1 average 0.166 0.014 0.244 119.1 0.164 0.023 0.226 172.5 TXL 1 w/ graph average 0.171 0.015 0.247 111.9 0.172 0.024 0.231 155.7 TXL 1 GRU 0.154 0.004 0.243 119.1 0.142 0.003 0.226 171.9 TXL 5 average 0.078 0.008 0.242 119.3 0.071 0.008 0.225 166.3 TXL 5 w/ graph average 0.080 0.007 0.239 120.6 0.076 0.007 0.224 161.7 GRU 1 average 0.155 0.011 0.220 138.4 0.159 0.017 0.216 156.8 GRU 1 w/ graph average 0.162 0.009 0.228 128.1 0.165 0.016 0.222 154.0 GRU 5 average 0.070 0.004 0.21 160.4 0.075 0.006 0.21 164.2 GRU 5 w/ graph average 0.075 0.005 0.222 139.8 0.073 0.005 0.219 154.8
Table 9: Results of the Self Attention Scores method.