Generating Lexical Representations of Frames using Lexical Substitution Saba Anwar Artem Shelmanov Universitat¨ Hamburg Skolkovo Institute of Science and Technology Germany Russia [email protected] [email protected] Alexander Panchenko Chris Biemann Skolkovo Institute of Science and Technology Universitat¨ Hamburg Russia Germany [email protected] [email protected] Abstract Seed sentence: I hope PattiHelper can helpAssistance youBenefited party soonTime . Semantic frames are formal linguistic struc- Substitutes for Assistance: assist, aid tures describing situations/actions/events, e.g. Substitutes for Helper: she, I, he, you, we, someone, Commercial transfer of goods. Each frame they, it, lori, hannah, paul, sarah, melanie, pam, riley Substitutes for Benefited party: me, him, folk, her, provides a set of roles corresponding to the sit- everyone, people uation participants, e.g. Buyer and Goods, and Substitutes for Time: tomorrow, now, shortly, sooner, lexical units (LUs) – words and phrases that tonight, today, later can evoke this particular frame in texts, e.g. Sell. The scarcity of annotated resources hin- Table 1: An example of the induced lexical represen- ders wider adoption of frame semantics across tation (roles and LUs) of the Assistance FrameNet languages and domains. We investigate a sim- frame using lexical substitutes from a single seed sen- ple yet effective method, lexical substitution tence. with word representation models, to automat- ically expand a small set of frame-annotated annotated resources. Some publicly available re- sentences with new words for their respective sources are FrameNet (Baker et al., 1998) and roles and LUs. We evaluate the expansion PropBank (Palmer et al., 2005), yet for many lan- quality using FrameNet. Contextualized mod- guages and domains, specialized resources do not els demonstrate overall superior performance exist. Besides, due to the inherent vagueness compared to the non-contextualized ones on of frame definitions, the annotation task is chal- roles. However, the latter show comparable performance on the task of LU expansion. lenging and requires semanticists or very complex crowd-sourcing setups (Fossati et al., 2013). 1 Introduction We suggest a different perspective on the prob- lem: expanding the FrameNet resource automati- The goal of lexical substitution (McCarthy and cally by using lexical substitution. Given a small Navigli, 2009) is to replace a given target word set of seed sentences with their frame annotations, in its context with meaning-preserving alterna- we can expand it by substituting the targets (words tives. In this paper, we show how lexical sub- corresponding to lexical units of the respective stitution can be used for semantic frame expan- frame) and arguments (words corresponding to sion. A semantic frame is a linguistic structure roles of the respective frame) of those sentences used to describe the formal meaning of a situa- and aggregating possible substitutions into an in- tion or event (Fillmore, 1982). Semantic frames duced frame-semantic resource. Table1 shows have witnessed a wide range of applications; such one such induced example. For this purpose, as question answering (Shen and Lapata, 2007; we have experimented with state-of-the-art non- Berant and Liang, 2014; Khashabi et al., 2018), contextualized (static) word representation mod- machine translation (Gao and Vogel, 2011; Zhai els including neural word embeddings, i.e. fast- et al., 2013), and semantic role labelling (Do et al., Text (Bojanowski et al., 2017), GloVe (Pennington 2017; Swayamdipta et al., 2018). The impact, et al., 2014), and word2vec (Mikolov et al., 2013); however, is limited by the scarce availability of and distributional thesaurus, i.e. JoBimText (Bie- 95 Proceedings of the Probability and Meaning Conference (PaM 2020), pages 95–103 October 14-15 2020 c 2020 Association for Computational Linguistics mann and Riedl, 2013); and compared their results WordNet-based methods to automatically induce with contextualized word representations of the new LUs and reported their results on FrameNet. state-of-the-art BERT model (Devlin et al., 2019), Our method is inspired by the recent work of which has set a new benchmark performance on Amrami and Goldberg(2018). They suggest to many downstream NLP applications. To complete predict the substitutes vectors for target words us- the comparison, we also include the lexical substi- ing pre-trained ELMo (Peters et al., 2018) and dy- tution model of Melamud et al.(2015), which uses namic symmetric patterns, then induced the word dependency-based word and context embeddings senses using clustering. Arefyev et al.(2019) takes and produces context-sensitive lexical substitutes. the idea of substitute vectors from (Amrami and To generate substitutes, we decompose the Goldberg, 2018) for the SemEval 2019 (Qasem- problem into two sub-tasks: Lexical unit expan- iZadeh et al., 2019) frame induction task and re- sion: Given a sentence and its target word, the task places ELMo with BERT (Devlin et al., 2019) for is to generate frame preserving substitutes for this improved performance. Zhou et al.(2019) show word. Frame role expansion: Given a sentence the utility of BERT for the lexical substitution and an argument, the task is to generate meaning- task. Lexical substitution has been used for a preserving substitutes for this argument. range of NLP tasks such as paraphrasing or text Contributions of our work are (i) a method for simplification, but here, we are employing it, as inducing frame-semantic resources based on a few far as we are aware, for the first time to perform frame-annotated sentences using lexical substitu- expansion of frame-semantic resources. tion, and (ii) an evaluation of various distribu- tional semantic models and lexical substitution 3 Inducing Lexical Representations of methods on the ground truth from FrameNet. Frames via Lexical Substitution 2 Related Work We experimented with two groups of lexical sub- stitution methods. The first one use no con- Approaches to semantic frame parsing with re- text: non-contextualized neural word embed- spect to a pre-defined semantic frame resource, ding models, i.e. fastText (Bojanowski et al., such as FrameNet, have received much atten- 2017), GloVe (Pennington et al., 2014), and tion in the literature (Das et al., 2010; Oepen word2vec (Mikolov et al., 2013), as well as dis- et al., 2016; Yang and Mitchell, 2017; Peng et al., tributional thesaurus based models in the form of 2018), with SEMAFOR (Das et al., 2014) be- JoBimText (Biemann and Riedl, 2013). The sec- ing a most widely known system to extract com- ond group of methods does use the context: here, plete frame structure including target identifica- we tried contextualized word embedding model tion. Some works focus on identifying par- BERT (Devlin et al., 2019) and the lexical substi- tial structures such as frame identification (Hart- tution model of Melamud et al.(2015). mann et al., 2017; Hermann et al., 2014), role labelling with frame identification (Swayamdipta 3.1 Static Word Representations et al., 2017; Yang and Mitchell, 2017), and sim- These word representations models are inherently ple role labelling (Kshirsagar et al., 2015; Roth non-contextualized as they learn one representa- and Lapata, 2015; Swayamdipta et al., 2018), tion of a word regardless of its context. which is considered very similar to standard Prop- Bank (Palmer et al., 2005) style semantic role la- Neural Word Embeddings Neural word em- belling, albeit more challenging because of the beddings represent words as vectors of continu- high granularity of frame roles. These super- ous numbers, where words with similar meanings vised models rely on a dataset of frame-annotated are expected to have similar vectors. Thus, to pro- sentences such as FrameNet. FrameNet-like re- duce substitutes, we extracted the k nearest neigh- sources are available only for very few languages bors using a cosine similarity measure. We use and cover only a few domains. In this paper, we pre-trained embeddings by authors models: fast- venture into the inverse problem, the case where Text trained on the Common Crawl corpus, GloVe the number of annotations is insufficient, simi- trained on Common Crawl corpus with 840 billion lar to the idea of Pennacchiotti et al.(2008) who words, word2vec trained on Google News. All investigated the utility of semantic spaces and these models produce 300-dimension vectors. 96 Distributional Thesaurus (DT) In this ap- implementation by Wolf et al.(2019). We produce proach, word similarities are computed using com- a substitute word distribution without masking and plex linguistic features such as dependency rela- selected substitutes with top k probabilities. tions (Lin, 1998). The representations provided by DTs are sparser, but similarity scores based 4 Experimental Setup on them can be better. JoBimText (Biemann and Riedl, 2013) is a framework that offers many DTs 4.1 Datasets computed on a range of different corpora. Context We experimented with FrameNet (Baker et al., features for each word are ranked using the lexi- 1998) version 1:7. It contains around 170k sen- cographer’s mutual information (LMI) score and tences annotated with 1; 014 frames, 7; 878 types used to compute word similarity by feature over- of frame roles, and 10; 340 lexical units. Frame lap. We extract the k nearest neighbors for the tar- roles and LUs can consist of a single token or mul- get word. We use two JoBimText DTs: (i) DT tiple tokens. For this work, we have only con- built on Wikipedia with n-grams as contexts and sidered a single-token substitution. The datasets (ii) DT built on a 59G corpus (Wikipedia, Giga- for evaluation were derived automatically from word, ukWaC, and LCC corpora combined) using FrameNet. To create a gold standard for LU ex- dependency relations as context. pansion task, for each sentence containing an an- 3.2 Contextualized Models notated LU, we consider other LUs of the cor- responding semantic frame as ground truth sub- Static word representations fail to handle poly- stitutes.