HHMM at SemEval-2019 Task 2: Unsupervised Frame Induction using Contextualized Word Embeddings Saba Anwar?, Dmitry Ustalov†, Nikolay Arefyev‡, §, Simone Paolo Ponzetto†, Chris Biemann?, and Alexander Panchenko?,

?Language Technology Group, Department of Informatics, University of Hamburg, Germany Skolkovo Institute of Science and Technology, Russia {anwar,biemann,panchenko}@informatik.uni-hamburg.de †Data and Web Science Group, University of Mannheim, Germany {dmitry,simone}@informatik.uni-mannheim.de ‡Samsung R&D Institute Russia §Lomonosov Moscow State University, Russia [email protected]

Abstract learning the frame type of the highlighted verb from the context in which it has been used; (B.1) We present our system for semantic frame in- clustering the highlighted arguments of the verb duction that showed the best performance in into specific roles as per the frame type of that Subtask B.1 and finished as the runner-up in Subtask A of the SemEval 2019 Task 2 on un- verb, e.g., Buyer, Goods, etc.; (B.2) clustering supervised semantic frame induction (Qasem- the arguments into generic roles as per VerbNet iZadeh et al., 2019). Our approach separates classes (Schuler, 2005), without considering the this task into two independent steps: verb clus- frame type of the verb, i.e., Agent, Theme, etc. tering using word and their context embed- Our approach to frame induction is similar to dings and role labeling by combining these the word sense induction approach by Arefyev embeddings with syntactical features. A sim- et al.(2018), which uses tf–idf-weighted context ple combination of these steps shows very competitive results and can be extended to pro- word embeddings for a shared task on word sense cess other datasets and languages. induction by Panchenko et al.(2018). In this unsu- pervised task, our approach for clustering mainly 1 Introduction consists of exploring the effectiveness of already available pre-trained models.1 Main contributions Recent years have seen a lot of interest in of this paper are: computational models of frame semantics, with the availability of annotated sources like Prop- 1. a method that uses contextualized distribu- Bank (Palmer et al., 2005) and FrameNet (Baker tional word representations (embeddings) for et al., 1998). Unfortunately, such annotated re- grouping verbs to frame type clusters (Sub- sources are very scarce due to their language and task A); domain specificity. Consequently, there has been 2. a method that combines word and con- work that investigated methods for unsupervised text embeddings for clustering arguments of frame acquisition and parsing (Lang and Lapata, arXiv:1905.01739v1 [cs.CL] 5 May 2019 verbs to frame slots (Subtasks B.1 and B.2). 2010; Modi et al., 2012; Kallmeyer et al., 2018; Ustalov et al., 2018). Researchers have used The key difference of our approach with re- different approaches to induce frames, includ- spect to prior work by Arefyev et al.(2018) and ing clustering verb-specific arguments as per their Kallmeyer et al.(2018) is that we have only used roles (Lang and Lapata, 2010), subject-verb-object pre-trained embeddings to disambiguate the verb triples (Ustalov et al., 2018), syntactic depen- senses and then combined these embeddings with dency representation using dependency formats additional features for semantic labeling of the like CoNLL (Modi et al., 2012; Titov and Kle- verb roles.2 mentiev, 2012), and latent-variable PCFG mod- 1HHMM is an abbreviation for Hansestadt Hamburg, els (Kallmeyer et al., 2018). Mannheim, and Moscow. It is chosen to avoid confusion with hidden Markov models. The SemEval 2019 task of semantic frame and 2Our code is available at https://github.com/ role induction consists of three subtasks: (A) uhh-lt/semeval2019-hhmm. 3 The remainder of the paper is organized as fol- Method Pu F1 B F1 lows. The methodology and the results for each t w2v[C+W]norm 76.68 68.10 subtask are discussed in Sections2,3 and4 re- Ȯ ELMo[C+W]norm 77.03 69.50 spectively, followed by the conclusion in Sec- A Cluster Per Verb 73.78 65.35 tion5. 3 Winner 78.15 70.70

2 Subtask A: Grouping Verbs to Frame Table 1: Our results on Subtask A: Grouping Verbs to Frame Type Clusters. Purity F1-score is denoted as Pu Type Clusters 3 F1, B-Cubed F1-score is denoted as B F1. t denotes In this subtask, each sentence has a highlighted our final submission (# 536426), Ȯ denotes our post- verb, which is usually the predicate. The goal is to competition result, A denotes a baseline, and 3 de- notes the submission of the winning team. label each highlighted verb according to the frame evoked by the sentence. The gold standard for this subtask is based on the FrameNet (Baker et al., 1998) definitions for frames. on TensorFlow Hub.3 Among all the layers of this model, we used the mean-pooling layer for word 2.1 Method and context embeddings. Since sentences evoking the same frame should receive the same labels, we used a verb cluster- 2.2 Results and Discussion ing approach and experimented with a number of We experimented with different clustering algo- pre-trained word and sentence embeddings mod- rithms provided by scikit-learn (Pedregosa et al., els, namely Word2Vec (Mikolov et al., 2013), 2011), namely agglomerative clustering, DB- ELMo (Peters et al., 2018), Universal Sentence SCAN, and affinity propagation. After the model Embeddings (Conneau et al., 2017), and fast- selection on the development dataset, we have Text (Bojanowski et al., 2017). This setup is sim- chosen agglomerative clustering for further eval- ilar to treating the frame induction task as a word uation. Although both ELMo and Word2Vec sense disambiguation task (Brown et al., 2011). showed the best results on the development dataset We experimented with embedding different lex- with single linkage, we opted average linkage after ical units, such as verb (V), its sentence (context, analyzing t-SNE plots (van der Maaten and Hin- C), subject-verb-object (SVO) triples, and verb ar- ton, 2008). guments. Combination of context and word rep- Table1 shows our results obtained on Sub- resentations (C+W) from Word2Vec and ELMo task A. Our final submission (t) used agglom- turned out to be the best combination in our case. erative clustering of normalized vectors obtained We used the standard Google News Word2Vec by concatenating the context and verb vectors embedding model by Mikolov et al.(2013). Since from the Word2Vec model. In particular, we this model is trained on individual words only and found that the best performance is attained for the SemEval dataset contained phrasal verbs, such Manhattan affinity and 150 clusters. During our as fall back and buy out, we have considered only post-competition experiments (Ȯ), we found that the first word in the phrase. If this word is not ELMo performed better than Word2Vec when a present in the model vocabulary, we fall back to higher number of clusters, 235, was specified. a zero-filled vector. When aggregating a context into a vector, we used the tf–idf-weighted average 3 Subtask B.1: Clustering Arguments of of the word embeddings for this context as pro- Verbs to Frame-Specific Slots posed by Arefyev et al.(2018). We tuned these In this subtask, each sentence has a set of high- weights on the development dataset. lighted nouns or noun phrases corresponding to We used the ELMo contextualized embedding the slots of the evoked frame. Additionally, each model by Peters et al.(2018) that generates sentence is provided with the same highlighted vectors of a whole context. Similarly to fast- verb as in Subtask A (Section2). The goal is Text (Bojanowski et al., 2017), ELMo can produce to label each highlighted verb according to the character-level word representations to handle out- evoked frame and to assign each highlighted to- of-vocabulary words. In all our experiments we used the same pre-trained ELMo model available 3https://tfhub.dev/google/elmo/2 3 Method Pu F1 B F1 4 Subtask B.2: Clustering Arguments of Agglomerative Clustering Verbs to Generic Roles Subtask A: w2v[C+W] t 62.10 49.49 Subtask B.2: ID In Subtask B.2, similarly to Subtask B.1 (Sec- Logistic Regression Subtask A: w2v[C+W]norm tion3), each sentence has a set of highlighted  66.81 55.61 Subtask B.2: ELMo[C+W+V]+ID+B+123 nouns or noun phrases that correspond to the slots Subtask A: ELMo[C+W]norm Ȯ 68.22 58.61 Subtask B.2: w2v[C+W+V]+ID+B+123 of the evoked frame. The goal is to label each A Cluster Per Dependency Role 57.99 45.79 highlighted token with a high-level generic class, 3 Winner 62.10 49.49 such as Agent or Patient. However, unlike Sub- task B.1, the verb frame labeling part is omitted. Table 2: Our results on Subtask B.1: Clustering Ar- The gold standard for this subtask is annotated as guments of Verbs to Frame-Specific Slots. Purity F - 1 according to the VerbNet classes (Schuler, 2005). score is denoted as Pu F1, B-Cubed F1-score is denoted as B3 F . t denotes our final submission (# 535483), 1 4.1 Method  denotes a supervised Logistic Regression submission that does not comply to the task rules, Ȯ denotes our When addressing this subtask, we experimented post-competition result, A denotes a baseline, and 3 with combining the embeddings of the word (W) denotes the submission of the winning team. filling the role, its sentence (context, C), and the highlighted verb (V). To handle the out-of- vocabulary roles in the case of Word2Vec embed- ken a frame-specific semantic role identifier. The dings, each role was tokenized and embeddings gold standard for this subtask is annotated with for each token were averaged. If a token is still not FrameNet frames and roles (Baker et al., 1998). present in the vocabulary, then a zero-filled vector was used as its embedding. During prototyping we developed several features that improved the 3.1 Method performance score, namely inbound dependencies (ID), which represent the dependency label from Since Subtask B.1 asks to assign role labels to the head to the role (dependent) and two trivial highlighted tokens as per the frame type of the baselines: Boolean (B) and 123. verb, we attempted this by merging the output of We built a negative one-hot encoding feature verb frame types from Subtask A (Section2) and vector to represent the inbound dependencies of the output of generic role labels from Subtask B.2 the word corresponding to the role. Thus, for each (Section4). We used UKN (unknown) slot identi- dependency of the given role (in case of a multi- fier for the tokens present in Subtask B.1, but miss- word expression), we fill -1 if the dependency re- ing in Subtask B.2. lationship holds, otherwise 0 is filled. During our experiments for the development test, we also used 3.2 Results and Discussion the outbound dependencies, which represent the dependency label from the role (head) to the de- Table2 shows the results from merging our so- pendent words. So we used -1 for inbound and 1 lutions for Subtasks A and B.2, as described in for outbound. But since they did not perform well Sections2 and4, correspondingly. For our final in comparison to inbound dependencies, they were submission (t), we merged the frame types ob- not considered for submitted runs. tained by clustering the Word2Vec embeddings of For the Boolean baseline, given the position of the sentence (context, C) and verb (word, W), and the verb in the sentence pv and the position of the the role labels obtained by clustering the vector target token pt, we assign the role 0 to t if pv < pt, of inbound dependencies (ID). However, we ob- otherwise 1. For the 123 baseline, we assign its served that the logistic regression model demon- index to each highlighted slot filler. For example, strated better performance in Subtask B.2 than any if five slots need to be labelled, the first one will be clustering technique we tried, including our final labelled as 1 and the last one will be labelled as 5. submission (t) and the baselines. But this per- formance was further improved by combining the 4.2 Results and Discussion results from post-competition experiments of Sub- Table3 shows our results on Subtask B.2. We task A and Subtask B.2 (Ȯ). found that the trivial Boolean approach outper- 3 Method Pu F1 B F1 formance of the underlying embedding models. Agglomerative Clustering As the model was trained on the development t w2v[C]+ID 62.00 42.10 dataset that contained 20 roles in contrast to the Ȯ ELMo[C]+ID 50.37 34.89 test set which contained 32 roles, this approach Logistic Regression has its limitations due to this difference of the  ELMo[C+W+V]+ID+B+123 73.14 57.37 number and meaning of roles. We believe that Ȯ w2v[C+W+V]+ID+B+123 74.36 58.83 the performance could be improved using semi- Cluster Per Dependency Role 56.05 39.03 supervised clustering methods, yet during proto- A Boolean Baseline 67.16 46.78 typing with the pairwise-constrained k-Means al- Inbound Dependencies (ID) 66.05 45.77 gorithm (Basu et al., 2004) we did not observe any 3 Winner 64.16 45.65 performance improvements. Table 3: Our results on Subtask B.2: Clustering Ar- 5 Conclusion guments of Verbs to Generic Roles. Purity F1-score is denoted as Pu F1, B-Cubed F1-score is denoted as 3 We presented an approach for unsupervised se- B F1. t denotes our final submission (# 535480),  denotes a supervised Logistic Regression submission mantic frame and role induction that uses word that does not comply to the task rules, Ȯ denotes our and context embeddings. It separates the task into post-competition result, A denotes a baseline, and 3 two independent steps: verb clustering and role denotes the submission of the winning team. labelling, using combination of these embeddings enhanced with syntactical features. Our approach formed LPCFG (Kallmeyer et al., 2018) and all showed the best performance in Subtask B.1 and the standard baselines, including cluster per de- also finished as the runner-up in Subtask A of this pendency role (OneClustPerGrType), on the shared task, and it can be easily extended to pro- development dataset.4 cess other datasets and languages. Similarly to our solution for Subtask A (Sec- Acknowledgments tion2), we tried different clustering algorithms to cluster arguments of verbs to generic roles We acknowledge the support of the Deutsche For- and found that the best clustering performance is schungsgemeinschaft (DFG) under the “JOIN-T” shown by agglomerative clustering with Euclidean and “ACQuA” projects and the German Academic affinity, Ward’s method linkage, and two clusters. Exchange Service (DAAD). We thank the Sem- Our final submission (t) used the combination Eval organizers for an inspiring shared task and of inbound dependencies and Word2Vec embed- their quick responses to all our questions. We are ding for sentence (context, C), which performed grateful to four anonymous reviewers who offered marginally better than the cluster per dependency useful comments. Finally, we thank the Linguis- role (OneClustPerGrType) baseline, but still tic Data Consortium (LDC) for the provided Penn not better than such trivial baselines as Boolean or Treebank dataset (Marcus et al., 1993). 123. Replacing Word2Vec with ELMo in our post- competition experiments have lowered the perfor- mance further. References In order to estimate our upper bound of the per- Nikolay Arefyev, Pavel Ermolaev, and Alexander formance, we compared our best-performing clus- Panchenko. 2018. How much does a word weight? tering algorithm, i.e., agglomerative clustering, to Weighting word embeddings for word sense in- a logistic regression model. We found that the duction. In Computational Linguistics and Intel- lectual Technologies: Papers from the Annual In- combination of sentence (context, C), target word ternational Conference “Dialogue”, pages 68–84, (W), and verb (V) vectors, enhanced with our other Moscow, Russia. RSUH. features, shows substantially better results than a simple clustering model (). However, we did not Collin F. Baker, Charles J. Fillmore, and John B. Lowe. observe a noticeable difference between the per- 1998. The Berkeley FrameNet Project. In Pro- ceedings of the 36th Annual Meeting of the Associ- 4On the development dataset for Subtask B.2, the Boolean ation for Computational Linguistics and 17th Inter- baseline demonstrated B-Cubed F1 = 57.98, while LPCFG national Conference on Computational Linguistics and cluster per dependency role yielded F1 = 40.05 and - Volume 1, ACL ’98, pages 86–90, Montreal,´ QC, F1 = 50.79, correspondingly. Canada. Association for Computational Linguistics. Sugato Basu, Arindam Banerjee, and Raymond J. Martha Palmer, Daniel Gildea, and Paul Kingsbury. Mooney. 2004. Active Semi-Supervision for Pair- 2005. The Proposition Bank: An Annotated Cor- wise Constrained Clustering. In Proceedings of the pus of Semantic Roles. Computational Linguistics, 2004 SIAM International Conference on Data Min- 31(1):71–106. ing, SDM 2004, pages 333–344, Lake Buena Vista, FL, USA. SIAM. Alexander Panchenko, Anastasia Lopukhina, Dmitry Ustalov, Konstantin Lopukhin, Nikolay Arefyev, Piotr Bojanowski, Edouard Grave, Armand Joulin, and Alexey Leontyev, and Natalia Loukachevitch. 2018. Tomas Mikolov. 2017. Enriching Word Vectors with RUSSE’2018: A Shared Task on Word Sense In- Subword Information. Transactions of the Associa- duction for the Russian Language. In Computa- tion for Computational Linguistics, 5:135–146. tional Linguistics and Intellectual Technologies: Pa- pers from the Annual International Conference “Di- Susan Windisch Brown, Dmitriy Dligach, and Martha alogue”, pages 547–564, Moscow, Russia. RSUH. Palmer. 2011. VerbNet Class Assignment as a WSD Task. In Proceedings of the Ninth Inter- Fabian Pedregosa, Gael¨ Varoquaux, Alexandre Gram- national Conference onComputational Semantics, fort, Vincent Michel, Bertrand Thirion, et al. 2011. IWCS 2011, pages 85–94, Oxford, UK. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825–2830. Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo¨ıc Barrault, and Antoine Bordes. 2017. Supervised Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Learning of Universal Sentence Representations Gardner, Christopher Clark, Kenton Lee, and Luke from Natural Language Inference Data. In Proceed- Zettlemoyer. 2018. Deep Contextualized Word Rep- ings of the 2017 Conference on Empirical Methods resentations. In Proceedings of the 2018 Conference in Natural Language Processing, pages 670–680, of the North American Chapter of the Association Copenhagen, Denmark. Association for Computa- for Computational Linguistics: Human Language tional Linguistics. Technologies, Volume 1 (Long Papers), NAACL- HLT 2018, pages 2227–2237, New Orleans, LA, Laura Kallmeyer, Behrang QasemiZadeh, and Jackie USA. Association for Computational Linguistics. Chi Kit Cheung. 2018. Coarse Lexical Frame Ac- quisition at the Syntax–Semantics Interface Using a Behrang QasemiZadeh, Miriam R. L. Petruck, Regina Latent-Variable PCFG Model. In Proceedings of the Stodden, Laura Kallmeyer, and Marie Candito. Seventh Joint Conference on Lexical and Computa- 2019. SemEval-2019 Task 2: Unsupervised Lexi- tional Semantics, *SEM 2018, pages 130–141, New cal Frame Induction. In Proceedings of the 13th In- Orleans, LA, USA. Association for Computational ternational Workshop on Semantic Evaluation, Min- Linguistics. neapolis, MN, USA. Association for Computational Linguistics. Joel Lang and Mirella Lapata. 2010. Unsupervised Induction of Semantic Roles. In Human Lan- Karin Kipper Schuler. 2005. VerbNet: A Broad- guage Technologies: The 2010 Annual Conference coverage, Comprehensive Verb Lexicon. Ph.D. the- of the North American Chapter of the Association sis, University of Pennsylvania, Philadelphia, PA, for Computational Linguistics, pages 939–947, Los USA. Angeles, CA, USA. Association for Computational Linguistics. Ivan Titov and Alexandre Klementiev. 2012.A Bayesian Approach to Unsupervised Semantic Role Laurens van der Maaten and Geoffrey Hinton. 2008. Induction. In Proceedings of the 13th Conference of Visualizing Data using t-SNE. Journal of Machine the European Chapter of the Association for Compu- Learning Research, 9:2579–2605. tational Linguistics, EACL 2012, pages 12–22, Avi- Mitchell P. Marcus, Mary Ann Marcinkiewicz, and gnon, France. Association for Computational Lin- Beatrice Santorini. 1993. Building a Large Anno- guistics. tated Corpus of English: The Penn Treebank. Com- Dmitry Ustalov, Alexander Panchenko, Andrei Kutu- putational Linguistics, 19(2):313–330. zov, Chris Biemann, and Simone Paolo Ponzetto. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S. Cor- 2018. Unsupervised Semantic Frame Induction us- rado, and Jeffrey Dean. 2013. Distributed Represen- ing Triclustering. In Proceedings of the 56th Annual tations of Words and Phrases and their Composition- Meeting of the Association for Computational Lin- ality. In Advances in Neural Information Processing guistics (Volume 2: Short Papers), ACL 2018, pages Systems 26, pages 3111–3119. Curran Associates, 55–62, Melbourne, VIC, Australia. Association for Inc., Harrahs and Harveys, NV, USA. Computational Linguistics. Ashutosh Modi, Ivan Titov, and Alexandre Klementiev. 2012. Unsupervised Induction of Frame-Semantic Representations. In Proceedings of the NAACL-HLT Workshop on the Induction of Linguistic Structure, pages 1–7, Montreal,´ QC, Canada. Association for Computational Linguistics.