Phd Thesis, 2016

EXPLOITING LINGUISTIC KNOWLEDGE IN LEXICAL AND COMPOSITIONAL SEMANTIC MODELS by Tong Wang A thesis submitted in conformity with the requirements for the degree of Doctor of Philosophy Graduate Department of Computer Science University of Toronto © Copyright 2016 by Tong Wang Abstract Exploiting Linguistic Knowledge in Lexical and Compositional Semantic Models Tong Wang Doctor of Philosophy Graduate Department of Computer Science University of Toronto 2016 A fundamental principle in distributional semantic models is to use similarity in linguistic environment as a proxy for similarity in meaning. Known as the distributional hypothesis, the principle has been successfully applied to many areas in natural language processing. As a proxy, however, it also suffers from critical limitations and exceptions, many of which are the result of the overly simplistic definitions of the linguistic environment. In this thesis, I hy- pothesize that the quality of distributional models can be improved by carefully incorporating linguistic knowledge into the definition of linguistic environment. The hypothesis is validated in the context of three closely related semantic relations, namely near-synonymy, lexical similarity, and polysemy. On the lexical level, the definition of linguistic environment is examined under three different distributional frameworks including lexical co-occurrence, syntactic and lexicographic dependency, and taxonomic structures. Firstly, combining kernel methods and lexical level co-occurrence with matrix factorization is shown to be highly effective in capturing the fine-grained nuances among near-synonyms. Secondly, syntactic and lexicographic information is shown to result in notable improvement in lexical embedding learning when evaluated in lexical similarity benchmarks. Thirdly, for taxonomy- based measures of lexical similarity, the intuitions for using structural features such as depth and density are examined and challenged, and the refined definitions are shown to improve correlation between the features and human judgements of similarity as well as performances of similarity measures using these features. ii On the compositional level, distributional models of multi-word contexts are also shown to benefit from incorporating syntactic and lexicographic knowledge. Analytically, the use of syntactic teacher-forcing is motivated by derivations of full gradients in long short-term memory units in recurrent neural networks. Empirically, syntactic knowledge helps achieve statistically significant improvement in language modelling and state-of-the-art accuracy in near-synonym lexical choice. Finally, a compositional similarity function is developed to measure the similarity between two sets of random events. Application in polysemy with lexicographic knowledge produces state-of-the-art performance in unsupervised word sense disambiguation. iii In loving memory of Jingli Wang. iv Acknowledgements It’s been a long journey, but nevertheless a fun ride. I thank my supervisor Professor Graeme Hirst for giving me the opportunity, the guidance, as well as the ultimate academic freedom to pursue my various research interests. I thank all the members of my supervisory committee, Professor Gerald Penn, Professor Ruslan Salakhutdinov, and Professor Suzanne Stevenson, for their help, support, and encouragement all along. I thank my external exam- iners Professor Ted Pedersen, Professor Frank Rudzicz, and Professor Sanja Fidler for their valuable time and for the insightful comments and discussions. I’d also like to express my gratitude to Doctor Xiaodan Zhu and Doctor Afsaneh Fazly for reaching out and helping me at the most challenging times in my program, to Doctor Ju- lian Brooke, Doctor Abdel-rahman Mohamed, and Nona Naderi for the collaborations and the thought-provoking discussions, and to all the fellow graduate students in the Computational Linguistics group for this memorable experience. Finally, I’d like to thank my grandmother Fanglan Wu for inspiring me to persist when I was tempted to give up. I also thank my wife Yi Zhang for her unconditional support through the many stressful spans over the years. Without either of you, I could not have completed the degree. Thank you. v Contents 1 Introduction1 1.1 Motivations and Outline.............................1 1.2 The Scope of Semantic Relations.........................4 1.2.1 Synonymy and its Variants........................4 1.2.2 Polysemy and Similarity.........................6 1.3 Distributional Semantic Models.........................8 1.3.1 Definition.................................8 1.3.2 Computational Challenges........................9 I Lexical Representations 13 2 Improving Representation Efficiency in Lexical Distributional Models 14 2.1 Truncation..................................... 15 2.2 Dimension Reduction............................... 16 2.2.1 Latent Semantic Analysis........................ 17 2.2.2 Lexical Embeddings........................... 18 2.2.3 Random Indexing............................. 21 2.3 Semantic Space Clustering............................ 22 3 Near-Synonym Lexical Choice in Latent Semantic Space 25 3.1 Introduction.................................... 25 vi 3.2 Near-synonym Lexical Choice Evaluation.................... 26 3.3 Context Representation.............................. 27 3.3.1 Latent Semantic Space Representation for Context........... 28 3.4 Model Description................................ 29 3.4.1 Unsupervised Model........................... 29 3.4.2 Supervised Learning on the Latent Semantic Space Features...... 30 3.5 Formalizing Subtlety in the Latent Space..................... 32 3.5.1 Characterizing Subtlety with Collocating Differentiator of Subtlety ... 32 3.5.2 Subtlety and Latent Space Dimensionality................ 33 3.5.3 Subtlety and the Level of Context.................... 35 4 Dependency-based Embedding Models and its Application in Lexical Similarity 36 4.1 Using Syntactic and Semantic Knowledge in Embedding Learning....... 36 4.1.1 Relation-dependent Models....................... 37 4.1.2 Relation-independent Models...................... 38 4.2 Optimization................................... 40 4.3 Evaluation..................................... 41 4.3.1 Data Preparation............................. 41 4.3.2 Lexical Semantic Similarity....................... 43 4.3.3 Baseline Models............................. 45 4.3.4 Relation-dependent Models....................... 46 4.3.5 Relation-independent Models...................... 49 4.3.6 Using Dictionary Definitions....................... 50 4.4 Input-output Embedding Co-training....................... 51 5 Refining the Notions of Depth and Density in Taxonomy-based Semantic Similar- ity Measures 54 5.1 Introduction.................................... 54 vii 5.1.1 Taxonomy-based Similarity Measures.................. 55 5.1.2 Semantic Similarity Measures Using Depth and Density........ 56 5.2 Limitations of the Current Definitions...................... 58 5.3 Refining the Definitions of Depth and Density.................. 63 5.4 Applying the New Definitions to Semantic Similarity.............. 65 6 Conclusions for PartI 68 II Semantic Composition 71 7 Learning and Visualizing Composition Models for Phrases 74 7.1 Related Work in Semantic Composition..................... 74 7.2 Phrasal Embedding Learning........................... 78 7.3 Phrase Generation................................. 80 7.4 Composition Function Visualization....................... 81 8 Composition in Language Modelling and Near-synonymy Lexical Choice Using Recurrent Neural Networks 84 8.1 Gradient Analysis in LSTM Optimization.................... 85 8.1.1 Gradients of Traditional RNNs...................... 85 8.1.2 Gradients of LSTM Units........................ 86 8.1.3 Gradient Verification........................... 90 8.1.4 Regulating LSTM Gating Behaviour with Syntactic Constituents.... 91 8.2 Language Modelling Using Syntax-Augmented LSTM............. 93 8.2.1 Model Implementation.......................... 93 8.2.2 Evaluation................................ 94 8.3 Near-synonymy Context Composition Using LSTM............... 95 8.3.1 Task Description and Model Configuration............... 96 viii 8.3.2 Evaluation................................ 97 9 Unsupervised Word Sense Disambiguation Using Compositional Similarity 101 9.1 Introduction.................................... 101 9.2 Related Work................................... 102 9.3 Model and Task Descriptions........................... 104 9.3.1 The Naive Bayes Model......................... 104 9.3.2 Model Application in WSD....................... 106 9.3.3 Incorporating Additional Lexical Knowledge.............. 106 9.3.4 Probability Estimation.......................... 108 9.4 Evaluation..................................... 109 9.4.1 Data, Scoring, and Pre-processing.................... 109 9.4.2 Comparing Lexical Knowledge Sources................. 110 9.4.3 The Effect of Probability Estimation................... 111 10 Conclusions for PartII 113 III Summary 115 11 Summary 116 11.1 Conclusions.................................... 116 11.2 Future Work.................................... 119 11.2.1 Near-synonymy.............................. 119 11.2.2 Lexical Representation Learning..................... 122 11.2.3 Semantic Composition.......................... 122 Bibliography 123 ix List of Tables 3.1 LSA-SVM performance on the FITB task..................... 31 4.1 Hyperparameters for the proposed embedding models.............

Load more