UCLA UCLA Electronic Theses and Dissertations
Total Page：16
File Type：pdf, Size：1020Kb
UCLA UCLA Electronic Theses and Dissertations Title Multi-relational Representation Learning and Knowledge Acquisition Permalink https://escholarship.org/uc/item/373677v6 Author Chen, Muhao Publication Date 2019 Peer reviewed|Thesis/dissertation eScholarship.org Powered by the California Digital Library University of California UNIVERSITY OF CALIFORNIA Los Angeles Multi-relational Representation Learning and Knowledge Acquisition A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Computer Science by Muhao Chen 2019 c Copyright by Muhao Chen 2019 ABSTRACT OF THE DISSERTATION Multi-relational Representation Learning and Knowledge Acquisition by Muhao Chen Doctor of Philosophy in Computer Science University of California, Los Angeles, 2019 Professor Carlo Zaniolo, Chair Multi-relational representation learning methods encode entities or concepts of a knowledge graph in a continuous and low-dimensional vector space, where the relational inferences of entities (con- cepts) are modeled as some simple vector algebras. Despite such knowledge representations being crucial to a wide range of knowledge-driven applications, state-of-the-art methods are limited to learning embeddings for simple relation facts in a single knowledge graph. In this dissertation, we pursue the goal of comprehensively capturing the multifaceted relational knowledge in various types of knowledge bases, and towards that we contribute on three fronts: (i) we introduce the first multi-relational representation learning framework that learns to transfer embeddings across multiple knowledge bases; (ii) we propose techniques for preserving relational facts with complex properties in the embedding space, including those enforce relational properties, form hierarchies, or endowed uncertainty; (iii) we investigate large-scale relational learning based on other modali- ties of data, with the aim of acquiring knowledge to enrich the knowledge bases. Each of these three research problems presents a series of key challenges which we address. Thus, for transferred embeddings, we develop joint learning of relational structure encoders that confront the heterogeneity of contents in knowledge graphs, together with diverse types of align- ment models that learn to transfer on the basis of simple, hierarchical or fuzzy alignment infor- mation. In addition, we extend the joint learning framework with semi-supervised co-training of entity descriptions, and proactive score propagation for fuzzy alignment, so as to conquer the sce- narios where alignment information is limitedly provided. To capture complex relation facts, we ii focus first on the relational properties that cause non-linearity in embedding structures, for which we leverage a non-linear component-specific mappings of embeddings to eliminate the conflicts, and strengthens the learning process with hierarchical regularization. For uncertain relation facts, we preserve the uncertainty by utilizing Probablistic Soft Logic to guide the non-linear regressor that is jointly trained with the structure encoder. We further study the support of relational learn- ing based on sequence data. Our model proposes generic neural sequence pair models to support large-scale relation detection, in which we incorporate different sequence encoders for heteroge- neous data such as structured articles, amino acid sequences, and lexicographic knowledge. The methods proposed in this dissertation extend the application of multi-relational embed- dings, and improve a wide spectrum of applications in different domains. These include knowl- edge alignment, monolingual and cross-lingual knowledge graph completion, semantic search, entity typing, paraphrase identification, uncertain relation prediction, protein-protein interaction prediction, protein binding affinity estimation, single-cell RNA-sequence imputation, and Web- scale sub-article matching. iii The dissertation of Muhao Chen is approved. Junghoo Cho Yizhou Sun Yingnian Wu Carlo Zaniolo, Committee Chair University of California, Los Angeles 2019 iv To life. v Table of Contents 1 Introduction ::::::::::::::::::::::::::::::::::::::: 1 1.1 Motivation . .1 1.2 Challenges . .3 1.3 Thesis Contributions . .5 1.4 Thesis Overview . .7 2 Background ::::::::::::::::::::::::::::::::::::::: 11 2.1 Multi-relational Representation Learning . 11 2.2 Knowledge Bases and Knowledge Graphs . 13 2.2.1 Monolingual and Multilingual Knowledge . 13 2.2.2 Ontology and Instance-level Knowledge Graphs . 13 2.2.3 Comprehensive Properties of Relation Facts . 14 2.2.4 Sequence Data As Side Information . 16 2.3 Neural Sequence Models . 17 2.3.1 The Convolution Layer with Pooling . 17 2.3.2 GRU and Residual GRU . 17 2.3.3 Self-attentive Encoder . 19 2.3.4 Linear Bag-of-words . 20 vi 3 Transfer Embeddings of Multilingual Knowledge Graphs :::::::::::::: 21 3.1 Introduction . 21 3.2 Related Work . 24 3.3 The Vanilla Multilingual Knowledge Graph Embeddings . 25 3.3.1 Multilingual Knowledge Graphs . 25 3.3.2 Knowledge Model . 26 3.3.3 Alignment Model . 26 3.3.4 Variants of MTransE ............................ 28 3.4 Cross-lingual Entity Matching . 30 3.4.1 Examples of Knowledge Alignment . 33 3.5 Semi-supervised Co-training . 35 3.5.1 Iterative Co-training . 37 3.6 Semi-supervised and Zero-shot Entity Alignment . 38 3.6.1 Cross-lingual Entity Alignment . 39 3.6.2 Zero-shot Alignment . 41 3.6.3 Cross-lingual KG Completion . 42 3.7 Conclusion . 43 4 Transfer Embeddings with Complex Alignment Information :::::::::::: 44 4.1 Introduction . 44 4.1.1 Ontology-level Concepts and Instance-level Entities . 44 4.1.2 Single-cell RNA-sequence Data As Fuzzy Alignment . 47 4.2 Joint Embedding of Ontological Concepts and Instance-view Entities . 48 4.2.1 Formalization of Two-view Knowledge Bases . 48 4.2.2 Alignment Model with Hierarchical Grouping Techniques . 49 vii 4.2.3 Knowledge Model . 52 4.2.4 Joint Training on Two Views . 55 4.2.5 Variants of JOIE and Complexity . 55 4.3 Entity Typing and Triple Completion . 56 4.3.1 Datasets . 56 4.3.2 Triple Completion . 57 4.3.3 Entity Typing . 59 4.3.4 Case Study . 61 4.4 Ablation Study . 64 4.4.1 Dimensionality . 65 4.4.2 Sufficiency of Type Information . 66 4.4.3 Effects of Negative Sampling . 67 4.5 Transfer Gene Knowledge Based on Fuzzy Alignment . 68 4.6 Cell Clustering . 70 4.6.1 Dataset . 70 4.6.2 Evaluation . 71 4.7 Conclusion . 72 5 Learning to Capture Relational Properties :::::::::::::::::::::: 74 5.1 Introduction . 74 5.2 Embedding Ontology Graphs With Complex Relational Properties . 76 5.2.1 Preliminary . 76 5.2.2 Modeling . 76 5.2.3 Learning Process . 81 5.3 Experiments . 82 viii 5.3.1 Relation Prediction . 82 5.3.2 Relation Verification . 85 5.3.3 Case Study . 87 5.4 Conclusion . 88 6 Embedding Uncertain Knowledge Graphs :::::::::::::::::::::: 89 6.1 Introduction . 89 6.2 Related Work . 91 6.3 Problem Definition . 92 6.4 Modeling . 92 6.4.1 Modeling Plausibility for Relation Facts . 93 6.4.2 Inferring Confidence Scores for Unseen Relation Facts . 94 6.4.3 Embedding Uncertain Knowledge Graphs . 97 6.4.4 Model variants . 98 6.4.5 Confidence score prediction . 100 6.5 Experiments . 100 6.5.1 Datasets . 100 6.5.2 Model Configurations . 103 6.5.3 Confidence Prediction . 104 6.5.4 Relation Fact Ranking . 105 6.5.5 Relation Fact Classification . 107 6.6 Conclusion and Future Work . 108 7 Large-scale Sub-article Relation Learning :::::::::::::::::::::: 109 7.1 Introduction . 109 ix 7.2 Related Work . 112 7.3 Modeling . 113 7.3.1 Preliminaries . 113 7.3.2 Document Encoders . 114 7.3.3 Explicit Features . 115 7.3.4 Training . 116 7.4 Experiments . 117 7.4.1 Evaluation . 119 7.4.2 Mining Sub-articles from the Entire English Wikipedia . 123 7.5 Conclusion and Future Work . 124 8 Multifaceted Protein-Protein Interaction Learning :::::::::::::::::: 125 8.1.