Utrecht University Repository
Total Page:16
File Type:pdf, Size:1020Kb
Utrecht University Master's Thesis Leveraging Static Models for Temporal Knowledge Graph Completion Supervisor: Dr. Melisachew Wudage Chekol Author: [email protected] Wessel Radstok (5692024) [email protected] Secondary supervisor: Prof. Dr. Yannis Velegrakis [email protected] A thesis submitted in partial fulfillment of the requirements for the degree of Master of Science in Computing Science Department of Information and Computing Science January 11, 2021 Utrecht University Abstract The inclusion of temporal information in knowledge graph embedding (KGE) has re- mained relatively unexplored, even though it stands to argue that this would result in better embeddings. Additionally, models that do include the temporal component per- form only marginally better than those that do not (static models). Noting this, we introduce SpliMe, a model-agnostic pre-processor for temporal knowledge graphs that makes it possible to embed them with static models. We show that SpliMe achieves state-of-the-art performance on two datasets commonly used for temporal KGE method evaluation and increases performance with regards to our baseline on another dataset. Furthermore, we uncover problems with existing evaluation procedures for static KGE models on temporal graphs and propose a simple method to fix these issues. Finally, we redefine the link prediction metric for temporal knowledge graph embedding to better suit temporal scope prediction. 2 of 85 Utrecht University Contents 1. Introduction5 2. Preliminaries6 2.1. Knowledge Graphs and the RDF Data Model................6 2.2. Knowledge Graph Construction and Applications..............7 2.3. KG Completion.................................9 2.3.1. Approaches...............................9 2.3.2. Knowledge Graph Embeddings.................... 10 2.4. Model Evaluation................................ 11 2.4.1. Method................................. 11 2.4.2. Datasets................................. 12 2.5. Loss functions.................................. 13 2.6. Sampling.................................... 14 2.7. Entity Evolution and Concept Drift..................... 14 2.7.1. Record Linkage............................. 15 2.7.2. Evolution Summaries......................... 16 2.7.3. Change Point Detection........................ 17 2.8. Network Proximity............................... 18 3. Overview of Knowledge Graph Embedding Models 19 3.1. Translational models.............................. 19 3.1.1. TransE................................. 19 3.1.2. TransH................................. 20 3.2. Tensor factorization.............................. 21 3.2.1. Polyadic Decomposition........................ 22 3.2.2. RESCAL & DistMult......................... 23 3.2.3. ComplEx................................ 24 3.2.4. SimplE................................. 25 3.3. Temporal KG embeddings........................... 26 3.3.1. t-TransE and variants......................... 26 3.3.2. Temporal-Aware Sequence Encoders................. 27 3.3.3. HyTE.................................. 27 3.3.4. Diachronic Embeddings........................ 28 3.3.5. ATiSE.................................. 29 4. Method 30 4.1. Timestamping.................................. 30 4.2. Splitting..................................... 31 4.3. Merging..................................... 32 4.4. Proximity.................................... 35 4.5. Granularity and Temporal Distortion..................... 36 3 of 85 Utrecht University 4.6. Complexity Analysis.............................. 39 4.6.1. SpliMe.................................. 39 4.6.2. Embedding............................... 40 5. Experiments 41 5.1. Data....................................... 41 5.2. Hyperparameters................................ 42 5.3. Entity Prediction................................ 43 5.4. On Evaluating TKGs with Static Methods.................. 43 5.4.1. Filtering................................. 45 5.5. Results...................................... 46 5.5.1. Comparison with other models.................... 48 5.6. Baseline..................................... 49 5.7. Ablation Studies................................ 50 5.7.1. Granularity............................... 50 5.7.2. Embedding size............................. 51 5.8. Qualitative Analysis.............................. 54 5.8.1. Predicate Prediction.......................... 54 5.8.2. t-SNE plots............................... 56 5.9. Temporal Evaluation.............................. 58 5.9.1. Evaluation............................... 59 6. Conclusion 61 6.1. Future Work.................................. 61 References 63 A. Alternative SpliMe Methods 68 A.1. Snapshots.................................... 68 A.2. Embedding Refinement............................ 68 B. Hybrid Graphs: YAGO15k 70 C. SpliMe on Entities 73 D. Full results 77 D.1. Vanilla TransE Results............................. 77 D.2. Timestamp Results............................... 78 D.3. Split (time) Results............................... 79 D.4. Split (count) Results.............................. 80 D.5. Merge Results.................................. 81 D.6. Distance Results................................ 82 D.7. Random Baseline Results........................... 85 4 of 85 Utrecht University 1. Introduction A knowledge graph (KG) is a graph used for representing structured information about the world, which can then be utilized in other tasks such as question answering, predicate extraction and recommender systems. Many large KGs have been created in recent years, such as DBPedia, Google Knowledge Graph and FreeBase. Using the Resource Description Framework (RDF) language created by the World Wide Web Consortium, these facts are stored as (subject, predicate, object) (s,p,o) triples. For instance, (Trump, BornIn, New York). The subject and object are also referred to as entities, and the predicate as the relationship between them. In the graph representation, the entities are nodes, and predicates are labeled directed edges connecting them. Existing knowledge graphs are very large: DBPedia contains around 10 billion triples (Lehmann et al., 2015). Yet, much information is still missing. For example, only 30% of the people in FreeBase have place of birth information (West et al., 2014). However, due to redundancy in the data, it is often possible to recover this information. For instance, we could deduce someones country of birth by looking at his city of birth. Predicting the most likely connection between entities is called link prediction. Using link prediction to fill in missing links in a KG is called knowledge graph completion. Another problem for KGs is dealing with evolving or historic information. Facts that may be true at one point in time might be invalidated later. As an example, (Obama, PresidentOf, USA) was only true between 2009-2017. One approach to solving this issue is by including temporal information with the facts. A temporal knowledge graph (TKG) is a KG where (a subset of) the facts have a temporal scope denoting the time during which they were valid, or the time at which they occurred. An example of the latter is (Obama, awarded, Nobel Peace Prize, 2015 ). Over the past few years there has been much research into how to perform knowledge graph completion. Much research has focussed on latent variable models (Nickel et al., 2011; Bordes et al., 2013; Trouillon et al., 2016). In these models, entities and predicates are represented as vectors, called embeddings. Link prediction is then done through transformations on these embeddings. It stands to reason that including temporal information inside these embeddings would result in improved performance. Yet, few embedding models have been created for this purpose (e.g. (Jiang et al., 2016; Garc´ıa-Dur´anet al., 2018)), and their results are only marginally better than static KG embedding models. Additionally, many TKG embedding models (e.g. (Xu et al., 2019; Goel et al., 2019)) require that all facts in the KG have temporal scopes, even though many KGs are only partially temporal. Noting this, this thesis will aim for the following goals: • Develop a model-agnostic TKG embedding method which utilizes feature engineer- ing to incorporate temporal information at the level of entities and predicates. • Reformulate the link prediction task to suit temporal scope prediction. • Provide a runnable implementation of the system in Python for reproducibility. 5 of 85 Utrecht University 2. Preliminaries 2.1. Knowledge Graphs and the RDF Data Model The first specification of the Resource Description Framework (RDF) was published by the World Wide Web Consortium in 2004, with a followup 1.1 specification made official in 2014. RDF is a general method for modelling information, specifically aimed to be used in web resources. Information is stored as triples which consists of a subject s, a predicate p and an object o, in that order i.e. (s,p,o). Examples of RDF triples are (Utrecht, provinceIn, Netherlands) and (Penrose, winnerOf, Nobel prize). In some literature, a triple is said to be composed of a head, relation and tail. In this thesis we use the terms relation and predicate interchangeably, but stick to the (s,p,o) notation. More formally, we consider a set of entities E = fe1; e2; : : : ; eng and a set of predicates P = fr1; r2; : : : ; rmg. Let jXj denote the cardinality of set X. Then jEj = n and jPj = m. A knowledge graph D is a set of observed RDF triples, i.e. D = f(s; p; o)js; o 2 E; p 2 Pg. Predicates have different cardinality types. That is, a relation