St. Cloud State University theRepository at St. Cloud State Culminating Projects in Computer Science and Department of Computer Science and Information Technology Information Technology 5-2020 Knowledge Graphs and Knowledge Graph Embeddings Catherine Tschida [email protected] Follow this and additional works at: https://repository.stcloudstate.edu/csit_etds Recommended Citation Tschida, Catherine, "Knowledge Graphs and Knowledge Graph Embeddings" (2020). Culminating Projects in Computer Science and Information Technology. 32. https://repository.stcloudstate.edu/csit_etds/32 This Starred Paper is brought to you for free and open access by the Department of Computer Science and Information Technology at theRepository at St. Cloud State. It has been accepted for inclusion in Culminating Projects in Computer Science and Information Technology by an authorized administrator of theRepository at St. Cloud State. For more information, please contact [email protected]. 1 Knowledge Graphs and Knowledge Graph Embeddings by Catherine Tschida A Starred Paper Submitted to the Graduate Faculty of St. Cloud State University in Partial Fulfillment of the Requirements for the Degree Master of Science in Computer Science May, 2020 Starred Paper Committee: Andrew Anda, Chairperson Richard Sundheim Bryant Julstrom 2 Abstract Knowledge graphs provide machines with structured knowledge of the world. Structured, machine-readable knowledge is necessary for a wide variety of artificial intelligence tasks such as search, translation, and recommender systems. These knowledge graphs can be embedded into a dense matrix representation for easier usage and storage. We first discuss knowledge graph components and knowledge base population to provide the necessary background knowledge. We then discuss popular methods of embedding knowledge graphs in chronological order. Lastly, we cover how knowledge graph embeddings improve both knowledge base population and a variety of artificial intelligence tasks. 3 Table of Contents Page List of Tables .......................................................................................................... 6 List of Figures ......................................................................................................... 7 List of Equations ..................................................................................................... 8 Chapter 1. Introduction .................................................................................................. 9 2. Knowledge Graph Components ................................................................... 13 2.1 Entity ................................................................................................... 13 2.2 RDF ..................................................................................................... 15 2.3 Different Knowledge Graphs ............................................................... 17 2.4 Adding Additional Information to the RDF Structure ............................ 20 2.5 Section Summary ................................................................................ 22 3. Knowledge Base Population ........................................................................ 24 3.1 Supervised Learning ............................................................................ 24 3.2 Syntactical Preprocessing Steps ......................................................... 25 3.3 Entity Disambiguation .......................................................................... 26 3.4 Relation Extraction .............................................................................. 27 3.5 Long Short-Term Memory (>STM) ...................................................... 28 3.6 Categorical Data and Keras ................................................................ 31 3.7 Preventing Overfit ................................................................................ 31 3.8 Noise in Extracting Data from Unstructured Text ................................. 32 3.9 Judging Accuracy of Techniques ......................................................... 32 4 Chapter Page 4. Embedding Background Knowledge ............................................................ 34 4.1 Initial Input ........................................................................................... 34 4.2 Relationship Categories ...................................................................... 35 4.3 Negative Sampling .............................................................................. 36 4.4 Loss Functions .................................................................................... 36 5. Embedding Types ........................................................................................ 39 5.1 RESCAL .............................................................................................. 40 5.2 Word2Vec Embeddings ....................................................................... 41 5.3 TransE Embeddings ............................................................................ 42 5.4 TransM Embeddings ........................................................................... 44 5.5 HolE Embeddings ................................................................................ 45 5.6 ComplEx Embeddings ......................................................................... 46 5.7 Spectrally Trained HolE ....................................................................... 48 6. The Use of Embeddings .............................................................................. 50 6.1 Abbreviation Disambiguation ............................................................... 50 6.2 Relation Extraction .............................................................................. 50 6.3 Link Prediction / Reasoning ................................................................. 51 6.4 Classifying Entities as Instances of a Class ........................................ 51 6.5 Language Translation .......................................................................... 51 6.6 Recommender Systems ...................................................................... 52 6.7 Question Answering ............................................................................ 52 7. Conclusion ................................................................................................... 53 5 Chapter Page References ........................................................................................................ 55 6 List of Tables Table Page 2.1 Date and creation method of knowledge graphs ......................................... 17 2.2 A comparison of knowledge graph components .......................................... 18 4.1 Embeddings and their loss functions ........................................................... 38 5.1 Knowledge graph embeddings .................................................................... 39 5.2 Mean hit at 10 for TransE and TransM on freebase15k .............................. 45 5.3 Comparison of TransE, RESCAL, and HolE ................................................ 46 5.4 Comparison of TransE, HolE, and ComplEx ................................................ 48 7 List of Figures Figure Page 2.1 KG showing information on the entity URI: Hamlet ...................................... 14 2.2 Google’s knowledge graph card .................................................................. 15 2.3 DBpedia entry for Michael Schumacher ...................................................... 20 2.4 <Mikołaj of Ściborz, date of death, unknown value> ........ 21 3.1 Information extraction flowchart ................................................................... 25 3.2a LSTM Forget gate .................................................................................... 29 3.2b LSTM Input gate ...................................................................................... 29 3.2c LSTM Output gate ..................................................................................... 29 4.1 Embedding input .......................................................................................... 35 5.1 3CosAdd for a-a'+b≠b' ............................................................................... 42 5.2 TransE embedding with translation vector r ................................................ 43 5.3 Modeling 1:n relations in TransE vs. TransM ............................................... 44 8 List of Equations Equation Page 4.1 Pointwise square error loss ......................................................................... 38 4.2 Pairwise hinge loss ...................................................................................... 38 4.3 Pointwise hinge loss .................................................................................... 38 4.4 Pointwise logistic loss .................................................................................. 38 5.1 Scoring function of RASCAL ........................................................................ 41 5.2 ComplEx scoring function ............................................................................ 47 5.3 The spectrally trained HolE scoring function ............................................... 49 9 Chapter 1: Introduction Knowledge graphs contain knowledge of the world in a format that is usable to computers. A knowledge graph is a directed graph. The nodes of the graph represent named objects such as Abraham Lincoln, concepts such as Gravity, or literal