Question Answering Over Temporal Knowledge Graphs Apoorv Saxena Soumen Chakrabarti Partha Talukdar Indian Institute of Science Indian Institute of Technology Google Research Bangalore Bombay India [email protected] [email protected] [email protected] Abstract time intervals as well. These temporal scopes on facts can be either automatically estimated (Taluk- Temporal Knowledge Graphs (Temporal KGs) extend regular Knowledge Graphs by provid- dar et al., 2012) or user contributed. Several such ing temporal scopes (e.g., start and end times) Temporal KGs have been proposed in the literature, on each edge in the KG. While Question An- where the focus is on KG completion (Dasgupta swering over KG (KGQA) has received some et al. 2018; Garc´ıa-Duran´ et al. 2018; Leetaru and attention from the research community, QA Schrodt 2013; Lacroix et al. 2020; Jain et al. 2020). over Temporal KGs (Temporal KGQA) is a The task of Knowledge Graph Question Answer- relatively unexplored area. Lack of broad- coverage datasets has been another factor lim- ing (KGQA) is to answer natural language ques- iting progress in this area. We address this tions using a KG as the knowledge base. This is challenge by presenting CRONQUESTIONS, in contrast to reading comprehension-based ques- the largest known Temporal KGQA dataset, tion answering, where typically the question is ac- clearly stratified into buckets of structural com- companied by a context (e.g., text passage) and plexity. CRONQUESTIONS expands the only the answer is either one of multiple choices (Ra- known previous dataset by a factor of 340×. jpurkar et al., 2016) or a piece of text from the We find that various state-of-the-art KGQA methods fall far short of the desired perfor- context (Yang et al., 2018). In KGQA, the an- mance on this new dataset. In response, swer is usually an entity (node) in the KG, and the we also propose CRONKGQA, a transformer- reasoning required to answer questions is either based solution that exploits recent advances in single-fact based (Bordes et al., 2015), multi-hop Temporal KG embeddings, and achieves per- (Yih et al. 2015, Zhang et al. 2017) or conjunc- formance superior to all baselines, with an in- tion/comparison based reasoning (Talmor and Be- crease of 120% in accuracy over the next best rant, 2018). Temporal KGQA takes this a step performing method. Through extensive experi- further where: ments, we give detailed insights into the work- ings of CRONKGQA, as well as situations 1. The underlying KG is a Temporal KG. where significant further improvements appear 2. The answer is either an entity or time duration. possible. In addition to the dataset, we have re- 3. Complex temporal reasoning might be needed. leased our code as well. KG Embeddings are low-dimensional dense vec- arXiv:2106.01515v1 [cs.LG] 3 Jun 2021 tor representations of entities and relations in a KG. 1 Introduction Several methods have been proposed in the litera- Temporal Knowledge Graphs (Temporal KGs) are ture to embed KGs (Bordes et al. 2013, Trouillon multi-relational graph where each edge is associ- et al. 2016, Vashishth et al. 2020). These embed- ated with a time duration. This is in contrast to a dings were originally proposed for the task of KG regular KG where no time annotation is present. completion i.e., predicting missing edges in the For example, a regular KG may contain a fact KG, since most real world KGs are incomplete. such as (Barack Obama, held position, President Recently, however, they have also been applied to of USA), while a temporal KG would contain the the task of KGQA where they have been shown to start and end time as well — (Barack Obama, held increase performance the settings of both of com- position, President of USA, 2008, 2016). Edges plete and incomplete KGs (Saxena et al. 2020; Sun may be associated with a set of non-contiguous et al. 2020). Temporal Question Types Dataset KG # questions facts Multi-Entity Multi-Relation Temporal SimpleQuestions FreeBase 7 7 7 0% 108k MetaQA MetaQA KG 7 7 3 0% 400k WebQuestions FreeBase 7 7 3 <16% 5,810 ComplexWebQuestions FreeBase 7 3 3 - 35k TempQuestions FreeBase 7 3 3 100% 1,271 CRONQUESTIONS (ours) WikiData 3 3 3 100% 410k Table 1: KGQA dataset comparison. Statistics about percentage of temporal questions for WebQuestions are taken from Jia et al.(2018a). We do not have an explicit number of temporal questions for ComplexWebQuestions, but since it is constructed automatically using questions from WebQuestions, we expect the percentage to be similar to WebQuestions (16%). Please refer to Section 2.1 for details. Temporal KG embeddings are another upcoming baselines across all question types. CRONKGQA area where entities, relations and timestamps in a achieves very high accuracy on simple temporal temporal KG are embedded in a low-dimensional reasoning questions, but falls short when it comes vector space (Dasgupta et al. 2018, Lacroix et al. to questions requiring more complex reasoning. 2020, Jain et al. 2020, Goel et al. 2019). Here too, Thus, although we get promising early results, the main application so far has been temporal KG CRONQUESTIONS leaves ample scope to improve completion. In our work, we investigate whether complex Temporal KGQA. Our source code along temporal KG Embeddings can be applied to the with the CRONQUESTIONS dataset can be found at task of Temporal KGQA, and how they fare com- https://github.com/apoorvumang/CronKGQA. pared to non-temporal embeddings or off-the-shelf methods without any KG Embeddings. 2 Related work In this paper we propose CRONQUESTIONS, a 2.1 Temporal QA data sets new dataset for Temporal KGQA. CRONQUES- TIONS consists of both a temporal KG and accom- There have been several KGQA datasets proposed panying natural language questions. There were in the literature (Table1). In SimpleQuestions (Bor- three main guiding principles while creating this des et al., 2015) one needs to extract just a single dataset: fact from the KG to answer a question. MetaQA 1. The associated KG must provide temporal an- (Zhang et al., 2017) and WebQuestionsSP (Yih notations. et al., 2015) require multi-hop reasoning, where 2. Questions must involve an element of temporal one must traverse over multiple edges in the KG reasoning. to reach the answer. ComplexWebQuestions (Tal- 3. The number of labeled instances must be large mor and Berant, 2018) contains both multi-hop and enough that it can be used for training models, conjunction/comparison type questions. However, rather than for evaluation alone. none of these are aimed at temporal reasoning, and Guided by the above principles, we present a the KG they are based on is non-temporal. dataset consisting of a Temporal KG with 125k Temporal QA datasets have mostly been studied entities and 328k facts, along with a set of 410k in the area of reading comprehension. One such natural language questions that require temporal dataset is TORQUE (Ning et al., 2020), where the reasoning. system is given a question along with some context On this new dataset, we apply approaches based (a text passage) and is asked to answer a multiple on deep language models (LM) alone, such as T5 choice question with five choices. This is in con- (Raffel et al., 2020), BERT (Devlin et al., 2019), trast to KGQA, where there is no context, and the and KnowBERT (Peters et al., 2019), and also answer is one of potentially hundreds of thousands hybrid LM+KG embedding approaches, such as of entities. Entities-as-Experts (Fevry´ et al., 2020) and Em- TempQuestions (Jia et al., 2018a) is a KGQA bedKGQA (Saxena et al., 2020). We find that dataset specifically aimed at temporal QA. It con- these baselines are not suited to temporal reason- sists of a subset of questions from WebQuestions, ing. In response, we propose CRONKGQA, an Free917 (Cai and Yates, 2013) and Complex- enhancement of EmbedKGQA, which outperforms Questions (Bao et al., 2016) that are temporal in Reasoning Example Template Example Question Simple time When did fheadg hold the position of ftailg When did Obama hold the position of President of USA Simple entity Which award did fheadg receive in ftimeg Which award did Brad Pitt receive in 2001 Before/After Who was the ftailg ftypeg fheadg Who was the President of USA before Obama First/Last When did fheadg play their fadjg game When did Messi play their first game Time join Who held the position of ftailg during feventg Who held the position of President of USA during WWII Table 2: Example questions for different types of temporal reasoning. fheadg, ftailg and ftimeg correspond to entities/timestamps in facts of the form (head, relation, tail, timestamp). feventg corresponds to entities in event facts eg. WWII. ftypeg can be one of before/after and fadjg can be one of first/last. Please refer to Section 3.2 for details. nature. They gave a definition for “temporal ques- requiring temporal reasoning. tion” and used certain trigger words (for example ‘before’, ‘after’) along with other constraints to 3.1 Temporal KG filter out questions from these datasets that fell un- To prepare our temporal KG, we started by taking der this definition. However, this dataset contains all facts with temporal annotations from the Wiki- only 1271 questions — useful only for evaluation Data subset proposed by Lacroix et al.(2020). We — and the KG on which it is based (a subset of removed some instances of the predicate “member FreeBase (Bollacker et al., 2008)) is not a temporal of sports team” in order to balance out the KG KG. Another drawback is that FreeBase has not since this predicate constituted over 50 percent of been under active development since 2015, there- the facts.