arXiv:1908.03650v4 [cs.IR] 25 Jan 2021 pae for’ ‘played 01AscainfrCmuigMachinery. Computing for Association 2021 © ol eur he-a onta oncsNya,hssis his Neymar, connects that join three-way a require would sesres[,1] a adesml usin ieteab the like questions simple handle can 17]) [9, surveys (see K-A ist nwrqetosoe ag nweg bases d structured knowledge other large or etc.) over YAGO, , questions DBpedia, (e.g., answer to aims (KB-QA) C utb ooe.Asrcigwt rdti emte.T permitted. is credit with Abstracting honored. be must ACM C SN978-x-xxxx-xxxx-x/YY/MM...$15.00 ISBN ACM rmcrepnigsbqetos o xml,tequesti the example, For sub-questions. th corresponding join to from need and in entities that different questions on conditions complex multiple with struggle but well, very ample Assestk siptqetossc as: such questions input as take systems QA Problem. and Motivation INTRODUCTION 1 state-of-t method. our against of viability Comparisons the question. show final baselines full compute the to to intervals swers temporal on reasoning TEQUILA constraint the Finally, engine. are KB-QA underlying sub-questions the from to trieved Answers constraints. intent temporal temporal and sub-q has non-temporal question into question a the rewrites if and composes detects It TEQUIL stages. engine. KB-QA four any of has top on run can that QA temporal for nqeyietfidentity, identified uniquely present need We relations handled. and temporal t discovered for is cues here, where addressed questions, case, temporal important decompo An be to sub-questions. chal need into that poses questions (KB-QA) complex handling bases in lenges knowledge over answering Question ABSTRACT https://doi.org/10.1145/nnnnnnn.nnnnnnn IM1,Otbr21,Trn Italy Turin, 2018, [email protected] October from CIKM’18, permissions requ Request lists, fee. to a redistribute to and/or or servers on post to work publish, this of components n for this Copyrights bear page. copies first that the and on thi tion advantage of commercial ar part copies or that or profit provided all for fee of without granted copies is hard use or classroom digital make to Permission mapping KB: the in predicates and and vo- question the types, input address the tities, to in need phrases methods between KB-QA answe mismatch so, retrieve cabulary doing to In queries KB. lang the the formal execute from and a SQL, in or queries, SPARQL structured like into them translate and aalaBcrn n ai Beckham. David and Beckran, Rafaella EUL:Tmoa eto nwrn vrKoldeBases Knowledge over Answering estion Temporal TEQUILA: Q2: Q1: otws ioogUiest,China University, Jiaotong Southwest Atrwo i emrssse hoehrls name?” last her choose sister Neymar’s did whom “After for?” play Neymar did teams “Which oteK predicate KB the to [email protected] hnJia Zhen ‘teams’ nweg-ae usinanswering question Knowledge-based [email protected] oc etrfrA,Germany AI, for Center Bosch memberOf oteK type KB the to ankStrötgen Jannik TEQUILA tt-fteatKB-QA State-of-the-art . rspirseicpermission specific prior ires nealrmethod enabler an , o aeo distributed or made not e g. oyohrie rre- or otherwise, copy o okfrproa or personal for work s footballClub tc n h ulcita- full the and otice we yohr than others by owned ‘Neymar’ P o nomtc,Germany Informatics, for MPI [email protected] bagaiAbujabal Abdalghani results e on: t.KB- ata. v ex- ove tde- It . uestions he-art othe to a of hat volve obe to uage re- n uses and sed an- en- ter rs - A ii ereigcniaeasesfrsbqetos and sub-questions, for answers candidate retrieving (iii) okta nopse orsae fpoesn:()detect (i) processing: of stages four encompasses that work w method, our compare experimentally we which to [4], by work h eprlpreposition temporal The prahadContributions. and Approach u-usinrslsit vrl nwr.Aldt n cod and data All answers. overall que co into complex for results reasoning decomposing sub-question time-constraint-based for the method (ii) a and contri tions, (i) Novel KB-QA. include: temporal also to tions specific pipeline complete a quest simple answering for us geared [6] are AQQU that and experiments), in [2] QUINT systems existi (state-of-the-art leverage pre we systems (iii), the stage of For results answers. final the into reconcile stage and combine to reasoning ral oa usin,(i eopsn usin n rewriting and questions decomposing (ii) questions, poral question composite such for system KB-QA any of aware not are th of exception the With step. final a as requ comparison would OVERLAP Q4 an Analogously, operator. BEFORE return a date the using with Q3.2, them by compare and KB, the from scopes time PSG?” join Neymar etti,bthv opoel iabgaei oanormaliz a could to We it disambiguate team). properly that to phrase have to but The transfer this, tect (Neymar’s nature. event temporal an a its so to mentioned, refers detect is to 2017) August is (e.g., lenge date explicit no Q3, In eut.Freape 3sol edcmoe n rewritten and decomposed Q3.1: sub-questio be the should Q3 on example, a reasoning For questions results. and full combining the by for computed answers then The system. KB-QA eval standard separately a be rewrite can they and that so questions sub-questions resulting temporal such decompose judiciously noelpts o h epcietm periods. time respective the for cont how test execution is overlap the an that challenge ensure second and A questions challenge. such decompose big detecting a Here, Neymar. is of nature that per temporal with job overlap whose cruci Barcelona selected is FC are it at However, coaches form. only that surface time-dependen answers be its to correct at seem looking not when does all, Q4 this. at for example an is Q2 like bu ie[6.Cnie h w examples: two the con Consider implicit [16]. or time explicit about with comes often Search needs. tion P o nomtc,Germany Informatics, for MPI otebs forkolde hsi h rtpprta prese that paper first the is this knowledge, our of best the To Q4: Q3: nipratcs fcmlxqetosaetmoa informa temporal are questions complex of case important An u ouin aldTQIA sbito uebsdframe rule-based a on built is TEQUILA, called solution, Our ‘before’ [email protected] WihtasddNya lyfor?” play Neymar did teams “Which UdrwihcahsddNya lyi Barcelona?” in play Neymar did coaches which “Under PSG?” joining before for play Neymar did teams “Which ehr Weikum Gerhard , ‘after’ o h eut fQ.,w ol hnretrieve then could we Q3.1, of results the For . t.aeas sdi o-eprlcontexts; non-temporal in used also are etc. , P o nomtc,Germany Informatics, for MPI ‘before’ [email protected] ihrjSh Roy Saha Rishiraj sasrn u swl,btwords but well, as cue strong a is h e dao hsppri to is paper this of idea key The n Q3.2: and jiigPSG’ ‘joining We did “When i)tempo- (iv) gKB-QA ng sub-questions, ae by uated mbining n tem- ing ddate. ed ditions ions. vious lfor al are e chal- iods ains into the the nts bu- de- ire ed ed to re s- n e s. e t - - CIKM’18, October 2018, Turin, Italy Z. Jia et al. public at https://github.com/zhenjia2017/tequila, and a demo is avail- Table 1: Decomposition and rewriting of questions. The con- able at https://tequila.mpi-inf.mpg.de/. straint is the fragment after the SIGNAL . wh∗ is the question word (e.g., who), and F8 are tokens in the question.

Expected input: wh∗ F ...F SIGNAL F + ...F ? 2 CONCEPTS 1 = = 1 ? Case 1: Constraint has both an entity and a relation In NLP, the markup language TimeML (www.timeml.org) is widely Sub-question 1 pattern: wh∗ F1 ...F=? used for annotating temporal information in text documents. Our Sub-question 2 pattern: when F=+1 ...F? ? definition of temporal questions is based on two of its concepts (tags E.g.: “where did neymar play before he joined barcelona?” for temporal expressions and temporal signals). Sub-question 1: “where did neymar play?” Temporal expressions. TIMEX3 tags demarcate four types of tem- Sub-question 2: “when neymar joined barcelona?” poral expressions. Dates and times refer to points in time of differ- Case 2: Constraint has no entity but a relation ent granularities (e.g., ‘May 1, 2010’ and ‘9 pm’, respectively). They Sub-question 1 pattern: wh∗ F1 ...F=? occur in fully- or under-specified forms (e.g., ‘May 1, 2010’ vs. ‘last Sub-question 2 pattern: when sq1-entity F=+1 ...F? ? year’). Durations refer to intervals (e.g., ‘two years’), and sets to E.g.: “where did neymar live before playing for clubs?” periodic events (e.g., ‘every Monday’). Going beyond TimeML, im- Sub-question 1: “where did neymar live?” plicit expressions (e.g., ‘the Champions League final’) are used to Sub-question 2: “when neymar playing for clubs?” capture events and their time scopes [14]. Expressions can be nor- Case 3: Constraint has no relation but an entity =3 malized into standard format (e.g., ‘May 2 , 2016’ into 2016-05-02). Sub-question 1 pattern: wh∗ F1 ...F=? Temporal signals. SIGNAL tags mark textual elements that de- Sub-question 2 pattern: when F=+1 ...F? F1 ...F= ? note explicit temporal relations between two TimeML entities (i.e., E.g.: “who was the brazil team captain before neymar?” events or temporal expressions), such as ‘before’ or ‘during’. We Sub-question 1: “who was the brazil team captain?” extend the TimeML definition to also include cues when an event Sub-question 2: “when neymar was the brazil team captain?” is mentioned only implicitly, such as ‘joining PSG’. In addition, we Case 4: Constraint is an event name consider ordinals like ‘first’, ‘last’, etc. These are frequent in ques- Sub-question 1 pattern: wh∗ F1 ...F=? tions when entities can be chronologically ordered, such as ‘last’ Sub-question 2 pattern: when did F=+1 ...F? happen? in “Neymar’s last club before joining PSG”. E.g.: “where did neymar play during south africa world cup?” Temporal questions. Based on these considerations, we can now Sub-question 1: “where did neymar play?” define a temporal question as any question that contains a temporal Sub-question 2: “when did south africa world cup happen?” expression or a temporal signal, or whose answer type is temporal. Temporal relations. Allen [3] introduced 13 temporal relations between time intervals for temporal reasoning: EQUAL, BEFORE, MEETS, OVERLAPS, DURING, STARTS, FINISHES, and their in- verses for all but EQUAL. However, for an input temporal question, it is not always straightforward to infer the proper relation. For ex- 3.1 Detecting temporal questions ample, in Q3 the relation should be BEFORE; but if we slightly vary Q3 to: A question is identified as temporal if it contains any of the fol- lowing: (a) explicit or implicit temporal expressions (dates, times, Q5: “Which team did Neymar play for before joining PSG?”, events), (b) temporal signals (i.e., cue for temporal relations), the singular form ‘team’ suggests that we are interested in the (c) ordinal words (e.g., first), (d) an indication that the answer type MEETS relation, that is, only the last team before the transfer. Fre- is temporal (e.g., the question starts with ‘When’). We use Heidel- quent trigger words suggesting such relations are, for instance, the Time [22] to tag TIMEX3 expressions in questions. Named events signals before, prior to (for BEFORE or MEETS), after, following (for are identified using a dictionary curated from Freebase. Specifi- AFTER), and during, while, when, in (for OVERLAP). cally, if the type of an entity is ‘time.event’, its surface forms are added to the event dictionary. SIGNAL words and ordinal words are detected using a small dictionary as per suggestions from Set- zer [21], and a list of temporal prepositions. To spot questions 3 METHOD whose answers are temporal, we use a small set of patterns like Given an input question, TEQUILA works in four stages: (i) detect when, what date, in what year, and which century. if the question is temporal, (ii) decompose the question into simpler sub-questions with some form of rewriting, (iii) obtain candidate answers and dates for temporal constraints from a KB-QA system, 3.2 Decomposing and rewriting questions and (iv) apply constraint-based reasoning on the candidates topro- TEQUILA decomposes a composite temporal question into one or duce final answers. Our method builds on ideas from the literature more non-temporal sub-questions (returning candidate answers), and on question decomposition for general QA [2, 5, 19]. Standard NLP one or more temporal sub-questions (returning temporal constraints). tasks like POS tagging, NER, and coreference resolution, are per- Results of sub-questions are combined by intersecting their an- formed on the input question before passing it on to TEQUILA. swers. The constraints are applied to time scopes associated with TEQUILA: Temporal estion Answering over Knowledge Bases CIKM’18, October 2018, Turin, Italy results of the non-temporal sub-questions. For brevity, the follow- Table 2: Temporal reasoning constraints. ing explanation focuses on the case with one non-temporal sub- question, and one temporal sub-question. We use a set of lexico- Relation Signal word(s) Constraint syntactic rules (Table 1) designed from first principles to decom- BEFORE ‘before’, ‘prior to’ 4=30=B ≤ 1468=2>=B pose and rewrite a question into its components. Basic intuitions AFTER ‘after’ 1468=0=B ≥ 4=32>=B driving these rules are as follows: OVERLAP ‘during’, ‘while’, ‘when’ 1468=0=B ≤ 4=32>=B ≤ 4=30=B • The signal word separates the non-temporal and temporal sub- ‘since’, ‘until’, ‘in’ 1468=0=B ≤ 1468=2>=B ≤ 4=30=B ‘at the same time as’ 1468= ≤ 1468= ≤ 4=3 ≤ 4=3 questions, acting as a pivot for decomposition; 2>=B 0=B 0=B 2>=B • Each sub-question needs to have an entity and a relation (gen- erally represented using verbs) to enable the underlying KB- 3.4 Reasoning on temporal intervals QA systems to handle sub-questions; For temporal sub-questions, the results are time points, time inter- • If the second sub-question lacks the entity or the relation, it is vals, or sets of dates (e.g., a set of consecutive years during which borrowed from the first sub-question; someone played for a football team). We cast all these into inter- • KB-QA systems are robust to ungrammatical constructs, thus vals with start point 1468=2>=B and end point 4=32>=B . These form precluding the need for linguistically correct sub-questions. the temporal constraints against which we test the time scopes of the non-temporal candidate answers, also cast into intervals [1468=0=B,4=30=B]. The test itself depends on the temporal opera- tor derived from the input question (e.g., BEFORE, OVERLAP, etc.) (Table 2). For questions with ordinal constraints (e.g., last), we sort 3.3 Answering sub-questions the (possibly open) intervals to select the appropriate answer. Sub-questions are passed on to the underlying KB-QA system, which translates them into SPARQL queries and executes them on the KB. 4 EXPERIMENTS This produces a result set for each sub-question. Results from the 4.1 Setup non-temporal sub-question(s) are entities of the same type (e.g., football teams). These are candidate answers for the full question. We evaluate TEQUILA on the TempQuestions benchmark [13], which With multiple sub-questions, the candidate sets are intersected. The contains 1, 271 temporal questions labeled as questions with ex- temporal sub-questions, on the other hand, return temporal con- plicit, implicit, and ordinal constraints, and those with temporal an- straints such as dates, which act as constraints to filter the non- swers. Questions are paired with their answers over Freebase. We temporal candidate set. Candidate answers need to be associated use three state-of-the-art KB-QA systems as baselines: AQQU [6], with time scopes, so that we can evaluate the temporal constraints. QUINT [2] (code from authors for both), and Bao et al. [4] (detailed Retrieving time scopes. To obtain time scopes, we introduce ad- results from authors). The first two are geared for simple questions, ditional KB lookups; details depend on the specifics of the under- while Bao et al. handle complex questions, including temporal ones. lying KB. Freebase, for example, often associates SPO triples with We use TEQUILA as a plug-in for the first two, and directly evalu- time scopes by means of value types (CVTs); other KBs ate against the system of Bao et al. on 341 temporal questions from may use =-tuples (= > 3) to attach spatio-temporal attributes to the ComplexQuestions test set [4]. For evaluating baselines, the full facts. For example, the Freebase predicate marriage is a CVT with question was fed directly to the underlying system. We report pre- attributes including marriage.spouse and marriage.date. When the cision, recall, and F1 scores of the retrieved answer sets w.r.t. the predicate marriage.spouse is used to retrieve answers, the time gold answer sets, and average them over all test questions. scope is retrieved by looking up marriage.date in the KB. On the other hand, playing for a footballclub could be captured in a predi- 4.2 Results and insights cate like team.players without temporal information attached, and Results on TempQuestions and the 341 temporal questions in Com- the job periods are represented as events in predicates like footballPlayer.plexQuestions are shown in Table 3. AQQU + TEQUILA and QUINT team. joinedOnDate and footballPlayer. team. leftOnDate). In such + TEQUILA refer to the TEQUILA-enabled versions of the respec- cases, TEQUILA considers all kinds of temporal predicates for the tive baseline systems. We make the following observations. candidate entity, and chooses one based on a similarity measure be- TEQUILA enables KB-QA systems to answer composite tween the non-temporal predicate (team.players) and potentially questions with temporal conditions. Overall and category-wise relevant temporal predicates (footballPlayer. team. joinedOnDate, F1-scores show that TEQUILA-enabled systems significantly out- footballPlayer.award.date). The similarity measure is implemented perform the baselines. Note that these systems neither have capa- by selecting tokens in predicate names (footballPlayer, team, etc.), bilities for handling compositional syntax nor specific support for contextualizing the tokens by computing embeddings for temporal questions. Our decomposition and rewrite methods are them, averaging per-token vectors to get a resultant vector for each crucial for compositionality, and constraint-based reasoning on an- predicate [24], and comparing the cosine distance between two swers is decisive for the temporal dimension. The improvement in predicate vectors. The best-matching temporal predicate is chosen F1-scores stems from a systematic boost in precision, across most for use. When time periods are needed (e.g., for a temporal con- categories. straint using OVERLAP), a pair of begin/end predicates is selected TEQUILA outperforms state-of-the-art baselines. Bao et (e.g., footballPlayer. team. joinedOnDate and leftOnDate). al. [4] represents the state-of-the-art in KB-QA, with a generic CIKM’18, October 2018, Turin, Italy Z. Jia et al.

Table 3: Detailed performance of TEQUILA-enabled systems on TempQuestions and ComplexQuestions.

TempQuestions Aggregate results Explicit constraint Implicit constraint Temporal answer Ordinal constraint (1,271 questions) Prec Rec F1 Prec Rec F1 Prec Rec F1 Prec Rec F1 Prec Rec F1 AQQU [6] 24.6 48.0 27.2 27.6 60.7 31.1 12.9 34.9 14.5 26.1 33.5 27.4 28.4 57.4 32.7 AQQU+TEQUILA 36.0* 42.3 36.7* 43.8* 53.8 44.6* 29.1* 34.7 29.3* 27.3* 29.6 27.7* 38.0* 41.3 38.6* QUINT [2] 27.3 52.8 30.0 29.3 60.9 32.6 25.6 54.4 27.0 25.2 38.2 27.3 21.3 54.9 26.1 QUINT+TEQUILA 33.1* 44.6 34.0* 41.8* 51.3 42.2* 13.8 43.7 15.7 28.6* 34.5 29.4* 37.0* 42.2 37.7* ComplexQuestions Aggregate results Explicit constraint Implicit constraint Temporal answer Ordinal constraint (341 questions) Prec Rec F1 Prec Rec F1 Prec Rec F1 Prec Rec F1 Prec Rec F1 Bao et al. [4] 34.6 48.4 35.9 41.1 53.2 41.9 26.4 36.5 27.0 18.6 40.2 22.3 31.1 60.8 36.1 AQQU [6] 21.5 50.0 23.3 25.0 60.1 28.4 11.2 31.2 11.4 19.6 35.7 19.2 22.2 54.9 25.3 AQQU+TEQUILA 36.2* 45.9 37.5* 41.2* 54.7 43.5* 27.5* 32.6 27.0* 29.5* 32.1 29.9* 40.2* 45.1 40.8* QUINT [2] 22.0 50.3 24.5 24.7 54.7 27.5 18.8 47.9 19.0 16.6 37.5 20.7 20.9 51.3 26.0 QUINT+TEQUILA 29.6* 44.9 31.1* 34.6* 47.3 36.3* 12.3 42.1 13.9 33.4* 37.5 33.9* 44.9* 51.6* 45.8* Aggregate results are averaged over the four categories. The highest value in a column for each dataset is in bold. An asterisk (*) indicates statistical significance of TEQUILA-enabled systems over their standalone counterparts, under the 2-tailed paired C-test at ? < 0.05 level. mechanism for handling constraints in questions. TEQUILA-enabled methods do not cope well with complex questions. Bao et al. [4] systems outperform Bao et al. on the temporal slice of ComplexQues- combined rules with deep learning to address a variety of complex tions, showing that a tailored method for temporal information questions. needs is worthwhile. TEQUILA enabled QUINT and AQQU to an- swer questions like: “who is the first husband of julia roberts?”, “when 6 CONCLUSION did francesco sabatini start working on the puerta de san vicente?”, Understanding the compositional semantics of complex questions and “who was governor of oregon when shanghai noon was released?”. is an open challenge in QA. We focused on temporal question an- Error analysis. Analyzing cases when TEQUILA fails yields in- swering over KBs, as a major step for coping with an important sights towards future work: (i) Decomposition and rewriting were slice of information needs. Our method showed boosted perfor- incorrect (for example, in “where did the pilgrims come from be- mance on a recent benchmark, and outperformed a state-of-the- fore landing in america?”, ‘landing’ is incorrectly labeled as a noun, art baseline on general complex questions. Our work underlines triggering case 3 instead of case 1 in Table 1); (ii) The correct tem- the value of building reusable modules that improve several KB- poral predicate was not found due to limitations of the similarity QA systems. function; and (iii) The temporal constraint or the time scope to use during reasoning was wrongly identified. REFERENCES [1] Abdalghani Abujabal, Rishiraj Saha Roy, Mohamed Yahya, and GerhardWeikum. 5 RELATED WORK 2018. Never-ending learning for open-domain over knowl- edge bases. In WWW. 1053–1062. QA has a long tradition in IR and NLP, including benchmarking [2] A. Abujabal, M. Yahya, M. Riedewald, and G. Weikum. 2017. Automated template tasks in TREC, CLEF, and SemEval. This has predominantly fo- generation for question answering over knowledge graphs. In WWW. [3] J. F. Allen. 1990. Maintaining knowledge about temporal intervals. In Readings cused on retrieving answers from textual sources. The recent TREC in qualitative reasoning about physical systems. CAR (complex answer retrieval) resource [10], explores multi-faceted [4] J. Bao, N. Duan, Z. Yan, M. Zhou, and T. Zhao. 2016. Constraint-based question passage answers, but information needs are still simple. In IBM answering with . In COLING. [5] J. Bao, N. Duan, M. Zhou, and T. Zhao. 2014. Knowledge-based question answer- [11], structured data played a role, but text was the main ing as . In ACL. source for answers. Question decomposition was leveraged, for ex- [6] H. Bast and E. Haussmann. 2015. More Accurate Question Answering on Free- base. In CIKM. ample, in [11, 19, 28] for QA over text. However, re-composition [7] J. Berant, A. Chou, R. Frostig, and P. Liang. 2013. Semantic on Freebase and reasoning over answers works very differently for textual sources [19], from Question-Answer Pairs. In EMNLP. and are not directly applicable for KB-QA. Compositional seman- [8] Q. Cai and A. Yates. 2013. Large-scale via Schema Matching and Lexicon Extension. In ACL. tics of natural language sentences has been addressed by [15] from [9] D. Diefenbach, V. Lopez, K. Singh, and P. Maret. 2017. Core techniques of ques- a general linguistic perspective. Although applicable to QA, exist- tion answering systems over knowledge bases: A survey. In Knowledge and In- ing systems support only specific cases of composite questions. formation systems. [10] L. Dietz and B. Gamari. 2017. TREC CAR: A Data Set for Complex Answer KB-QA is a more recent trend, starting with [7, 8, 12, 23, 26]. Retrieval. In TREC. Most methods have focused on simple questions, whose SPARQL [11] D. A. Ferrucci et al. 2012. This is Watson. In IBM Journal of R&D. [12] A. Fader, L. Zettlemoyer, and O. Etzioni. 2014. Open question answering over translations contain only a single variable (and a few triple pat- curated and extracted knowledge bases. In KDD. terns for a single set of qualifying entities). For popular bench- [13] Z. Jia, A. Abujabal, R. Saha Roy, J. Strötgen, and G. Weikum. 2018. Temp- marks like WebQuestions [7], the best performing systems use tem- Questions: A Benchmark for Temporal Question Answering. In HQA. [14] Erdal Kuzey, Vinay Setty, Jannik Strötgen, and Gerhard Weikum. 2016. As time plates and grammars [1, 2, 6, 18, 28], leverage additional text [20, goes by: Comprehensive tagging of textual phrases with temporal scopes. In 25], or learn end-to-end with extensive training data [25, 27]. These Proceedings of the 25th international conference on . 915–925. TEQUILA: Temporal estion Answering over Knowledge Bases CIKM’18, October 2018, Turin, Italy

[15] P. Liang, M. I. Jordan, and D. Klein. 2011. Learning Dependency-Based Compo- [21] A. Setzer. 2002. Temporal information in Newswire articles: An annotation scheme sitional Semantics. In ACL. and corpus study. Ph.D. Dissertation. University of Sheffield. [16] D. Metzler, R. Jones, F. Peng, and R. Zhang. 2009. Improving Search Relevance [22] J. Strötgen and M. Gertz. 2010. HeidelTime: High Quality Rule-Based Extraction for Implicitly Temporal Queries. In SIGIR. and Normalization of Temporal Expressions. In SemEval. [17] Alessandro Moschitti, Kateryna Tymoshenko, Panos Alexopoulos, Andrew [23] C. Unger, L. Bühmann, J. Lehmann, A. N. Ngomo, D. Gerber, and P. Cimiano. Walker, Massimo Nicosia, Guido Vetere, Alessandro Faraotti,MarcoMonti, JeffZ 2012. Template-based question answering over RDF data. In WWW. Pan, Honghan Wu, et al. 2017. Question answering and knowledge graphs. In [24] J. Wieting, M. Bansal, K. Gimpel, and K. Livescu. 2016. Towards universal para- Exploiting Linked Data and Knowledge Graphs in Large Organisations. Springer, phrastic sentence embeddings. In ICLR. 181–212. [25] K. Xu, S. Reddy, Y. Feng, S. Huang, and D. Zhao. 2016. Question answering on [18] S. Reddy, M. Lapata, and M. Steedman. 2014. Large-scale semantic parsing with- freebase via relation extraction and textual evidence. ACL. out question-answer pairs. In TACL. [26] M. Yahya, K. Berberich, S. Elbassuoni, M. Ramanath, V. Tresp, and G. Weikum. [19] E. Saquete, J. L. Vicedo, P. Martínez-Barco, R. Muñoz, and H. Llorens. 2009. En- 2012. Natural language questions for the web of data. In EMNLP. hancing QA Systems with Complex Temporal Question Processing Capabilities. [27] W. Yih, M. Chang, X. He, and J. Gao. 2015. Semantic Parsing via Staged Query J. Artif. Int. Res. (2009). Graph Generation: Question Answering with . In ACL. [20] D. Savenkov and E. Agichtein. 2016. When a Knowledge Base Is Not Enough: [28] P. Yin, N. Duan, B. Kao, J. Bao, and M. Zhou. 2015. Answering Questions with Question Answering over Knowledge Bases with External Text Data. In SIGIR. Complex Semantic Constraints on Open Knowledge Bases. In CIKM.