Knowledge Graph and Corpus Driven Segmentation and Answer Inference for Telegraphic Entity-seeking Queries Mandar Joshi ∗ Uma Sawant Soumen Chakrabarti IBM Research IIT Bombay, Yahoo Labs IIT Bombay [email protected] [email protected] [email protected] Abstract 1 Introduction A majority of Web queries mention an entity or Much recent work focuses on formal in- type (Lin et al., 2012), as users increasingly ex- terpretation of natural question utterances, plore the Web of objects using Web search. To with the goal of executing the resulting better support entity-oriented queries, commercial structured queries on knowledge graphs Web search engines are rapidly building up large (KGs) such as Freebase. Here we address catalogs of types, entities and relations, popu- two limitations of this approach when ap- larly called a “knowledge graph” (KG) (Gallagher, plied to open-domain, entity-oriented Web 2012). Despite these advances, robust, Web-scale, queries. First, Web queries are rarely well- open-domain, entity-oriented search faces many formed questions. They are “telegraphic”, challenges. Here, we focus on two. with missing verbs, prepositions, clauses, case and phrase clues. Second, the KG is 1.1 “Telegraphic” queries always incomplete, unable to directly an- First, the surface utterances of entity-oriented Web swer many queries. We propose a novel queries are dramatically different from TREC- technique to segment a telegraphic query or Watson-style factoid question answering (QA), and assign a coarse-grained purpose to where questions are grammatically well-formed. each segment: a base entity e1, a rela- Web queries are usually “telegraphic”: they are tion type r, a target entity type t2, and short, rarely use function words, punctuations contextual words s. The query seeks en- or clausal structure, and use relatively flexible tity e t where r(e , e ) holds, fur- word orders. E.g., the natural utterance “on the 2 ∈ 2 1 2 ther evidenced by schema-agnostic words bank of which river is the Hermitage Museum lo- s. Query segmentation is integrated with cated” may be translated to the telegraphic Web the KG and an unstructured corpus where query hermitage museum river bank. Even mentions of entities have been linked to on well-formed question utterances, 50% of in- the KG. We do not trust the best or any terpretation failures are contributed by parsing or specific query segmentation. Instead, evi- structural matching failures (Kwiatkowski et al., dence in favor of candidate e2s are aggre- 2013). Telegraphic utterances will generally be gated across several segmentations. Ex- even more challenging. tensive experiments on the ClueWeb cor- Consequently, whereas TREC-QA/NLP-style pus and parts of Freebase as our KG, us- research has focused on parsing and precise in- ing over a thousand telegraphic queries terpretation of a well-formed query sentence to adapted from TREC, INEX, and Web- a strongly structured (typically graph-oriented) Questions, show the efficacy of our ap- query language (Kasneci et al., 2008; Pound et proach. For one benchmark, MAP im- al., 2012; Yahya et al., 2012; Berant et al., 2013; proves from 0.2–0.29 (competitive base- Kwiatkowski et al., 2013), the Web search and in- lines) to 0.42 (our system). NDCG@10 formation retrieval (IR) community has focused improves from 0.29–0.36 to 0.54. on telegraphic queries (Guo et al., 2009; Sarkas et al., 2010; Li et al., 2011; Pantel et al., 2012; Lin et al., 2012; Sawant and Chakrabarti, 2013). In ∗Work done as Masters student at IIT Bombay terms of target schema richness, these efforts may 1104 Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1104–1114, October 25-29, 2014, Doha, Qatar. c 2014 Association for Computational Linguistics appear more modest. The act of query ‘interpre- Other contextual matching words s (some- • tation’ is mainly a segmentation of query tokens times called selectors), by purpose. In the example above, one may re- with the simultaneous intent of finding and rank- port segments “Hermitage Museum” (a located ar- ing entities e t , such that r(e , e ) is likely 2 ∈ 2 1 2 tifact or named entity), and “river bank” (the target to hold, evidenced near the matching words in un- type). This is reminiscent of record segmentation structured text. in information extraction (IE). Over well-formed Given the short, telegraphic query utterances, utterances, IE baselines are quite competitive (Yao we limit our scope to at most one relation mention, and Van Durme, 2014). But here, we are interested unlike the complex mapping of clauses in well- exclusively in telegraphic queries. formed questions to twig and join style queries (e.g., “find an actor whose spouse was an Italian 1.2 Incomplete knowledge graph bookwriter”). On the other hand, we need to deal The second problem is that the KG is always with the unhelpful input, as well as consolidate work in progress (Pereira, 2013), and connec- the KG with the corpus for ranking candidate e2s. tions found within nodes of the KG, between the Despite the modest specification, our query tem- KG and the query, or the KG and unstructured plate is quite expressive, covering a wide range of text, are often incomplete or erroneous. E.g., entity-oriented queries (Yih et al., 2014). Wikipedia is considered tiny, and Freebase rather We present a novel discriminative graphical small, compared to what is needed to answer all model to capture the entity ranking inference task, but the “head” queries. Google’s Freebase an- with query segmentation as a by-product. Ex- notations (Gabrilovich et al., 2013) on ClueWeb tensive experiments with over a thousand entity- (ClueWeb09, 2009) number fewer than 15 per seeking telegraphic queries using the ClueWeb09 page to ensure precision. Fewer than 2% are to corpus and a subset of Freebase show that we can entities in Freebase but not in Wikipedia. accurately predict the segmentation and intent of It may also be difficult to harness the KG for telegraphic relational queries, and simultaneously answering certain queries. E.g., answering the rank candidate responses with high accuracy. We query fastest odi century batsman, the intent of also present evidence that the KG and corpus have which is to find the batsman holding the record for synergistic salutary effects on accuracy. the fastest century in One Day International (ODI) 2 explores related work in more detail. 3 § § cricket, may be too difficult for most KG-only sys- gives some examples fitting our query template, tems, but may be answered quite effectively by a explains why interpreting some of them is nontriv- system that also utilizes evidence from unstruc- ial, and sets up notation. 4 presents our core tech- § tured text. nical contributions. 5 presents experiments. Data § There is a clear need for a “pay-as-you-go” ar- can be accessed at http://bit.ly/Spva49 chitecture that involves both the corpus and KG. A and http://bit.ly/WSpxvr. query easily served by a curated KG should give accurate results, but it is desirable to have a grace- 2 Related work ful interpolation supported by the corpus: e.g., if The NLP/QA community has traditionally as- the relation r(e1, e2) is not directly evidenced in the KG, but strongly hinted in the corpus, we still sumed that question utterances are grammatically want to use this for ranking. well-formed, from which precise clause structure, ground constants, variables, and connective rela- 1.3 Our contributions tions can be inferred via semantic parsing (Kas- neci et al., 2008; Pound et al., 2012; Yahya et Here, we make progress beyond the above frontier al., 2012; Berant et al., 2013; Kwiatkowski et of prior work in the following significant ways. al., 2013) and translated to lambda expressions We present a new architecture for structural in- (Liang, 2013) or SPARQL style queries (Kasneci terpretation of a telegraphic query into these seg- et al., 2008), with elaborate schema knowledge. ments (some may be empty): Such approaches are often correlated with the as- Mention/s e of an entity e , • 1 1 sumption that all usable knowledge has been cu- Mention r of a relation type r, rated into a KG. The query is first translated to a • Mention t bof a target type t , and structured form and then “executed” on the KG. A • 2 2 b b 1105 Telegraphic query eb1 rb tb2 s nobel prize winner african american first first african american nobel prize winner nobel prize - winner first african american - - winner first african american nobel prize dave navarro band - first dave navarro first band dave navarro band band first merril lynch headquarters - - merril lynch headquarters merril lynch - headquarters - spanish died in poet civil war spanish poet died in civil war civil war died - spanish poet spanish in poet died civil war - - - first american in space first american in space - - american first, in space Figure 1: Example queries and some potential segmentations. large corpus may be used to build relation expres- and e1, r, t2 to represent their textual mentions or sion models (Yao and Van Durme, 2014), but not hints, if any, in the query. s is a set of uninterpreted as supporting evidence for target entities. textualb tokensb b in the query that are used to match In contrast, the Web and IR community gener- and collect corpus contexts that lend evidence to ally assumes a free-form query that is often tele- candidate entities. graphic (Guo et al., 2009; Sarkas et al., 2010; Li Figure 1 shows some telegraphic queries et al., 2011). Queries being far more noisy, the with possible segmentation into the above goal of structure discovery is more modest, and of- parts.