Text as a Virtual Knowledge Base Bhuwan Dhingra Language Technologies Institute Carnegie Mellon University Work done with: Haitian Sun, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Russ Salakhutdinov, William Cohen Question Answering When did Kendrick Lamar’s first album come out? A. July 2, 2011 Source: Natural Questions (https://ai.google.com/research/NaturalQuestions/dataset) Sources of Information Text Knowledge Bases Sources of Information Text Knowledge Bases [Moldovan, 2002], [Voorhees, 1999], [Kwiatkowski, 2013], [Berant, 2013], [Ferrucci, 2012], [Yih, 2013], [Reddy, 2014], [Pasupat, 2015], [Hermann, 2016], [Chen, 2017], [Seo, [Bordes, 2015], [Yih, 2015], [Jain, 2017], [Peters, 2018], [Devlin, 2018] … 2016], [Liang, 2017], [Das, 2017] … Sources of Information Text Knowledge Bases [Moldovan, 2002], [Voorhees, 1999], [Kwiatkowski, 2013], [Berant, 2013], [Ferrucci, 2012], [Yih, 2013], [Reddy, 2014], [Pasupat, 2015], [Hermann, 2016], [Chen, 2017], [Seo, [Bordes, 2015], [Yih, 2015], [Jain, 2017], [Peters, 2018], [Devlin, 2018] … 2016], [Liang, 2017], [Das, 2017] … • Lexical pattern matching • Clear semantics via parsing • No grounding • Grounded • High recall • High precision Sources of Information Text Knowledge Bases [Moldovan, 2002],[Gardner [Voorhees, & 1999], Krishnamurthy, 2017],[Kwiatkowski, [Ryu, 2017] 2013], [Berant, 2013], [Ferrucci,Universal 2012], Schema [Yih, 2013], [Riedel, 2013], [Verga,[Reddy, 2016], 2014], [Das, [Pasupat, 2017] 2015],… [Hermann, 2016], [Chen, 2017], [Seo, [Bordes, 2015], [Yih, 2015], [Jain, 2017], [Peters, 2018], [Devlin, 2018] … Our Work 2016], [Liang, 2017], [Das, 2017] … • Lexical pattern matching • Clear semantics via parsing • No grounding • Grounded • High recall • High precision Why Text + KBs? Why Text + KBs? • Engineering motivation - QA performance • Text can complete missing information in KBs • KBs can provide background context for understanding text Why Text + KBs? • Engineering motivation - QA performance • Text can complete missing information in KBs • KBs can provide background context for understanding text • Scientific motivation - Knowledge Representation • Text is expressive • KBs support reasoning This Talk 1. “Reading” heterogeneous graphs of 2. Traversing text corpora like KBs facts and text [EMNLP’18] [ongoing] Open-domain QA using Early Fusion of KBs & Text Haitian Sun*, Bhuwan Dhingra*, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William Cohen *equal contribution EMNLP 2018 Entity-Relation Knowledge Bases Meg Griffin has-character network • Collection of (subject, relation, object) facts voiced-by • Organize information around entity nodes network Mila Kunis Lacey Chabert starred in • Freebase: 44M entities, >2B facts directed-by David Trainer Reasoning in KBs Meg Griffin Relation following operation: has-character network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by network Mila Kunis Lacey Chabert starred in directed-by David Trainer Reasoning in KBs Meg Griffin Relation following operation: has-character network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by • Given a set of entities X network Mila Kunis Lacey Chabert starred in directed-by David Trainer Reasoning in KBs Meg Griffin Relation following operation: has-character network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by • Given a set of entities X • Follow edges labeled with the relation R network Mila Kunis Lacey Chabert starred in directed-by David Trainer Reasoning in KBs Meg Griffin Relation following operation: has-character network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by • Given a set of entities X • Follow edges labeled with the relation R network Mila Kunis Lacey Chabert starred in • To arrive at a set of entities Y directed-by David Trainer Reasoning in KBs Meg Griffin Relation following operation: has-character network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by • Given a set of entities X • Follow edges labeled with the relation R network Mila Kunis Lacey Chabert starred in • To arrive at a set of entities Y E.g. {Family_Guy, directed-by That_70s_Show}.follow(network) = {Fox} David Trainer Question Answering Meg Griffin Who voiced Meg in Family Guy? has-character network voiced-by network Mila Kunis Lacey Chabert starred in directed-by David Trainer Question Answering Meg Griffin Who voiced Meg in Family Guy? has-character network voiced-by network Mila Kunis Lacey Chabert starred in directed-by David Trainer Question Answering Meg Griffin Who voiced Meg in Family Guy? has-character network voiced-by {Meg_Griffin}.follow(voiced-by) network Mila Kunis Lacey Chabert starred in directed-by David Trainer Question Answering Meg Griffin Who voiced Meg in Family Guy? has-character network voiced-by {Meg_Griffin}.follow(voiced-by) network Mila Kunis Lacey Chabert starred in {Mila_Kunis, Lacey_Chabert} directed-by David Trainer Question Answering Meg Griffin Who voiced Meg in Family Guy? has-character network Annotating semantic parses of voiced-by questions is expensive {Meg_Griffin}.follow(voiced-by) network Mila Kunis Lacey Chabert starred in {Mila_Kunis, Lacey_Chabert} directed-by David Trainer Question Answering Meg Griffin Who voiced Meg in Family Guy? has-character network Search for parses which lead to voiced-by the correct answer in the KB ??? network Mila Kunis Lacey Chabert starred in {Mila_Kunis, Lacey_Chabert} directed-by Learning from Denotations David Trainer [Liang, 2011], [Berant, 2013], [Reddy, 2014], [Krishnamurthy, 2017] But KBs are often incomplete Meg Griffin has-character network voiced-by network Mila Kunis Lacey Chabert starred in Min et al, NAACL 2013 directed-by David Trainer Graphs of Facts and Text Meg Griffin has-character Who voiced Meg in Family Guy? voiced-by Mila Kunis Lacey Chabert Graphs of Facts and Text Meg Griffin Inject relevant text into the graph has-character Who voiced Meg in Family Guy? voiced-by Mila Kunis Lacey Chabert Graphs of Facts and Text Meg Griffin Inject relevant text into the graph has-character Who voiced Meg in Family Guy? voiced-by TF-IDF Retrieval Mila Kunis Lacey Chabert Graphs of Facts and Text Meg Griffin Inject relevant text into the graph has-character Who voiced Meg in Family Guy? links-to links-to voiced-by Megatron "Meg" Griffin TF-IDF is a character from the Retrieval television series Family Guy. Mila Kunis Lacey Chabert Entity Linking Search via Representation Learning Meg Griffin has-character links-to links-to voiced-by Megatron "Meg" Griffin is a character from the television series Family Guy. Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Graph Neural Networks Meg Griffin Embedding Vector has-character links-to links-to voiced-by Megatron "Meg" Griffin is a character from the television series Family Guy. Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Graph Neural Networks Meg Griffin Embedding Vector has-character links-to “Pagerank” score - Initially uniform over entities in the links-to question voiced-by Megatron "Meg" Griffin is a character from the television series Family Guy. Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Meg Griffin has-character links-to links-to voiced-by Megatron "Meg" Griffin is a character from the television series Family Guy. Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Meg Griffin has-character Self connection links-to Entity neighbors Text mentions links-to voiced-by Megatron "Meg" Griffin is a character from the television series Family Guy. Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Meg Griffin has-character Self connection links-to Entity neighbors Text mentions links-to voiced-by • Only propagate embeddings from nodes with non-zero pagerank score Megatron "Meg" Griffin is a character from the television series • This constrains learning along valid paths Family Guy. in the graph Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Meg Griffin has-character Pagerank update links-to Learned weights links-to voiced-by • Only propagate embeddings from nodes with non-zero pagerank score Megatron "Meg" Griffin is a character from the television series • This constrains learning along valid paths Family Guy. in the graph Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Classify each entity as Answer / Not-Answer: Meg Griffin log has-character links-to T hv : Entity Representation links-to voiced-by Megatron "Meg" Griffin is a character from the television series Family Guy. Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Summary: Meg Griffin 1. Inject text into KB using retrieval has-character links-to 2. Propagate entity embeddings along paths starting from links-to voiced-by question entities Megatron "Meg" Griffin 3. Classify each entity as answer / is a character from the television series no-answer Family Guy. Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Evaluation — WebQuestionsSP & WikiMovies Evaluation — WebQuestionsSP & WikiMovies • WebQuestionsSP [Berant, 2013; Yih, 2016]: E.g. “What language do they speak in Afghanistan?” – Pashto language • KB - Freebase, Text - Wikipedia • 5K QA pairs for training Evaluation — WebQuestionsSP & WikiMovies • WebQuestionsSP [Berant, 2013; Yih, 2016]: E.g. “What language do they speak