Bhuwan Dhingra Language Technologies Institute Carnegie Mellon University Work Done With

Text as a Virtual Knowledge Base

Bhuwan Dhingra Language Technologies Institute Carnegie Mellon University

Work done with: Haitian Sun, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Russ Salakhutdinov, William Cohen Question Answering

When did Kendrick Lamar’s ﬁrst album come out?

A. July 2, 2011

Source: Natural Questions (https://ai.google.com/research/NaturalQuestions/dataset) Sources of Information Text Knowledge Bases Sources of Information Text Knowledge Bases

[Moldovan, 2002], [Voorhees, 1999], [Kwiatkowski, 2013], [Berant, 2013], [Ferrucci, 2012], [Yih, 2013], [Reddy, 2014], [Pasupat, 2015], [Hermann, 2016], [Chen, 2017], [Seo, [Bordes, 2015], [Yih, 2015], [Jain, 2017], [Peters, 2018], [Devlin, 2018] … 2016], [Liang, 2017], [Das, 2017] … Sources of Information Text Knowledge Bases

• Lexical pattern matching • Clear semantics via parsing

• No grounding • Grounded

• High recall • High precision Sources of Information Text Knowledge Bases

[Moldovan, 2002],[Gardner [Voorhees, & 1999], Krishnamurthy, 2017],[Kwiatkowski, [Ryu, 2017] 2013], [Berant, 2013], [Ferrucci,Universal 2012], Schema [Yih, 2013], [Riedel, 2013], [Verga,[Reddy, 2016], 2014], [Das, [Pasupat, 2017] 2015],… [Hermann, 2016], [Chen, 2017], [Seo, [Bordes, 2015], [Yih, 2015], [Jain, 2017], [Peters, 2018], [Devlin, 2018] … Our Work 2016], [Liang, 2017], [Das, 2017] …

• Lexical pattern matching • Clear semantics via parsing

• No grounding • Grounded

• High recall • High precision Why Text + KBs? Why Text + KBs?

• Engineering motivation - QA performance

• Text can complete missing information in KBs

• KBs can provide background context for understanding text Why Text + KBs?

• Engineering motivation - QA performance

• Text can complete missing information in KBs

• KBs can provide background context for understanding text

• Scientiﬁc motivation - Knowledge Representation

• Text is expressive

• KBs support reasoning This Talk

1. “Reading” heterogeneous graphs of 2. Traversing text corpora like KBs facts and text [EMNLP’18] [ongoing] Open-domain QA using Early Fusion of KBs & Text Haitian Sun*, Bhuwan Dhingra*, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, William Cohen *equal contribution EMNLP 2018 Entity-Relation Knowledge Bases

Meg Griffin

has-character

network • Collection of (subject, relation, object) facts voiced-by

• Organize information around entity nodes network Mila Kunis Lacey Chabert

starred in • Freebase: 44M entities, >2B facts

directed-by

David Trainer Reasoning in KBs

Meg Griffin Relation following operation: has-character

network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by

network Mila Kunis Lacey Chabert

starred in

directed-by

David Trainer Reasoning in KBs

Meg Griffin Relation following operation: has-character

network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by

• Given a set of entities X

network Mila Kunis Lacey Chabert

starred in

directed-by

David Trainer Reasoning in KBs

Meg Griffin Relation following operation: has-character

network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by

• Given a set of entities X

• Follow edges labeled with the relation R network Mila Kunis Lacey Chabert

starred in

directed-by

David Trainer Reasoning in KBs

Meg Griffin Relation following operation: has-character

network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by

• Given a set of entities X

• Follow edges labeled with the relation R network Mila Kunis Lacey Chabert

starred in • To arrive at a set of entities Y

directed-by

David Trainer Reasoning in KBs

Meg Griffin Relation following operation: has-character

network Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds { 9 2 } voiced-by

• Given a set of entities X

• Follow edges labeled with the relation R network Mila Kunis Lacey Chabert

starred in • To arrive at a set of entities Y

E.g. {Family_Guy, directed-by That_70s_Show}.follow(network) = {Fox} David Trainer Question Answering

Meg Griffin Who voiced Meg in Family Guy? has-character

network

voiced-by

network Mila Kunis Lacey Chabert

starred in

directed-by

David Trainer Question Answering

Meg Griffin Who voiced Meg in Family Guy? has-character

network

voiced-by

network Mila Kunis Lacey Chabert

starred in

directed-by

David Trainer Question Answering

Meg Griffin Who voiced Meg in Family Guy? has-character

network

voiced-by

{Meg_Griffin}.follow(voiced-by)

network Mila Kunis Lacey Chabert

starred in

directed-by

David Trainer Question Answering

Meg Griffin Who voiced Meg in Family Guy? has-character

network

voiced-by

{Meg_Griffin}.follow(voiced-by)

network Mila Kunis Lacey Chabert

starred in {Mila_Kunis, Lacey_Chabert}

directed-by

David Trainer Question Answering

Meg Griffin Who voiced Meg in Family Guy? has-character

network Annotating semantic parses of voiced-by questions is expensive

{Meg_Griffin}.follow(voiced-by)

network Mila Kunis Lacey Chabert

starred in {Mila_Kunis, Lacey_Chabert}

directed-by

David Trainer Question Answering

Meg Griffin Who voiced Meg in Family Guy? has-character

network Search for parses which lead to voiced-by the correct answer in the KB

???

network Mila Kunis Lacey Chabert

starred in {Mila_Kunis, Lacey_Chabert}

directed-by Learning from Denotations David Trainer [Liang, 2011], [Berant, 2013], [Reddy, 2014], [Krishnamurthy, 2017] But KBs are often incomplete

Meg Griffin

has-character

network

voiced-by

network Mila Kunis Lacey Chabert

starred in Min et al, NAACL 2013

directed-by

David Trainer Graphs of Facts and Text

Meg Griffin

has-character Who voiced Meg in Family Guy?

voiced-by

Mila Kunis Lacey Chabert Graphs of Facts and Text

Meg Griffin Inject relevant text into the graph

has-character Who voiced Meg in Family Guy?

voiced-by

Mila Kunis Lacey Chabert Graphs of Facts and Text

Meg Griffin Inject relevant text into the graph

has-character Who voiced Meg in Family Guy?

voiced-by TF-IDF Retrieval

Mila Kunis Lacey Chabert Graphs of Facts and Text

Meg Griffin Inject relevant text into the graph

has-character Who voiced Meg in Family Guy? links-to links-to voiced-by

Megatron "Meg" Griffin TF-IDF is a character from the Retrieval television series Family Guy.

Mila Kunis Lacey Chabert

Entity Linking Search via Representation Learning

Meg Griffin

has-character links-to

links-to voiced-by

Megatron "Meg" Griffin is a character from the television series Family Guy.

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Graph Neural Networks

Meg Griffin Embedding Vector

has-character links-to

links-to voiced-by

Megatron "Meg" Griffin is a character from the television series Family Guy.

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning Graph Neural Networks

Meg Griffin Embedding Vector

has-character links-to “Pagerank” score - Initially uniform over entities in the links-to question voiced-by

Megatron "Meg" Griffin is a character from the television series Family Guy.

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning

Meg Griffin

has-character links-to

links-to voiced-by

Megatron "Meg" Griffin is a character from the television series Family Guy.

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning

Meg Griffin

has-character Self connection links-to Entity neighbors Text mentions links-to voiced-by

Megatron "Meg" Griffin is a character from the television series Family Guy.

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning

Meg Griffin

has-character Self connection links-to Entity neighbors Text mentions links-to voiced-by • Only propagate embeddings from nodes with non-zero pagerank score Megatron "Meg" Griffin is a character from the television series • This constrains learning along valid paths Family Guy. in the graph

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning

Meg Griffin

has-character Pagerank update links-to Learned weights links-to voiced-by • Only propagate embeddings from nodes with non-zero pagerank score Megatron "Meg" Griffin is a character from the television series • This constrains learning along valid paths Family Guy. in the graph

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning

Classify each entity as Answer / Not-Answer: Meg Griffin log

has-character links-to T hv : Entity Representation

links-to voiced-by

Megatron "Meg" Griffin is a character from the television series Family Guy.

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Search via Representation Learning

Summary: Meg Griffin 1. Inject text into KB using retrieval

has-character links-to 2. Propagate entity embeddings along paths starting from links-to voiced-by question entities

Megatron "Meg" Griffin 3. Classify each entity as answer / is a character from the television series no-answer Family Guy.

Mila Kunis Lacey Chabert Who voiced Meg in Family Guy? Evaluation — WebQuestionsSP & WikiMovies Evaluation — WebQuestionsSP & WikiMovies

• WebQuestionsSP [Berant, 2013; Yih, 2016]: E.g. “What language do they speak in Afghanistan?” – Pashto language

• KB - Freebase, Text - Wikipedia

• 5K QA pairs for training Evaluation — WebQuestionsSP & WikiMovies

• WebQuestionsSP [Berant, 2013; Yih, 2016]: E.g. “What language do they speak in Afghanistan?” – Pashto language

• KB - Freebase, Text - Wikipedia

• 5K QA pairs for training

• Wikimovies [Miller, 2016]: E.g. “What movies did Quentin Tarantino direct?” – Reservoir dogs, Pulp fiction, …

• KB - Subset of OMDB, Text - Subset of Wikipedia

• 10K QA pairs for training Evaluation — WebQuestionsSP & WikiMovies

• WebQuestionsSP [Berant, 2013; Yih, 2016]: E.g. “What language do they speak in Afghanistan?” – Pashto language

• KB - Freebase, Text - Wikipedia

• 5K QA pairs for training

• Wikimovies [Miller, 2016]: E.g. “What movies did Quentin Tarantino direct?” – Reservoir dogs, Pulp fiction, …

• KB - Subset of OMDB, Text - Subset of Wikipedia

• 10K QA pairs for training

• Both datasets are answerable using KBs

• But we simulate an incomplete KB setting WikiMovies Results Hits @1 Performance WikiMovies Results Hits @1 Performance Text-SOTA KB-SOTA Universal Schema Ours (KB-only) Ours (KB+Text) Watanabe et al (arxiv’17) Das et al (ICLR’18) 100 97.0

88 85.8

50 0% KB 50% KB 100% KB 10% Training Data WikiMovies Results Hits @1 Performance Text-SOTA KB-SOTA Universal Schema Ours (KB-only) Ours (KB+Text) Watanabe et al (arxiv’17) Das et al (ICLR’18) Das et al (ACL’17) 100 97.0 93.8

88 85.8 80.3 75.3 75

88 85.8 80.3 75.3 75 67.7 63

88.4 88 86.6 85.8 80.3 75.3 75 67.7 63

50 0% KB 50% KB 100% KB 10% Training Data WebQuestionsSP Results Hits @1 Performance WebQuestionsSP Results Hits @1 Performance Text-SOTA KB-SOTA Universal Schema Ours (KB-only) Ours (KB+Text) Chen et al (ACL’17) Liang et al (ACL’17) 85 ~75

21 21.5

0 0% KB 50% KB 100% KB WebQuestionsSP Results Hits @1 Performance Text-SOTA KB-SOTA Universal Schema Ours (KB-only) Ours (KB+Text) Chen et al (ACL’17) Liang et al (ACL’17) Das et al (ACL’17) 85 ~75

43 40.5 32.5

23.2 21 21.5

66.7 64

47.7 43 40.5 32.5

23.2 21 21.5

66.7 68.7 64 52.3 47.7 43 40.5 32.5

23.2 25.3 21 21.5

0 0% KB 50% KB 100% KB Analysis

• Pagerank propagation leads to ~10% improvement

• Errors:

• Retrieval (both text + facts) has only 90% recall

• Complex questions:

“Who first voiced Meg in Family Guy?” “Which club did Cristiano Ronaldo play for in 2007?” Differentiable Reasoning over a Virtual Knowledge Base Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, William Cohen Title and authors removed for anonymity In preparation Multi-Hop Question Answering

Meg Griffin

has-character

network Q. Which TV Networks has Mila Kunis appeared on?

voiced-by

network Mila Kunis Lacey Chabert

starred in

directed-by

David Trainer Multi-Hop Question Answering

Meg Griffin

has-character

network Q. Which TV Networks has Mila Kunis appeared on?

voiced-by

Y1 = {Mila_Kunis}.follow(starred-in).follow(network)

Y2 = {Mila_Kunis}.follow(voiced)....follow(network) network Mila Kunis Lacey Chabert Ans = Y1 UNION Y2

starred in A. Fox

directed-by Y = X.follow(R)= x0 : x X s.t. R(x, x0) holds David Trainer { 9 2 } Can we do this with a Text Corpus? Can we do this with a Text Corpus?

Q. Which TV Networks has Mila Kunis appeared on? Can we do this with a Text Corpus?

Q. Which TV Networks has Mila Kunis appeared on?

Since 1999, she has voiced Meg Griffin on the animated series Family Guy. Can we do this with a Text Corpus?

Q. Which TV Networks has Mila Kunis appeared on?

Since 1999, she has voiced Meg Griffin on the animated series Family Guy.

Family Guy is an American animated sitcom created by Seth MacFarlane for the Fox Broadcasting Company. Can we do this with a Text Corpus?

Q. Which TV Networks has Mila Kunis appeared on?

Since 1999, she has voiced Meg Griffin on the animated series Family Guy.

Family Guy is an American animated sitcom created by Seth MacFarlane for the Fox Broadcasting Company.

A. Fox Can we do this with a Text Corpus?

Q. Which TV Networks has Mila Kunis appeared on?

Since 1999, she has voiced Meg Griffin on the animated series Family Guy.

Family Guy is an American animated sitcom created by Seth MacFarlane for the Fox Broadcasting Company.

A. Fox Information can be spread out across the corpus MetaQA Benchmark MetaQA Benchmark

• Expands Wikimovies dataset to 2-hop and 3-hop questions MetaQA Benchmark

• Expands Wikimovies dataset to 2-hop and 3-hop questions

• 1-hop: What movies did Quentin Tarantino direct? MetaQA Benchmark

• Expands Wikimovies dataset to 2-hop and 3-hop questions

• 1-hop: What movies did Quentin Tarantino direct?

• 2-hop: Which movies have the same director as that of Pulp Fiction? MetaQA Benchmark

• Expands Wikimovies dataset to 2-hop and 3-hop questions

• 1-hop: What movies did Quentin Tarantino direct?

• 2-hop: Which movies have the same director as that of Pulp Fiction?

• 3-hop: Who acted in the movies which have the same director as Pulp Fiction? MetaQA Benchmark

• Expands Wikimovies dataset to 2-hop and 3-hop questions

• 1-hop: What movies did Quentin Tarantino direct?

• 2-hop: Which movies have the same director as that of Pulp Fiction?

• 3-hop: Who acted in the movies which have the same director as Pulp Fiction?

• Text corpus: 17K ﬁrst paragraphs from Wikipedia articles about the movies

Pulp Fiction is a 1994 American crime film written and directed by Quentin Tarantino, who conceived it with Roger Avary. Starring John Travolta, Samuel L. Jackson, Bruce Willis, Tim Roth, Ving Rhames, and Uma Thurman, it tells several stories of criminal Los Angeles. The title refers to the pulp magazines and hardboiled crime novels popular during the mid-20th century, known for their graphic violence and punchy dialogue. Graph Networks on MetaQA Hits @1 Performance Graph Networks on MetaQA Hits @1 Performance Using KB Using Text 100

0 1-hop 2-hop 3-hop Graph Networks on MetaQA Hits @1 Performance Using KB Using Text

100 97.0 94.8

77.7 75

0 1-hop 2-hop 3-hop Graph Networks on MetaQA Hits @1 Performance Using KB Using Text

100 97.0 94.8

82.5 77.7 75

50 40.2 36.2

0 1-hop 2-hop 3-hop Problem: Retrieval

Q. Which TV Networks has Mila Kunis appeared on?

Since 1999, she has voiced Meg Griffin on the animated series Family Guy.

Family Guy is an American animated sitcom created by Seth MacFarlane for the Fox Broadcasting Company.

A. Fox Information can be spread out across the corpus Problem: Retrieval

Q. Which TV Networks has Mila Kunis appeared on?

Since 1999, she has voiced Meg Griffin on the animated series Family Guy.

Family Guy is an American animated sitcom created by Seth MacFarlane for the Fox Broadcasting Company.

A. Fox Information can be spread out across the corpus

• Shallow information retrieval does not work

• Can expand retrieval using e.g. pseudo-relevance-feedback

• But “reading” large number of documents is expensive Our Approach: Read ofﬂine, Reason online Our Approach: Read ofﬂine, Reason online

Reading

Pre-train contextual representations into an index

ELMo [Peters, 2018], BERT [Devlin, 2018], Phrase-Indexed QA [Seo, 2018]

Ofﬂine and slow Our Approach: Read ofﬂine, Reason online

Reading Reasoning

Pre-train contextual Use inner product search and representations into an index sparse operations

ELMo [Peters, 2018], BERT MIPS [Andoni, 2015; Johnson, [Devlin, 2018], Phrase-Indexed 2017], Ragged Tensors QA [Seo, 2018] [Tensorﬂow]

Ofﬂine and slow Online and fast Our Approach: Read ofﬂine, Reason online

Reading Reasoning

Pre-train contextual Use inner product search and representations into an index sparse operations

ELMo [Peters, 2018], BERT MIPS [Andoni, 2015; Johnson, [Devlin, 2018], Phrase-Indexed 2017], Ragged Tensors QA [Seo, 2018] [Tensorﬂow]

Ofﬂine and slow Online and fast

Differentiable Reasoning over a KB of Indexed Text Reading: Ofﬂine Index of Entity Mentions

Based on Phrase-Indexed QA (Seo et al, EMNLP’18, ACL’19) Reading: Ofﬂine Index of Entity Mentions

Entity Mentions

… Family Guy is an American …

… their children, Meg, Chris, Stewie …

… In 1999, Kunis replaced Lacey Chabert …

Entity … created by Seth MacFarlene for Fox … Linking … cast members were Topher Grace, Mile Kunis …

… originally aired on Fox from August 23 …

… wanted the show to have a 1970s feel …

Based on Phrase-Indexed QA (Seo et al, EMNLP’18, ACL’19) Reading: Ofﬂine Index of Entity Mentions

Contextual Representations Entity Mentions

… Family Guy is an American …

… their children, Meg, Chris, Stewie …

… In 1999, Kunis replaced Lacey Chabert …

Entity … created by Seth MacFarlene for Fox … Linking … cast members were Topher Grace, Mile Kunis …

… originally aired on Fox from August 23 …

… wanted the show to have a 1970s feel …

Based on Phrase-Indexed QA (Seo et al, EMNLP’18, ACL’19) Reading: Ofﬂine Index of Entity Mentions

Contextual Representations Entity Mentions Sparse

… Family Guy is an American …

… their children, Meg, Chris, Stewie …

… In 1999, Kunis replaced Lacey Chabert …

Entity … created by Seth MacFarlene for Fox … Linking … cast members were Topher Grace, Mile Kunis …

… originally aired on Fox from August 23 …

… wanted the show to have a 1970s feel …

(Fixed)

Based on Phrase-Indexed QA (Seo et al, EMNLP’18, ACL’19) Reading: Ofﬂine Index of Entity Mentions

Contextual Representations Entity Mentions Sparse Dense

… Family Guy is an American …

… their children, Meg, Chris, Stewie …

… In 1999, Kunis replaced Lacey Chabert …

Entity … created by Seth MacFarlene for Fox … Linking … cast members were Topher Grace, Mile Kunis …

… originally aired on Fox from August 23 …

… wanted the show to have a 1970s feel …

(Fixed) (Pre-trained)

Based on Phrase-Indexed QA (Seo et al, EMNLP’18, ACL’19) Reasoning: A “Soft” Textual Follow Op

Sparse Dense

“Virtual” Knowledge Base Reasoning: A “Soft” Textual Follow Op

A soft set of entities Sparse Dense vX1 Retrieve Mention Spans A relation qR1 vector X1.follow(R1)

vX2 A soft set of entities

“Virtual” Knowledge Base Reasoning: A “Soft” Textual Follow Op

Which TV Networks has Mila Kunis appeared on? Sparse Dense vX1 Retrieve qR1 Mention Spans X1.follow(R1)

vX2

“Virtual” Knowledge Base Reasoning: A “Soft” Textual Follow Op

Which TV Networks has Mila Kunis appeared on? Sparse Dense vX1 Retrieve qR1 Mention Spans X1.follow(R1)

vX2 qR2 X2.follow(R2)

“Virtual” Knowledge Base Fox Reasoning: A “Soft” Textual Follow Op

Which TV Networks has Mila Kunis appeared on? Sparse Dense vX1 Retrieve qR1 Mention Spans X1.follow(R1)

vX2 qR2 X2.follow(R2)

Key Idea: We can do this with “Virtual” Knowledge Base sparse matrix - vector products and inner product search Fox X1.follow(R1) Which TV Networks has Mila Kunis appeared on?

Sparse Dense X1.follow(R1) Which TV Networks has Mila Kunis appeared on?

vX1 k-hot sparse vector of entities Sparse Dense

Mila_Kunis Mila_(ﬁlm)

Entities are represented as a sparse vector

• Size = # entities in the corpus

• Non-zero values are conﬁdence scores

• E.g. from an entity linking model X1.follow(R1) Which TV Networks has Mila Kunis appeared on?