Identifying Constant and Unique Relations by using Time-Series Text

Yohei Takaku Nobuhiro Kaji Naoki Yoshinaga Masashi Toyoda Toyo Keizai Inc. Institute of Industrial Science, University of Relation Extraction from the Evolving Web

• Web (text) as a growing goldmine for extracting relations between real-world entities [Pantel+ 06; Banko+ 07; Suchanek+ 07; Wu+ 08,10; Zhu+ 09; Mintz+ 09] – Processing more text leads to more relations, but – Relations in text could be obsolete / will become outdated Apple’s CEO is Apple’s CEO is Steve Jobs Tim Cook obsolete 2009 2010 2011 2012 How to Consistently Compile Extracted Relations? • • • How to Consistently Compile Extracted Relations? • • • Accumulate all the relations, because the relation does not evolve over time

Moselle river flows Moselle river flows through Germany through France

2009 2010 2011 2012 How to Consistently Compile Extracted Relations? • • • Overwrite with new one, because the relation can take one value of arg2

Apple’s CEO is Apple’s CEO is Steve Jobs Tim Cook

overwritten 2009 2010 2011 2012 Apple sells Apple sells MacBook Air iPhone 4S Constant and Unique Relations

• Given value of arg1, – Constant rel.: value of arg2 is independent of time – Unique rel.: value of arg2 is one at any point in time constant, unique non-constant, unique

constant, non-unique non-constant, non-unique Overview

• Constancy and Uniqueness of Relations • Our Approach • Features for Constancy Classification • Features for Uniqueness Classification • Experiments • Conclusion Two Binary Classification Tasks

Two Binary Classification Tasks

Task 1: constancy classification

constant non-constant

Two Binary Classification Tasks

Task 2: uniquness classification

unique

non-unique Two Kinds of Features for Training Supervised Classifiers • Frequency obtained from time-series text – Detailed later – Based on blog posts crawled from 2006 to 2011

• Linguistic cues

e.g., non-const. Prefix George Bush is ex-president of USA

e.g., non-uniq.

Coordination France borders on Italy as well as Spain Overview

• Constancy and Uniqueness of Relations • Our Approach • Features for Constancy Classification • Features for Uniqueness Classification • Experiments • Conclusion Using Time-series Text for Constancy Classification

non-const.

2008 – 2010, VVV-Venlo 2010 – now, CSK Moskow Using Time-series Text for Constancy Classification

non-const.

2008 – 2009 2010 – 2011 30 30

25 VVV Venlo 25 CSKA Moscow

20 20

15 15

10 10 frequency frequency 5 5

0 0

VVV Italy PAO VVV Italy PAO Chelsea Chelsea VVV Venlo VVV Venlo Chairman… CSKA Moscow Dutch league CSKA Moscow Dutch league Luzhniki Stadium Luzhniki Stadium Chairman Haiberuden arg2 fillers arg2 fillers Cosine Similarity as a Feature Value

non-const. 30 30 25 25 20 20 15 15 10 10 5 cosine = 0.0 5 0 0 ( 0, 27, 7, 0, 0, 0, 1, 0, 0) (19, 0, 0, 0, 0, 0, 0, 0, 1) VVV Italy PAO VVV Italy PAO CSKA… Dutch… Chelsea Luzhniki… Chelsea VVV Venlo VVV Venlo Chairman… CSKA Moscow Dutch league Luzhniki Stadium Chairman Haiberuden const. 25 25 20 20 15 15 10 cosine = 0.39 10 5 ( 4, 0, 1, 3, 1, 21, 3, 1, 1) ( 4, 52, 1, 1, 1, 1, 0, 0, 0) 0 0

Asa river Sakai river Onda river Asa river Edo river Ka nnda river Tama river Sakai river Onda river Meguro river SumidaMeguro river riverKa nnda river Minamiasa river Minamiasa river Importance of Choosing Time Windows

non-const.

VVV Venlo CSKA Moscow 30 30 25 25 20 20 15 15 10 10 5 5 0 0

VVV Italy PAO VVV Italy PAO Chelsea Chelsea VVV Venlo VVV Venlo CSKA Moscow Dutch league CSKA Moscow Dutch league Luzhniki Stadium Luzhniki Stadium Chairman Haiberuden Chairman Haiberuden

X 2009 2010 2011 2012 Team changed Importance of Choosing Time Windows

non-const.

CSKA Moscow CSKA Moscow 30 30 25 25 20 20 15 15 10 10 5 5 0 0

VVV Italy PAO VVV Italy PAO Chelsea Chelsea VVV Venlo VVV Venlo CSKA Moscow Dutch league CSKA Moscow Dutch league Luzhniki Stadium Luzhniki Stadium Chairman Haiberuden Chairman Haiberuden

X 2009 2010 2011 2012 Team changed Using Multiple Time Windows

0.4 + 0.3 + 0.0 = 0.233 3

0.4 0.3 40 30 30 20 10 10 0 ...... -10 -10 ... VVV Dutc…Chai… PAO CSKA… VVV PAO VVV PAO CSKA… Dutch…Chair… CSKA… Dutch…Chair…

T months 0.0 time 2009 2010 2011 2012

Window size T = { 1, 3, 6, 12 (months) } 12 (= 4 x 3) features Integration method = { ave., min., max. } Overview

• Constancy and Uniqueness of Relations • Our Approach • Features for Constancy Classification • Features for Uniqueness Classification • Experiments • Conclusion Using Time-series Text for Uniqueness Classification

uniq. non-uniq.

…… 30 30

25 25 20 20 12 entities 15 single entity 15 10 10 5 5 number of entity types 0 0

Slide DSP VVV Italy PAO Fflick Jaiku SayNowTwitter Chelsea YouTubeGroupon Demand Widevine PushLife VVV Venlo Chairman… FeedBurner CSKA Moscow Dutch league Luzhniki Stadium arg2 fillers arg2 fillers Using Time-series Text for Uniqueness Classification (Cont.)

uniq. non-uniq.

Frequency ratio between the 1st and 2nd most frequent entities 30 30

25 25 19 20 20 15 15 1/19 ≒ 0.05 9 7/9 = 0.77 10 10 7 5 5 1 0 0

Slide DSP Jaiku VVV Italy PAO Fflick SayNowTwitter PushLife Chelsea YouTubeGroupon Demand Widevine VVV Venlo Chairman… FeedBurner CSKA Moscow Dutch league Luzhniki Stadium values of arg2data can be noisy values of arg2 Setting Appropriate Time Windows is also Important

uniq.

VVV Venlo CSKA Moscow 30 25 20 15 10 5 0

VVV Italy PAO Chelsea VVV Venlo CSKA Moscow Dutch league Luzhniki Stadium Chairman Haiberuden

X 2009 2010 2011 2012 Team changed Overview

• Constancy and Uniqueness of Relations • Our Approach • Features for Constancy Classification • Features for Uniqueness Classification • Experiments • Conclusion Experiments

• Evaluate our method with relations extracted from time-series text – Classifier: Passive-aggressive algorithm w/ proposed features – Time-series text: 6-year’s worth of Japanese blog posts (2.3-billion sentences)

• Conduct experiments to: – Evaluate the constancy classification – Evaluate the uniqueness classification – Investigate the impact of multiple window sizes Data and Settings

• Parse time-series text to extract dependency paths connecting two named entities as relation instances

• Annotate 1000 relations (majority vote, 3 humans) Constancy Uniqueness Kappa [Fleiss 1971] 0.346 (fair) 0.428 (moderate) – Major reason for disagreement: type ambiguity Ex. non-const. const. • Use the labeled relations for training & testing – Evaluation metric: Precision & Recall (5-fold cross validation) Classification Result: Constancy

• Varying the threshold to classifier’s output (margin) to plot recall-precision curve – Baseline: cosine similarity between distributions over arg2 in the first and last month

Recovered most of the constant Proposed relations while keeping precision

Baseline Suffered from data sparseness Classification Result: Uniqueness

• Varying the threshold to classifier’s output (margin) to plot recall-precision curve – Baseline: re-implementation of [Lin+ 10] (based on gross distributions over arg2)

Proposed Time-series distributions over arg2 greatly improved the precision

Confused non-constant, unique Baseline rels. with non-unique rels. Impact of using Multiple Time Windows, T

• Compare our method (multiple time windows) with methods using a single time window

Constancy Uniqueness

– Combining features computed from multiple time windows greatly improved the precision of uniqueness classification Error Analysis

• Investigate 200 misclassified relations

Error Type Const. Uniq. Total Paraphrases 16 52 68

Topicalex. uniq. Short/longex. 47non-uniq. co-starred in a film th Obsolete/Speculativeex. uniq.10 19Apr. 7 ,29 2012 st uniq. Apr. 21 , 2012 ex. uniq. (outdated fact) const. (speculation) Related Work

• TempEval temporal relation identification [Verhagen+ 2007, 2010] – Associate event (relation instance) with time ex. OVERLAP 1889 – Do not address constancy of relation

• Functional relation identification [Ritter+ 10] – Use distributions over arg2 to identify uniqueness [Lin+ 10] – Time dimension should be considered to identify uniqueness ex. uniq. Overview

• Constancy and Uniqueness of Relations • Our Approach • Features for Constancy Classification • Features for Uniqueness Classification • Experiments • Conclusion

2019/11/7 EMNLP-CoNLL2012 31 Conclusion

• A novel notion of constancy of relations • A method of identifying constant and unique relations – Use massive time-series text to induce features – Timer-series distributions over arg2 computed with multiple time windows were quite effective

• Future work – Apply our method to relations with typed arguments [Lin+ 10] Ex. const. – Use constancy/uniqueness to compile extracted relations