From Strings to Things: KELVIN in TAC KBP and EDL Tim Finin1, Dawn Lawrie2, James Mayfield, Paul McNamee and Craig Harman Human Language Technology Center of Excellence Johns Hopkins University 1University of MarylanD, Baltimore County 2Loyola University MarylanD December 2017 Kelvin • KELVIN: Knowledge Extraction, Linking, Validation anD Inference • Developed at the Human Language Technology Center of Excellence at JHU anD useD in TAC KBP (2010-17), EDL (2015-17) anD other projects • Takes English, Chinese & Spanish Documents anD produce a knowledge graph in several formats • We’ll review its monolingual processing, look at the multi-lingual use case NIST TAC NIST Text Analysis Conference • Annual evaluation workshops since 2008 on natural language processing & relateD applications with large test collections anD common evaluation procedures • Knowledge Base Population (KBP) tracks focus on building KBs from information extracteD from text • Cold Start KBP: construct a KB from text • Entity discovery & linking: cluster anD link entity mentions • Slot filling • Slot filler valiDation • Sentiment • Events: Discover anD cluster events in text http://nist.gov/tac NIST TAC Cold Start When Lisa's mother After two years in Marge Simpson went the academic to a weekend quagmire of getaway at Rancho Springfield Relaxo, the movie Elementary, Lisa The Happy Little finally has a teacher Elves Meet Fuzzy that she connects Snuggleduck was one with. But she soon of the R-rated learns that the european adult problem with being movies available on middle-class is that their cable channels. After two years in When Lisa's mother the academic Marge Simpson went quagmire of to a weekend Springfield getaway at Rancho Elementary, Lisa Relaxo, the movie finally has a teacher The Happy Little that she connects Elves Meet Fuzzy Springfield with. But she soon Snuggleduck was one learns that the of the R-rated problem with being european adult middle-class is that movies available on their cable channels. Marge Simpson Homer Simpson Springfield Elementary Lisa Simpson Bart Simpson After two years in When Lisa's mother the academic Marge Simpson went quagmire of to a weekend Springfield getaway at Rancho Elementary, Lisa Relaxo, the movie finally has a teacher The Happy Little that she connects Elves Meet Fuzzy Springfield with. But she soon Snuggleduck was one learns that the of the R-rated problem with being european adult middle-class is that Bottomless Pete, Nature’s movies available on their cable channels. Cruelest Mistake per:alternate_names per:cities_of_residence Marge Simpson Homer Simpson per:children per:children Springfield Elementary per:schools_attended Lisa Simpson Bart Simpson Entity-Valued Relations Relation Inverse(s) per:children per:parents per:other_family per:other_family per:parents per:children per:siblings per:siblings per:spouse per:spouse per:employee_of {org,gpe}:employees* per:member_of org:membership* per:schools_attended org:students* per:city_of_birth gpe:births_in_city* per:stateorprovince_of_birth gpe:births_in_stateorprovince* per:country_of_birth gpe:births_in_country* per:cities_of_residence gpe:residents_of_city* per:statesorprovinces_of_residence gpe:residents_of_stateorprovince per:countries_of_residence gpe:residents_of_country* per:city_of_death gpe:deaths_in_city* per:stateorprovince_of_death gpe:deaths_in_stateorprovince* per:country_of_death gpe:deaths_in_country* org:shareholders {per,org,gpe}:holds_shares_in* org:founded_by {per,org,gpe}:organizations_founded* org:top_members_employees per:top_member_employee_of* {org,gpe}:member_of org:members org:members {org,gpe}:member_of org:parents {org,gpe}:subsidiaries org:subsidiaries org:parents org:city_of_headquarters gpe:headquarters_in_city* org:stateorprovince_of_headquarters gpe:headquarters_in_stateorprovince* org:country_of_headquarters gpe:headquarters_in_country* String-Filled Relations per:alternate_names org:alternate_names per:date_of_birth org:political_religious_affiliation per:age org:number_of_employees_members per:origin org:date_founded per:date_of_death org:date_dissolved per:cause_of_death org:website per:title per:religion per:charges Cold Start Schema per:children per:other_family per:parents per:siblings per:spouse per:employee_of per:member_of per:schools_attended per:city_of_birth per:stateorprovince_of_birth per:country_of_birth per:cities_of_residence per:statesorprovinces_of_reside nce per:countries_of_residence per:city_of_death per:stateorprovince_of_death per:country_of_death org:shareholders org:founded_by The Task You are given: Schema When Lisa's mother Marge SimpsonWhenwent Lisa's tomother a weekend Marge per:children getaway at Rancho Relaxo, the SimpsonWhenwent Lisa's tomother a weekend Marge per:other_family movie The Happy Little Elves getawaySimpson at Ranchowent toRelaxo, a weekend the per:parents Meet Fuzzy SnuggleduckWhen Lisa's wasmother Marge moviegetaway The Happy at Rancho Little Relaxo, Elves the per:siblings one of the RSimpson-ratedWhen europeanwent Lisa's to mother a weekend Marge Meetmovie Fuzzy The Snuggleduck Happy Little was Elves per:spouse adult moviesgetaway available at on Rancho their Relaxo, the one of the RSimpson-ratedWhen europeanwent Lisa's to mother a weekend Marge Meetmovie Fuzzy The Snuggleduck Happy Little was Elves per:employee_of cableadult channels. moviesgetaway available at on Rancho their Relaxo, the one of the RSimpson-ratedWhen europeanwent Lisa'sWhen to mother a weekend Lisa's Marge mother per:member_of Meetmovie Fuzzy The Snuggleduck Happy Little was Elves cableadult channels. moviesgetaway available at on Rancho their Relaxo, the per:schools_attended one of the RSimpson-ratedWhen europeanMargewent Lisa's to mother Simpsona weekend Marge went cable channels.Meetmovie Fuzzy The Snuggleduck Happy Little was Elves per:city_of_birth adult moviesgetaway availableSimpson toat on Ranchoa theirwent weekend toRelaxo, a weekend thegetaway one Meetof the Fuzzy R-rated Snuggleduck european was per:stateorprovince_of_bi cable channels.moviegetaway The Happy at Rancho Little Relaxo, Elves the adultone movies of the available Rat-rated Rancho on european their Relaxo, the rth Meetmovie Fuzzy The Snuggleduck Happy Little was Elves cableadult channels. movies available on their per:country_of_birth one of themovie R-rated The european Happy Little cable channels.Meet Fuzzy Snuggleduck was per:cities_of_residence adultone movies Elvesof the available RMeet-rated on europeanFuzzy their per:statesorprovinces_of_ cableadult channels.Snuggleduck movies available was on their one residence cable channels. per:countries_of_residenc of the R-rated e european adult movies per:city_of_death available on their cable per:stateorprovince_of_de ath channels. per:country_of_death org:shareholders org:founded_by When Lisa's mother Marge Schema SimpsonWhenwent Lisa's tomother a weekend Marge getaway at Rancho Relaxo, the SimpsonWhenwent Lisa's tomother a weekend Marge movie The Happy Little Elves per:children getawaySimpson at Ranchowent toRelaxo, a weekend the Meet Fuzzy SnuggleduckWhen Lisa's wasmother Marge per:other_family moviegetaway The Happy at Rancho Little Relaxo, Elves the one of the RSimpson-ratedWhen europeanwent Lisa's to mother a weekend Marge per:parents Meetmovie Fuzzy The Snuggleduck Happy Little was Elves adult moviesgetaway available at on Rancho their Relaxo, the per:siblings one of the RSimpson-ratedWhen europeanwent Lisa's to mother a weekend Marge Meetmovie Fuzzy The Snuggleduck Happy Little was Elves per:spouse cableadult channels. moviesgetaway available at on Rancho theirWhen Relaxo, Lisa' the s mother You Must Produce: one of the RSimpson-ratedWhen europeanwent Lisa's to mother a weekend Marge per:employee_of cable channels.Meetmovie Fuzzy The Snuggleduck Happy Little was Elves adult moviesgetaway availableSimpson at on RanchoMarge theirwent toRelaxo, Simpsona weekend the went per:member_of one Meetof the Fuzzy R-rated SnuggleduckWhen european Lisa's wasmother Marge cable channels.moviegetaway The Happy at Rancho Little Relaxo, Elves the per:schools_attended adultone movies of the available RSimpson-ratedto on european a theirwent weekend to a weekend getaway Meetmovie Fuzzy The Snuggleduck Happy Little was Elves per:city_of_birth cableadult channels. moviesgetaway availableat Rancho at on Rancho their Relaxo, Relaxo, the the one Meetof the Fuzzy R-rated Snuggleduck european was per:stateorprovince_of_b cable channels.movie The Happy Little Elves adultone movies of themovie available R-rated The on european their Happy Little irth Meet Fuzzy Snuggleduck was cableadult channels. movies available on their per:country_of_birth one Elvesof the RMeet-rated europeanFuzzy cable channels. per:cities_of_residence adultSnuggleduck movies available was on their one per:statesorprovinces_of Springfield cableof channels. the R-rated _residence per:countries_of_residen european adult movies ce available on their cable per:city_of_death channels. per:stateorprovince_of_d eath per:country_of_death org:shareholders org:founded_by Bottomless Pete, Nature’s Cruelest Mistake per:alternate_names per:cities_of_residence Marge Simpson Homer Simpson per:children per:children Springfield Elementary per:schools_attended Lisa Simpson Bart Simpson How do you know that your KB is any good? Springfield Bottomless Pete, Nature’s Cruelest Mistake per:alternate_names per:cities_of_residence Marge Simpson Homer Simpson per:children per:children Springfield Elementary per:schools_attended Lisa Simpson Bart Simpson How