Survey on English Entity Linking on Wikidata

Semantic Web 0 (2021) 1 1 IOS Press

1 1 2 2 3 3 4Survey on English Entity Linking on Wikidata 4 5 5 6 6 7Approaches and Datasets 7 8 8 9Cedric Möller a,*, Jens Lehmann a,b and Ricardo Usbeck a 9 10a NetMedia Department, Fraunhofer IAIS, Zwickauer Straße 46, 01069 Dresden, Germany 10 11E-mail: fi[email protected] 11 12b University of Bonn, Endenicher Allee 19a, 53115, Bonn, Germany 12 13E-mail: [email protected] 13 14 14 15 15 16 16 17 17 18Abstract. Wikidata is an always up-to-date, community-driven, and multilingual knowledge graph. Hence, Wikidata is an attrac- 18 19tive basis for Entity Linking, which is evident by the recent increase in published papers. This survey focuses on four subjects: 19 20(1) How do current Entity Linking approaches exploit the specific characteristics of Wikidata? (2) Which unexploited Wikidata 20 21characteristics are worth to consider for the Entity Linking task? (3) Which Wikidata Entity Linking datasets exist, how widely 21 used are they and how are they constructed? (4) Do the characteristics of Wikidata matter for the design of Entity Linking datasets 22 22 and if so, how? 23 23 24Our survey reveals that most Entity Linking approaches use Wikidata in the same way as any other knowledge graph missing 24 25the chance to leverage Wikidata-specific characteristics to increase quality. Almost all approaches employ specific properties 25 26like labels and sometimes descriptions but ignore characteristics like the hyper-relational structure. Thus, there is still room 26 27for improvement, for example, by including hyper-relational graph embeddings or type information. Many approaches also 27 include information from Wikipedia which is easily combinable with Wikidata and provides valuable textual information which 28 28 is Wikidata lacking. 29 29 30The current Wikidata-specific Entity Linking datasets do not differ in their annotation scheme from schemes for other knowl- 30 31edge graphs like DBpedia. The potential for multilingual and time-dependent datasets, naturally suited for Wikidata, is not lifted. 31 32 32 Keywords: Entity Linking, Entity Disambiguation, Wikidata 33 33 34 34 35 35 36 36 1. Introduction Knowledge Graph (KG). That is, the elements are 37 37 added and edited by the community. The number of 38 38 1.1. Motivation active editors is continuously increasing, see Figure 2. 39 39 This allows Wikidata to stay up-to-date while automat- 40 40 Entity Linking (EL) is the task of connecting already ically, one-time generated KGs such as Yago4 or Free- 41 41 marked mentions in an utterance to their correspond- base become outdated over time [91]. Note, DBpedia 42 42 ing entities in a knowledge base, see Figure 1. stays also up-to-date but has a delay of a month.1 DB- 43 43 pedia Live [21] exists, which is consistently updated 44There are multiple knowledge bases such as DB- 44 with Wikipedia information. But it is more challeng- 45pedia [68], Freebase [10], Yago4 [108] or Wiki- 45 ing to work with as no full dump is provided. Further- 46data [120]. In contrast to DBpedia, Yago4, or Free- 46 more, the DBpedia ontology is not continuously up- 47base, which mostly extract information from exist- 47 dated, for example, with new emerging classes. The 48ing sources, Wikidata is a curated, community-based 48 49addition of new classes only comes with an update 49 50*Corresponding author. E-mail: 50 51fi[email protected]. 1https://release-dashboard.dbpedia.org/ 51

1into the field of Wikidata EL. Similarly, dataset land- 1 2scape is structured, which helps researchers finding the 2 3KG correct dataset for their EL problem. 3 4(Wikidata) 4 5 5 6 6 710 7 8 8 Ennio Quentin Inglourious 8 9Tarantino Morricone Basterds 9 10(Q3772) (Q23848) (Q153723) 10 6 11 11 12 12 4 13 13 Published papers 14In 2009, Tarantino originally wanted Morricone to compose the 14 film score for Inglourious Basterds. 2 15 15 16 16 0 17Fig. 1. Entity Linking - Mentions in text are linked to the correspond- 2017 2018 2019 2020 17 Years 18ing entities (color-coded) in a knowledge base (here: Wikidata). 18 19 19 20of the mapping-based extraction. On the other hand, Fig. 3. Publishing years of included Wikidata EL papers. 20 21new classes in Wikidata can be added continuously by 21 22the community. Furthermore, Wikidata is an inherently The focus of this survey lies on EL approaches, which 22 23multilingual knowledge base. Both of these factors at- operate on already marked mentions of entities, as the 23 24tract novel EL research over Wikidata in recent years task of Entity Recognition (ER) is much less depen- 24 25cf. Figure 3. While Wikidata has its advantages regard- dent on the characteristics of a KG. However, due to 25 26ing EL, exploiting those, for example in the form of the only recent uptake of research on EL on Wikidata 26 27hyper-relational structure (see Figure 4 for an example there is only a low number of EL-only publications. To 27 28graph), is also challenging. broaden the survey’s scope, we also consider methods 28 29that include the task of ER. We do not restrict ourselves 29 3050000 to either rule-, statistical- or deep learning-based algo- 30 31rithms on Wikidata. This survey limits itself to the En- 31 3240000 glish language as it is the most dominant language in 32 33EL, and thus a better comparison of the approaches and 33 3430000 datasets is possible. Nevertheless, the topic of multilin- 34 35gualism is still of relevance in the analyses and discus- 35 20000 36sions, as it is an essential characteristic of Wikidata. 36 Active editors 37Since all multilingual Entity Linkers found also target 37 3810000 38 English, none were excluded. 39 39 400 40 1.2. Research Questions and Contributions 41 41 2011 2012 2013 2014 2015 2016 2017 2018 2019 2020 2021 42Time 42 43EL approaches use many different kinds of informa- 43 44Fig. 2. Active editors in Wikidata [126]. tion like labels, popularity measures, graph structure, 44 45and more. This multitude of possible signals raises the 45 46Primarily, this survey strives to expose the benefits and question of how the characteristics of Wikidata are 46 47associated challenges stemming from the effective use used by the current state of the art of EL over Wikidata. 47 48of Wikidata as the target KG for EL. Additionally, Thus, the first research question is: 48 49the survey provides a concise overview of existing ap- 49 50proaches, which is essential to (1) avoid duplicated re- RQ 1: How do current Entity Linking approaches 50 51search in the future and (2) enable a smoother entry exploit the specific characteristics of Wikidata? 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 3

1Wikidata in mind and if so, in what way? Thus, we 1 2post the following two research questions: 2 388th Academy 3 The Hatefull Eight Awards 4(Q18225084) 4 (Q20022969) RQ 3: Which Wikidata EL datasets exist, how 5widely used are they and how are they constructed? 5 6 6 7RQ 4: Do the characteristics of Wikidata matter for 7 8the design of EL datasets and if so, how? 8 (P805) (P1686) for work

9subject of 9 statement is 10To answer those two last research questions, all cur- 10 11rent Wikidata-specific EL datasets are gathered and an- 11 12Ennio alyzed with the research questions in mind. Further- 12 Academy Award for Morricone 13Best Original Score more, we discuss how the characteristics of Wikidata 13 (Q23848) nominated for 14(P1411) (Q488651) might affect the design of datasets. 14 15 15 This survey makes the following contributions: 16 16 instance of 17 17 child (P40) (P31) – A concise list of future research avenues. 18 18 – A list and comparison of datasets focusing on Wiki- 19 19 data. 20 20 Andrea – An analysis of current evaluation results. human 21Morricone 21 (Q5) – A discussion of the relevance of Wikidata for Entity 22(Q494547) instance of 22 (P31) Linking. 23 23 24 24 25 25 Fig. 4. Wikidata subgraph - Dashed rectangle represents a claim with 2. Survey Methodology 26attached qualifiers. 26 27There are several different types of surveys which de- 27 28In particular, which Wikidata-specific characteristics sire to accomplish different contributions to the re- 28 29contribute to the solution? We answer this question by search field [59]: 29 30gathering all existing approaches working on Wikidata 30 31systematically (see Section 2) and analyzing them. The 1. Providing an overview of the current prominent ar- 31 32 32 focus lies mainly on the usage of Wikidata’s graph eas of research in a field 33 33 characteristics. 2. Identification of open problems 343. Providing a novel approach tackling the extracted 34 35Secondly, we identify what kind of characteristics of open problems (in combination with the identifica- 35 36Wikidata are of importance for EL but are insuffi- tion of open problems) 36 37ciently considered. This raises the second research 37 38question: Our related work section analyses different recent and 38 39older surveys on EL and highlights specific areas not 39 40RQ 2: Which unexploited Wikidata characteristics covered and our survey’s novelties. While some very 40 41are are worth to consider for the Entity Linking recent surveys exist, they do not consider the different 41 42task? underlying Knowledge Graphs as a significant factor 42 43affecting the performance of EL approaches. Further- 43 44We tackle this question by giving an overview of the more, barely any approaches included in other surveys 44 45structure of Wikidata and the amount of information are working on Wikidata and take the particular char- 45 46it contains, and then discussing the potential and chal- acteristics of Wikidata into account. To fill in the gaps, 46 47lenges for EL. our survey gives an overview and examines all current 47 48EL approaches and datasets, focusing on Wikidata. 48 49Furthermore, we want to give an overview of which Additionally, we identify less-utilized but promising 49 50datasets for EL over Wikidata exist. Lastly, it is of in- characteristics of Wikidata regarding EL. Therefore, 50 51terest if it is essential that datasets are designed with this survey provides contributions 2 and 3. 51 4 C. Möller et al. / Survey on English Entity Linking on Wikidata

1Table 1: Qualifying and disqualifying criteria for approaches. 1 2 2 3Criteria 3 4Must satisfy all Must not satisfy any 4 5 5 6 6 7– Approaches that consider the problem – Approaches conducting Semi- 7 8of unstructured EL over Knowledge structured EL 8 9Graphs – Approaches not doing EL in the En- 9 10– Approaches where the target Knowl- glish language 10 11edge Graph is Wikidata 11 12 12 13 13 14Until December 18, 2020, we continuously searched tence of Wikidata. The systematic search process re- 14 15for existing and newly released scientific work suitable sulted in 150 papers and theses (including duplicates). 15 16for the survey. Note, this survey includes only scien- 16 Following this search, the resulting papers were fil- 17tific articles that were accessible to the authors.2 17 18tered again using the qualifying and disqualifying cri- 18 teria. This resulted in 16 papers and one master thesis 192.1. Approaches 19 20in the end. 20 21 21 This survey’s qualifying and disqualifying criteria for The search resulted in papers in the period from 22 22 including papers can be found in Table 1. "Semi- 2018 to 2020. While there exist EL approaches from 23 23 structured" in this table means that the entity men- 2016 [4, 106] working on Wikidata, they did not qual- 24 24 tions do not occur in natural language utterances but ify according to the criteria above. 25 25 more structured formats such as tables. The different 262.2. Datasets 26 27approaches were searched for by using multiple differ- 27 ent search engines (see Table 3). 28The dataset search was conducted in two ways. First, a 28 29To gather a wide choice of approaches the follow- search for potential datasets was performed using mul- 29 30ing filters were applied. Any approach where Wiki- tiple search engines, see Table 3. Second, the datasets 30 31data was not occurring once in the full text was not on which the approaches were evaluated were consid- 31 32considered. Entity Linking or Entity Dis- ered. The criteria for the inclusion of a dataset can be 32 33ambiguation had to occur in the title of the paper. found in Table 2. 33 34 34 The publishing year was not a criterion due to the small 35We scanned the dataset papers in the following way. 35 number of valid papers and the relatively recent exis- 36First, in the title, Entity Linking or Entity 36 37Disambiguation had to occur once. Due to those 37 382https://www.projekt-deal.de/max-planck-gesellschaft- keywords, other datasets suitable for EL but con- 38 39verzichtet-ab-2019-auf-elsevier/ structed for a different purpose like KG population 39 40 40 Table 2: Qualifying and disqualifying criteria for the dataset search. 41 41 42 42 Criteria 43 43 44Must satisfy all Must not satisfy any 44 45 45 46 46 – Datasets that are designed for EL or are – Datasets without English utterances 47 47 used for evaluation of Wikidata EL 48 48 – Datasets must include Wikidata identi- 49 49 fiers from the start 50 50 51 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 5

1Table 3: Search engines. Entity Recognition. ER is the task of identifying the 1 2spans 2 3Search Engines 3 4(wi, ..., wk)|0 6 i 6 k 6 n − 1 4 5– Google Scholar 5 6– Springer Link of all entities in an utterance u. Each such a span is 6 7– Science Direct called an entity mention m. The word or word se- 7 8– IEEE Xplore Digital Library quence referring to an entity is also known as the sur- 8 9– ACM Digital Library face form of the entity. An utterance can contain more 9 10than one entity, often also consisting of more than one 10 11word. Sometimes, also some broad type of an entity 11 12is classified too. Normally, those are person, loca- 12 13were not included. Additionally, dataset must oc- tion and organization. Some of the considered 13 14cur in the title and Wikidata has to appear at least approaches do this classification task and also use it to 14 15once in the full text. This resulted in 20 papers (includ- improve the EL. It is also up to debate what an entity 15 16ing duplicates). Of those, only two included Wikidata mention is. In general, a literal reference to an entity 16 17identifiers and focused on English. is considered a mention. But whether to include pro- 17 18nouns or how to handle overlapping mentions depends 18 19Eighteen datasets are accompanying the different ap- on the use-case. 19 20proaches. Many of those did not include Wikidata 20 Entity Linking. EL is the task of linking the recog- 21identifiers from the start. This makes them less opti- 21 nized entity mention to the correct entity in a KG. A 22mal for the examination of the influence of Wikidata 22 KG is defined as a directed graph G = (V, E, R) con- 23on the design of datasets. They are included in the sec- 23 sisting of vertices V, edges E and relations R. Often, 24tion about the approaches but not in the section about 24 vertices correspond to entities E or literals L, which 25the Wikidata datasets. 25 are concrete values like the height or a name. E is a list 26 26 After removal of duplicates, 11 Wikidata datasets are (e1,..., en) of edges with e j ∈ V × R × V where re- 27 27 included in the end. lations R specify a certain meaning for the connection 28 28 between entities. Such edges are also called triples. But 29 29 there exists no single definition of a KG; vertices and 30 30 edges can also be defined differently. A concrete defi- 313. Problem Definition 31 nition of the Wikidata KG is provided in the next sec- 32 32 tion. 33EL is the task of linking an entity mention in unstruc- 33 34tured or semi-structured data to the correct entity in a In general, EL takes the utterance u and all identi- 34 35KG. The focus of this survey lies in unstructured data, fied entity mentions M = (m1, ...mn) in the utter- 35 36namely natural language utterances. ance and links each of them to an element of the set 36 37(E ∪ {unknown}). The unknown element is added to 37 An utterance is defined as a sequence of n words. 38the set of vertices to be able to map to an unknown en- 38 39tity that is not available in the KG. Such an entity is 39 s = (w , w , ...w ) 400 1 n−1 also called a NIL or an emerging entity [52]. 40 41 41 The goal of EL is to find a mapping function that maps 42 42 Since not only approaches that solely do EL were in- all found mentions to the correct KG entities and also 43 43 cluded in the survey, Entity Recognition will also be to identify if an entity mention does not exist in the 44 44 defined. KG. 45 45 46There exists no universally agreed on definition of an EL is often split into two subtasks. First, potential can- 46 47entity. In general, named entities like a specific person didates for an entity are retrieved from a KG. This is 47 48or an organization are desirable to link. But sometimes, necessary as doing EL over the whole set of entities is 48 49also common entities, such as interview or the- often intractable. Candidate generation is usually per- 49 50ater, are included. What exactly needs to be linked, formed via efficient metrics measuring the similarities 50 51depends on the use case [95]. between entities in the utterance and entities in the KG. 51 6 C. Möller et al. / Survey on English Entity Linking on Wikidata

1The result is a set of candidates C = {c0, ··· , cl} for 4. Wikidata 1 2each entity mention m in the utterance. 2 3Wikidata is a community-driven knowledge graph 3 After limiting the space of possible entities, one of the 4edited by humans and machines. As of July 2020, it 4 available candidates is chosen for each entity. This is 5contained around 87 million items of structured data 5 done via a candidate ranking algorithm, which assigns 6about various domains. Seventy-three million items 6 7a rank to each candidate, signalizing how likely it is 7 the correct one. can be interpreted as entities due to the existence of 8a is_instance property. As a comparison, DBpe- 8 9dia contains around 5 million entities [108]. Note that 9 10the is_instance property includes a much broader 10 11 11 ranklocal : C × M → R scope of entities than the ones interpreted as entities 12for DBpedia. However, Wikidata contains around 8.5 12 given by (c, m) 7→ rank (c, m) 13local million persons while DBpedia only contains around 13 141.8 million (in October 2020). Thus, a large difference 14 15 15 where ranklocal is a ranking function of a candidate. in size is obvious. 16The goal is then to optimize the objective function: 16 17 17 4.1. Definition 18n 18 ∗ X 19A = arg max ranklocal(ai, mi)|ai ∈ Ci 19 A Wikidata is a collection of entities where each such 20i=1 20 21an entity has a page on Wikidata. An entity can be ei- 21 22where A = {a1, ..., an} ∈ P(E) is an assignment of ther an item or a property. Note that an entity in the 22 23one candidate to each entity mention mi. P(∗) is the sense of Wikidata is generally not the same as an entity 23 24power set operator. one links to via EL. For example, Wikidata entities are 24 25also properties which describe relations between dif- 25 The rank calculation of the candidates of one entity is 26ferent items. Linking to such relations is closer to Re- 26 often not independent of the other entities’ candidates. 27lation Extraction [7, 71, 103]. Furthermore, many of 27 In this case, another global ranking function will in- 28the items are more abstract classes, which are usually 28 clude the whole assignment: 29also not considered as entities linked-to in EL. Note 29 30that if not mentioned otherwise, if we speak about en- 30 rank : P(E) → given by A 7→ rank (A) 31global R global tities, entities in the context of EL are meant. 31 32 32 Item. Topics, classes, or objects are defined as items. 33The objective function is then: 33 An example of an item can be found in Figure 5. An 34 34 " n # item is enriched with more information using state- 35∗ X 35 A = arg max ranklocal(ai, mi) ments about the item itself. In general, items consist 36A 36 i=1 of one label, one description, and aliases in different 37 37 languages. An unique and language-agnostic identifier 38+ rankglobal(A) | ai ∈ Ci 38 identifies items in the form Q[0-9]+. 39 39 40Those two different categories of reranking methods For example, the item with the identifier Q23848 40 41are called local or global [90]. has the label Ennio Morricone, two aliases, Dan 41 42There exists also some ambiguity in the object of link- Savio and Leo Nichols, and Italian com- 42 43ing itself. For example, there exists an Wikidata en- poser, orchestrator and conductor 43 44tity 2014 FIFA World Cup and an entity FIFA (1928-2020) as description at the point of writing. 44 45World Cup. There is no unanimous solution on how The corresponding Wikidata page can also be seen in 45 46 46 to link the entity mention in the utterance In 2014, Figure 5. 47 47 Germany won the FIFA World Cup. 48Not all items are entities in the context of EL. In gen- 48 49Sometimes EL is also called Entity Disambiguation, eral, items which are unique instances of some class 49 50which we see more as part of EL, namely where enti- are interpreted as entities. Of course, this also depends 50 51ties are disambiguated via the candidate ranking. on the use case. 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 7

1Table 4: KG statistics by [108]. 1 2 2 3KG #Entities in million #Labels/Aliases in million last updated 3 4Wikidata 78 442 always 4 5DBpedia 5 22 monthly 5 6Yago4 67 371 November 2019 6 7 7 8 8 9Property. A property specifies a relation between one person was the spouse of another person. Ranks 9 10items/literals. Each property also has an identifier sim- are used if multiple values are valid in a statement. If 10 11ilar to an item, specified by P[0 − 9]∗. For instance, the population of a country is specified in a statement, 11 12a property P19 specifies the place of birth Rome for it might be also useful to have the populations of past 12 13Ennio Morricone. In NLP, the term relation years available. The most up-to-date population infor- 13 14is commonly used to refer to a certain connection be- mation usually has then the highest rank and is thus 14 15tween entities. A property in the sense of Wikidata is usually the most desirable claim to use. 15 a type of relation. To not break with the terminology 16Statements can be also seen in Figure 5 at the bottom. 16 used in the examined papers, when we talk about rela- 17For example, it is defined that Ennio Morricone 17 tions, we always mean Wikidata properties if not men- 18is an instance of the class human. This is also an 18 tioned otherwise. 19example for the different types of items. While En- 19 20Statement. A statement introduces information by nio Morricone is an entity in our sense, human is 20 21 21 giving structure to the data in the graph. It is specified a class. 22by a claim, and references, qualifiers and ranks related 22 23Hyper-Relational Graphs. Wikidata can thus be de- 23 to the claim. Statements are assigned to items in Wiki- 24fined as a hyper-relational knowledge graph as state- 24 data. A claim is defined as a pair of a property and 25ments can be specified by more information than a 25 some value. A value can be another item or some lit- 26single claim. Multiple properties/relations are there- 26 eral. Multiple values are possible for a property. Even 27fore part of a statement. In case of a hyper-relatio- 27 an unknown value and a no value exists. 28nal graph G = (V, E, R), E is a list (e1,..., en) 28 29References point to sources making the claims inside of edges with e j ∈ V × R × V × P(R × V) for 29 30the statements verifiable. In general, they consist of the 1 6 j 6 n, where P denotes the power set. A 30 31 31 source and date of retrieval of the claim. Qualifiers de- hyper-relational fact e j ∈ E is usually written as a tu- 32fine the value of a claim further by contextual informa- ple (s, r, o, Q), where Q is the set of qualifier pairs 32 33 33 tion. For example, a qualifier could specify how long {(qri, qvi)} with qualifier relations qri ∈ R and qual- 34 34 ifier values qvi ∈ V. (s, r, o) is referred to as the main 35 35 triple of the fact. We use the notation Q j to denote 36 36 the qualifier pairs of e j [39]. For example, under this 37representation scheme, the nominated for edge in 37 38Fig. 4 has two additional claims and would be rep- 38 39resented as (Ennio Morricone, nominated 39 40for, Academy Award for Best Original 40 41Score, (for work, The Hateful Eight), 41 42(statement is subject of, 88th Acade- 42 43my Awards)) Structures similar to qualifiers ex- 43 44ist also in some other knowledge graphs, such as the 44 45inactive Freebase in the form of Compound Value 45 46Types [10]. 46 47 47 48Other structural elements. The aforementioned ele- 48 49ments are essential for Wikidata but more do exist. For 49 50example, there are entities (in the sense of Wikidata) 50 51Fig. 5. Example of an item in Wikidata corresponding to Lexemes, Forms, Senses or Schemas. 51 8 C. Möller et al. / Survey on English Entity Linking on Wikidata

11e7 1 6.0 2 2 8 35.5 3

45.0 4 56 5 4.5 6 6 74 4.0 7 8 8

Number of items 3.5 92 9 103.0 10

11Average number of labels per item 11 0 2.5 120 500 1000 1500 2000 2500 0 500 1000 1500 2000 2500 12 Days since launch Days since launch 13 13 14(a) Number of items of Wikidata since launch [75]. (b) Average number of labels (+ aliases) per item [75]. 14 15 15

1665 80 16 17 17 60 70 18 18 1955 60 19 20 20 50 50 21 21 45 40 22 22 2340 30 23

2420 24 % of items without aliases 35

25% of items without description 25 30 10 26 26 0 500 1000 1500 2000 2500 0 500 1000 1500 2000 2500 27 27 Days since launch Days since launch 28 28 29(c) Percentage of items without any aliases [75]. (d) Percentage of items without a description [75]. 29 30 30 31Fig. 6. Statistics on Wikidata based on [75]. 31 32 32 33Yet, as those are in general not of relevance for EL, we Strengths. Due to the inclusion of information by the 33 34refrain from introducing them in more detail. community, recent events will always be included. The 34 35knowledge graph is thus much more up to date than 35 For more information on Wikidata, see the paper by other KGs. Freebase is unsupported for years now, and 36Denny Vrandeciˇ c´ and Markus Krötzsch [120]. 36 37DBpedia updates its dumps only every month. Thus, 37 38Wikidata is much more suitable and useful for indus- 38 4.2. Discussion 39try applications such as smart assistants since it is the 39 40most complete open accessible data source to date. In 40 41Novelties. As already mentioned, a useful character- Figure 6a, one can see that number of items in Wiki- 41 42istic of Wikidata is that the community can openly edit data is increasing steadily. The existence of labels and 42 43it. Another novelty is that there can be a plurality of additional aliases (see Figure 6b) helps EL as a too 43 44facts, as contradictory facts based on different sources small amount of possible surface forms often lead to a 44 45are allowed. Similarly, time-sensitive data can also be failure in the candidate generation. DBpedia does for 45 46included easily by qualiﬁers and ranks. The population example not include aliases, only a single exact label; 46 47of a country, for example, changes from year to year to compensate, additional resources like Wikipedia are 47 48which can be represented easily in Wikidata. Lastly, often used to extract a label dictionary of adequate 48 49due to their language-agnostic identiﬁers, Wikidata is size [77]. Even each property in Wikidata has a la- 49 50inherently multilingual. Language only starts playing bel [120]. Fully language-model based approaches are 50 51a role in the labels and descriptions of an item. therefore more naturally usable [80]. Also, nearly all 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 9

1Table 5: Statistics - Languages Wikidata (Extracted from dump [125]) 1 2 2 3Items Properties 3 4Number of languages 457 427 4 5(average, median) of # languages per element (labels + descriptions) 29.04, 6 21.24, 13 5 6(average., median) of # languages per element (labels) 5.59 , 4 21.18, 6 6 7(average, median) of # languages per element (descriptions) 26.10, 4 9.77, 6 7 8% elements without English labels 15.41% 0% 8 9% elements without English descriptions 26.23% 1.08% 9 10 10 11 11 12items have a description, see Figure 6d. Thus, this Weaknesses. However, this community-driven ap- 12 13short natural language phrase can be used for context proach does also introduce challenges. For example, 13 14similarity measures with the utterance. The inherent the list of labels of an item will not be exhaustive, as 14 15multilingual structure is intuitively useful for multilin- shown in Figures 6b and 6c. The graphs consider la- 15 16gual Entity Linking. Table 5 shows information about bels and aliases of all languages. While the average 16 17the use of different languages in Wikidata. As can number of labels/aliases is around 5, not all are use- 17 18be seen, are item labels/aliases available in up to 457 ful for Entity Linking in English. Ennio Morri- 18 19languages. Of course, not all items have labels in all cone does not have an alias solely consisting of En- 19 20languages. On average, labels/aliases/descriptions are nio while he will certainly sometimes be referenced 20 21available in 29.04 different languages. However, the by that. Thus, one can not rely on the exact labels 21 22median is only 6 languages. Many entities will there- alone. But interestingly, Wikidata has properties for 22 23fore certainly not have information in many languages. the fore- and surname alone, just not as a label or alias. 23 24The most dominant language is English but not all el- A close examination of what information to use is es- 24 25ements have label/alias/description information in En- sential. However, this is also a problem in other KGs. 25 26glish. For less dominant languages, this is of course Also, Wikidata often has items with very long, noisy, 26 27more severe. German labels exist for example only for error-prone labels, which can be a challenge to link 27 2814 %, and Samoan labels for 0.3 %. Context informa- to [80]. Nearly 20 percent of labels have a length larger 28 29tion in the form of descriptions is also given in multiple than 100 letters, see Figure 7. Due to the community- 29 languages but many languages are again not covered 30driven approach, false statements, due to errors or van- 30 for each entity (as can be seen by a median of only 4). 31dalism [49], also occur. 31 While the multilingual label and description informa- 32 32 tion of items might be useful for language model based 33 33 variants, the same information for properties enables 34 34 multilingual language models. Because, on average, 100 35 35 21.18 different languages are available per property for h

36t 80 36 g

labels, one could train multilingual models on the con- n 37 37 e l catenations of the labels of triples to include context 60 h

38t 38 i

information. But of course, there are again many prop- w 39 39 s

l 40

erties with a lower number of languages, as the median e 40 40 b a

is also only 6 languages. Cross-lingual EL is therefore l

41f 20 41 o

certainly necessary to use language-model based EL in 42 42 multiple languages. % 430 43 0 44By using the qualifiers of hyper-relational statements 50 100 150 200 250 300 44 45more detailed information is available, useful not only Length 45 46for Entity Linking but also for other problems like 46 47Question Answering. The inclusion of hyper-relational Fig. 7. Percentiles of English label lengths (Extracted from 47 48statements is of course also more challenging. Novel dump [125]) 48 49graph embeddings have to be developed and utilized 49 50which can represent the structure of a claim enriched Another problem may be the lack of facts (here defined 50 51with qualifiers [39, 96]. as statements not being labels, descriptions, or aliases) 51 10 C. Möller et al. / Survey on English Entity Linking on Wikidata

1for some entities. According to Tanon et al. [108], in 5. Approaches 1 2March 2020, DBpedia had, on average, 26 facts per 2 3entity while Wikidata had only 12.5. This is still more 5.1. Overview 3 4than YAGO4 with 5.1. However, those entities with 4 5fewer facts are probably also not occurring in DBpe- Currently, the number of methods intended to work ex- 5 6dia, which has a much lower amount of entities [108]. plicitly on Wikidata is still relatively small, while the 6 7To tackle such long-tail entities, different approaches amount of the ones utilizing the structure of Wikidata 7 8is even smaller. 8 are necessary. The lack of descriptions can also be a 9problem. Currently, around 10% of all items do not There exist several KG-agnostic EL approaches [78, 9 10have a description, as shown in Figure 6d. However, 112, 136]. However, they were omitted as their fo- 10 11 11 the situation is increasingly improving. cus is being independent of the KG. Of course, they 12do use Wikidata information like labels as this infor- 12 13A general problem of Entity Linking is that a label or mation also exists in other KGs, but it is no explicit 13 14alias can reference multiple entities, see Table 6. While usage of Wikidata-specific characteristics. While the 14 15around 70 million mentions point each to an unique approach by Zhou et al. [135] does utilize Wikidata 15 16item, 2.9 million do not. Not all of those are entities aliases in the candidate generation process, the target 16 17by our definition but, e.g., also classes or topics. Also, KB is Wikipedia and was therefore also excluded. 17 18longer labels or aliases often correspond to non-entity 18 19Tools without accompanying publications are not con- 19 items. Thus, the percentage of entities with overlap- 20sidered due to the lack of information about the ap- 20 ping labels/aliases is certainly larger than for all items. 21proach and its performance. Hence, for instance, the 21 To use Wikidata as a Knowledge Graph, one needs 22Entity Linker in the DeepPavlov [16] framework is not 22 to be cautious of the items one will include as enti- 23included, though it targets Wikidata and appears to use 23 Wikimedia disam- 24ties. For example, there exist label and description information successfully to link 24 25biguation page items which often have the same entities. 25 26label as an entity in the classic sense. Both, Q76 vs 26 Q61909968 have Barack Obama as the label. In- We distinguish three different kind of approaches: 27(1) Rule-based approaches, (2) approaches employing 27 cluding those will make disambiguation more difficult. 28statistical methods and (3) neural network-based ap- 28 Also, the possibility of contradictory facts will make 29proaches. The vast amount of methods are using neu- 29 30EL over Wikidata harder. 30 ral networks to solve the EL task [6, 13, 14, 18, 54, 61, 31 31 In Wikification, also known as EL on Wikipedia, large 67, 80, 81, 85, 88, 89, 104]. Some of those approaches 32solve the ER and EL jointly as an end-to-end task. Be- 32 33text documents for each entity exist in the knowledge 33 base, enabling text-heavy methods [127]. Such large sides those, there exists one purely rule based approach 34[98] and two based on statistical methods [23, 70]. 34 35textual contexts (besides the descriptions and the labels 35 36of triples itself) do not exist in Wikidata requiring other The approaches mentioned above solve the EL prob- 36 37methods or the inclusion of Wikipedia. However, as lem as specified in Section 3. That is, other EL meth- 37 38Wikidata is closely related to Wikipedia, an inclusion ods with a different problem definition also exist. For 38 39is easily doable. example, Almeida et al. [4] try to link street names to 39 40entities in Wikidata by using additional location infor- 40 41One can conclude that characteristics of Wikidata, like mation and limiting the entities only to locations. As it 41 42being up to date, multilingual and hyper-relational, in- uses additional information about the true entity via the 42 43troduce new possibilities while the existence of long- location, it is less comparable to the other approaches. 43 44tail entities, noise or contradictory facts is also chal- Thawani et al. [110] link entities only over columns of 44 45lenging. Thus, RQ 2 is answered. tables. It is not comparable since it does not use natural 45 46 46 Table 6: Number of English labels/aliases pointing to a certain number of items in Wikidata (Extracted from 47 47 dump [125]) 48 48 49 49 # Labels/aliases 70,124,438 2,041,651 828,471 89,210 3329 50 50 # Items per label/alias 1 2 3 − 10 11 − 100 < 100 51 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 11

1language utterances. The approach by Klie et al. [63] sents if the graph embedding is consistent with the text 1 2is concerned with Human-In-The-Loop EL. While its embedding. One crucial problem is that those meth- 2 3target KB is Wikidata, the focus on the inclusion of ods only work for a single entity in the text. Thus, it 3 4a human in EL process makes it incomparable to the has to be applied multiple times, and there will be no 4 5other approaches. EL methods working on other lan- information exchange between the entities. While the 5 6guages than English [28, 30, 31, 60, 114] were not con- examined algorithms do utilize the underlying graph 6 7sidered but also did not use any novel characteristics of of Wikidata, the hyper-relational structure is not taken 7 8Wikidata. In connection to the CLEF HIPE 2020 chal- into account. The paper is more concerned with com- 8 9lenge [28], multiple Entity Linkers working on Wiki- paring how basic neural networks work on the triples 9 10data were built. While short descriptions of the ap- of Wikidata. Due to the pure analytical nature of the 10 11proaches are available in the challenge-accompanying paper the usefulness of the designed approaches to a 11 12paper, only approaches described in an own published real-world setting is limited. The reliance on graph em- 12 13paper were included in this survey. The approach by beddings make it susceptible to change in the Wikidata 13 14Kristanti and Romary [65] was not included as it used KG. 14 15pre-existing tools for EL over Wikidata for which no 15 16sufficient documentation was available. Deeptype [89] is a novel approach using the type infor- 16 17mation of Wikidata or Wikipedia. Developed in 2018, 17 Due to the limited number of methods, we also eval- 18first, a type system was optimized via stochastic op- 18 uated methods that are not solely using Wikidata but 19timization. A type system is a grouping of multiple 19 also additional information from a separate KG or 20type axes where a type axis is a set of mutually ex- 20 Wikipedia. This is mentioned accordingly. Approaches 21clusive types. The idea is to classify entities according 21 linking to knowledge graphs different from Wiki- 22to the different type axes. Various methods to gener- 22 data, but for which a mapping between the knowledge 23ate the type system were compared, such as a genetic 23 graphs and Wikidata exists, are also not included. Such 24algorithm. The objective was a type system which im- 24 methods would not use the Wikidata characteristics at 25proves the EL performance while also being learnable. 25 all and their performance depends only on the quality 26The learnability is important to guarantee that a clas- 26 of the other KG and the mapping. 27sifier can be trained for the type system. After opti- 27 28In the following, the different approaches are described mization, it consists of 128 different types. The authors 28 29and examined according to the characteristics of Wiki- do not mention how the candidates are generated. It is 29 30data used. For an overview, see Table 7. only stated that commonly it is done via a dictionary, 30 31therefore, one can only assume that they used a dictio- 31 5.1.1. Entity Linking 32nary. Then the words in an utterance are classified via a 32 In the following, we will first focus on methods only 33windowed Bi-LSTM according to the type system. The 33 doing EL. 34type probabilities are then used together with a link 34 35In 2018, Cetoli et al. [18] evaluated how different types probability score to get the final score per candidate. 35 36of basic neural networks perform solely over Wiki- This link probability a statistic on how often a men- 36 37data. Notably, they compared the different ways to en- tion is linked to an article of an entity in Wikipedia. 37 38code the graph context via neural methods, especially The approach is multilingual as its learned type sys- 38 39the usefulness of including topological information via tem is agnostic to language. Thus it can be easily used 39 40GNNs [105, 129] and RNNs [51]. However, there is no with entity mentions in different languages. It is im- 40 41candidate generation as it was assumed that the can- portant to note that they used Wikipedia categories to 41 42didates are available. The process consists of combin- train their type system and Wikipedia articles to train 42 43ing text and graph embeddings. The text embedding the type classifier. However, the authors claim that the 43 44is calculated by applying a Bi-LSTM over the Glove algorithm is easily changeable to Wikidata. Neverthe- 44 45Embeddings of all words in an utterance. The result- less, as it is also possible to adapt other algorithms, 45 46ing hidden states are then masked by the position of initially created for different KGs, to Wikidata, this 46 47the entity mention in the text and averaged. A graph method may not be suitable to be compared to the other 47 48embedding is calculated in parallel via different meth- algorithms. Assuming it could be used over Wikidata 48 49ods utilizing GNNs or RNNs. The end score is the out- types, it seems to produce quite good results while 49 50put of one feed-forward layer having the concatenation only using a basic disambiguation algorithm besides 50 51of the graph and text embedding as its input. It repre- the type classifier. The results show that incorporating 51 12 C. Möller et al. / Survey on English Entity Linking on Wikidata

1Table 7: Comparison between the utilized Wikidata characteristics of each approach. 1 2 2 3Approach Labels/ Descrip- Knowledge Hyper- Types Additional 3 4Aliases tions graph relational Informa- 4 5structure structure tion 5 6OpenTapioca [23] 6 7NED using DL on 7 8Graphs [18] 8 9Falcon 2.0 [98] 3 9 10Arjun [80] 10 11DeepType [89] 1 1 Wikipedia 4 11 12Hedwig [61] Wikipedia 12 13VCG [104] 13 14KBPearl [70] 14 15PNEL [6] 15 16Mulang et al. [81] 2 16 17Perkins [85] 17 18Huang et al. [54] Wikipedia 18 19Boros et al. [13] Wikipedia, 19 20DBpedia 20 21Provatorov et al. [88] Wikipedia 21 22Labusch and Wikipedia 22 23Neudecker [67] 23 24Botha et al. [14] Wikipedia 24 25Tweeki [48] Wikipedia 25 26 26 1 3 27In paper, just demonstrated for Wikipedia Only querying the existence of triples 27 2 4 28Appears in the set of triples used for disambiguation Wikidata not used in implementation/evaluation 28 29 29 detailed type information improves EL considerably. models Roberta [73], XLNet [132] and the DCA-SL 30 30 As Wikidata contains many more types (≈2,400,000) model [130]. There is no global coherence technique 31 31 than other KGs, e.g., DBpedia (≈484,000) [108], it applied. Overall, up to 2-hop triples of any kind are 32 32 seems to be more suitable for this fine-grained type used. For example, labels, aliases, descriptions, or gen- 33 33 classificaton. Yet, not only the amount of types plays eral relations to other entities are all incorporated. It is 34 34 a role but also how many types are assigned per en- not mentioned if the hyper-relational structure in the 35 35 tity. In this regard, Wikipedia provides much more type form of qualifiers were used. On the one hand, the 36 36 information per entity than Wikidata [124]. A shift to purely language-based EL results in less need of re- 37 37 Wikidata is, therefore, not that simple. As Wikidata is training if the KG changes. On the other hand, the re- 38 38 growing every minute, it may also be challenging to liance on the triple information might be problematic 39 39 keep the type system up to date. for long-tail entities. 40 40 41The approach by Mulang et al. [81] is tackling the EL The master thesis by Perkins [85] is performing can- 41 42problem with Transformer [116] models. It is assumed didate generation by using anchor link probability 42 43that the candidate entities are given. For each entity, the over Wikipedia and LSH over labels and mention bi- 43 44labels of 1-hop and 2-hop triples are extracted. Those grams. Contextual word embeddings of the utterance 44 45are then concatenated together with the utterance and (ELMo [86]) are used together with KG embeddings 45 46the entity mention. The concatenation is the input of a (TransE [11]), calculated over Wikipedia and Wiki- 46 47pre-trained Transformer model. With a fully connected data, respectively. The context embeddings are sent 47 48layer on top, it is then optimized according to a bi- through a recurrent neural network. The output is con- 48 49nary cross-entropy loss. This architecture results in a catenated with the KG embedding and then fed into a 49 50similarity measure between the entity and the entity feed-forward neural network giving a similarity mea- 50 51mention. The examined models are the Transformer sure between the KG embedding of the entity candi- 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 13

1date and the utterance. The KG is used in the form KG for multilingual EL as its entities themselves are 1 2of the calculated TransE embeddings. Hyper-relational language-agnostic. The approach was tested on zero- 2 3structures like qualifiers are not mentioned in the the- and few-shot settings showing that the model can han- 3 4sis and not considered by the TransE embedding algo- dle an evolving knowledge base with newly added enti- 4 5rithm. Thus, probably not included. The used KG em- ties that were never seen before. This is also more eas- 5 6beddings make it necessary to retrain when the Wiki- ily achievable due to its missing reliance on the graph 6 7data KG changes as they are not dynamic. structure of Wikidata or the structure of Wikipedia. 7 8It is the case that some Wikidata entities do not ap- 8 The approach designed by Botha et al. [14] tackles 9pear in Wikipedia and are therefore invisible to the ap- 9 multilingual EL. It is also crosslingual. That means, 10proach. But this is less problematic here than for other 10 it can link entity mentions to entities in a knowledge 11approaches. The model is trained over descriptions of 11 graph in a language different to the utterance one. 12entities in multiple languages. Other approaches only 12 The idea is to train one model to link entities in ut- 13use the English Wikipedia, which misses entities avail- 13 terances of 100+ different languages to a KG con- 14able in other languages. Thus, the amount of available 14 taining not necessarily textual information in the lan- 15entities is larger. 15 16guage of the utterance. While the target KG is Wiki- 16 17data, they mainly use Wikipedia descriptions as in- 5.1.2. Entity Recognition and Entity Linking 17 18put. This is the case as extensive textual information The following methods all include ER in their EL pro- 18 19is not available in Wikidata. But as Wikipedia arti- cess. 19 20cles are easily linkable to the corresponding Wiki- 20 In 2018, Sorokin and Gurevych [104] were doing joint 21data entities, gathering the desired textual informa- 21 end-to-end ER and EL on short texts. The algorithm 22tion is easy. Furthermore, as the Wikidata entities have 22 tries to incorporate multiple context embeddings into 23language-agnostic identifiers, Wikidata is suited to be 23 a mention score, signaling if a word is a mention, 24the target KG. The approach resembles the Wikifica- 24 and a ranking score, signaling the candidate’s correct- 25tion method by Wu et al. [127] but extends the training 25 ness. First, it generates several different tokenizations 26process to be multilingual and targets Wikidata. Can- 26 of the same utterance. For each token, a search is con- 27didate generation is done via a dual-encoder architec- 27 28ture. Here, two BERT-based Transformer models en- ducted over all labels in the KG to gather candidate 28 29code both the context-sensitive mentions and the en- entities. If the token is a substring of a label, the en- 29 30tities to the same vector space. The mentions are entity is added. Each token sequence gets then a score as- 30 31coded using local context, the mention and surround- signed. The scoring is tackled from two sides. On the 31 32ing words, and global context, the document title. Enti- utterance side, a token-level context embedding and a 32 33ties are encoded by using the Wikipedia article descrip- character-level context embedding (based on the men- 33 34tion available in different languages. In both cases, the tion) is computed. The calculation is handled via di- 34 35encoded CLS-token are projected to the desired en- lated convolutional networks (DCNN) [134]. On the 35 36coding dimension. The goal is to embed mentions and KG side, one includes the labels of candidate entity, 36 37entities in such a way that the embeddings are simi- the labels of relations connected to a candidate entity, 37 38lar. The model is trained over Wikipedia by using the the embedding of the candidate entity itself, and em- 38 39anchors in the text as entity mentions. Now, after the beddings of the entities and relations related to the can- 39 40model is trained, all entities are embedded. The candidate entity. This is again done by DCNNs and, addi- 40 41didates are generated by embedding the mention and tionally, by fully connected layers. The best solution is 41 42searching for the nearest neighbors. A certain num- then found by calculating a ranking and mention score 42 43ber of neighbors are then the generated candidates. A for each token for each possible tokenization of the 43 44cross-encoder is employed to rank the entity candi- utterance. All those scores are then summed up into 44 45dates, fed with the concatenation of the entity descrip- a global score. The global assignment with the high- 45 46tion and mention text. Final scores are obtained and the est score is then used to select the entity mentions and 46 47entity mention is linked. Wikidata information is only entity candidates. The approach uses the underlying 47 48used to gather all the Wikipedia descriptions in the dif- graph, label and alias information of Wikidata. Graph 48 49ferent languages for all entities. Besides that, one re- information is used via connected entities and rela- 49 50lies mainly on Wikipedia. While that is the case, it is tions. They also use TransE embeddings, and there- 50 51also clear that Wikidata is very suitable as the target fore no hyper-relational structure. Due to the usage of 51 14 C. Möller et al. / Survey on English Entity Linking on Wikidata

1static graph embeddings, retraining will be necessary as well as the triples itself. The relations and entities 1 2if Wikidata changes. are recognized by applying linguistic principles. The 2 3candidates are then generated by comparing mentions 3 OpenTapioca [23] is a mainly statistical EL approach 4to the labels using the Levenshtein distance. The rank- 4 published in 2019. While the paper never mentions 5ing of the entities and relations is done by creating 5 ER, the approach was evaluated with it. In the code 6triples between the relations and entities and check- 6 one can see that the ER is done by a SolrTextTagger 7ing if the query is successful. The more successful the 7 analyzer of the Solr search platform3. The candidates 8queries, the higher the candidate will be ranked. If no 8 are generated by looking up if the mention corresponds 9query is successful, the algorithm returns to the ER 9 to an entity label or alias in Wikidata stored in a Solr 10phase and splits some of the recognized entities again. 10 11collection. Entities are filtered out which do not cor- As Falcon 2.0 is an extension of Falcon 1.0 from DB- 11 12respond to the type person, location or organization. pedia to Wikidata, the usage of specific Wikidata char- 12 13OpenTapioca is based on two main features, which are acteristics is limited. Falcon 2.0 is tuned for EL on 13 14local compatibility and semantic similarity. First, lo- questions and short texts, as well as the English lan- 14 15cal compatibility is calculated via a popularity mea- guage. It is thus not very generalizable on longer, more 15 16sure and a unigram similarity measure between entity noisy, non-question texts. As it only based on rules it 16 17label and mention. The popularity measure is based is clearly independent of changes in the KG. 17 18on the number of sitelinks, PageRank scores, and the 18 Arjun [80] tries to tackle specific challenges of Wiki- 19number of statements. Second, the semantic similar- 19 data like long entity labels and implicit entities. Pub- 20ity strives to include context information in the deci- 20 lished in 2020, Arjun is an end-to-end approach utiliz- 21sion process. All entity candidates are included in a 21 ing the same model for ER and EL. It is based on an 22graph and are connected via weighted edges. Those 22 Encoder-Decoder-Attention model. First, the entities 23weights are calculated via a statistical similarity mea- 23 are detected via feeding Glove [84] embedded tokens 24sure. This measure includes how likely it is to jump 24 of the utterance into the model and classifying each to- 25from one entity candidate to another while discount- 25 ken as being an entity or not. Afterward, candidates are 26ing it by the distance between the corresponding men- 26 generated in the same way as in Falcon 2.0 [98]. The 27tions in the utterance. The resulting adjacency matrix 27 candidates are then ranked by feeding the mention, the 28is then normalized to a stochastic matrix that defines a 28 entity label, and its aliases into the model and calculat- 29Markov Chain. One now propagates the local compat- 29 ing the score. Thus, the model is a similarity measure 30ibility using this Markov Chain. Several iterations are 30 between the mention and the entity labels. It does not 31then taken, and a final score is inferred via a Support 31 use any global ranking. Wikidata information is used 32Vector Machine. It supports multiple entities per utter- 32 in the form of labels and aliases in the candidate gen- 33ance. OpenTapioca is only trained on and evaluated for 33 eration and candidate ranking. As it relies purely on 34three types of entities: locations, persons, and organi- 34 labels, it is not that susceptible to changes in the KG. 35zations. It facilitates Wikidata-specific labels, aliases, 35 36and sitelinks information. More importantly, it also Hedwig [61] is a multilingual entity linker specialized 36 37uses qualifiers of statements in the calculation of the on the TAC 2017 task but published in 2020. Another 37 38PageRank scores. But the qualifiers are only seen as entity linker [62], developed by the same authors, is 38 39additional edges to the entity. The usage in special do- not included in this survey as Hedwig is partly an evo- 39 40mains is limited due to its restriction to only three types lution of it. The entities to be linked are limited to only 40 41of entities but this is just an artificial restriction. It is a subset of all possible entity classes. Hedwig employs 41 42easily updatable if the Wikidata graph changes as no Wikidata and Wikipedia at the same time. The Entity 42 43immediate retraining is necessary. Recgontion uses word embeddings, character embed- 43 44dings, and dictionary features where the character em- 44 Falcon 2.0 [98] is a fully linguistic approach and a 45beddings are calculated via a Bi-LSTM. The dictionary 45 transformation of Falcon 1.0 [97] to Wikidata. Falcon 46features are class-dependent, but this is not defined in 46 2.0 was published in 2019 and its focus lies on short 47more detail. Those embeddings and features are com- 47 texts, especially questions. It links entities and rela- 48puted and concatenated for each token. Afterward, the 48 tions jointly. Falcon 2.0 uses entity and relation labels 49whole sequence of token features is fed into a Bi- 49 50LSTM with a linear chain Conditional Random Field 50 513https://lucene.apache.org/solr/ (CRF) layer at the end to recognize the entities. The 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 15

1candidates for each detected entity mention are then herence can perform sub-optimally since not all enti- 1 2generated by using a mention dictionary. The dictio- ties/relations in a document are related. Thus, two vari- 2 3nary is created from Wikidata and Wikipedia informa- ants of the algorithm are proposed. First, a pipeline 3 4tion, utilizing labels, aliases, titles or anchor texts. The version that separates the full document into sentences. 4 5candidates are disambiguated by constructing a graph Second, a near neighbor mode, limiting the interaction 5 6consisting of all candidate entities, mentions, and oc- of the nodes in the graph by the distances of the corre- 6 7curring words in the utterance. The edges between en- sponding noun-phrases and relation-phrases. The ap- 7 8tities and other entities, words, or mentions have the proach includes label and alias information of entities 8 9normalized pointwise mutual information (NPMI) as- and predicates. Additionally, one-hop statement infor- 9 10signed as their weights. The NPMI specifies how fre- mation is used, but hyper-relational features are not 10 11quent two entities, an entity and a mention or an en- mentioned. However, the paper does not claim that its 11 12tity and a word, occur together. Those scores are cal- focus is entirely on Wikidata. Thus, the weak special- 12 13culated over a Wikipedia dump. Finally, the PageR- ization is understandable. While it utilizes EL, the fo- 13 14ank of each node in the graph is calculated via power cus of the approach is still knowledge base population. 14 15iteration, and the highest-scoring candidates are cho- No training is necessary which makes the approach 15 16sen. In contrast to DeepType, the type classification is suitable for a dynamic graph like Wikidata. 16 17 17 used to determine the types of entities, not mentions. PNEL [6] is an E2E model jointly solving ER and EL 18 18 As this is only relevant for the TAC2017 task, the clas- focused on short texts. PNEL employs a Pointer net- 19 19 sifier can be ignored. Labels and aliases of multiple work [118] working on a set of different features. An 20 20 languages are used. It also uses sitelinks to connect the utterance is tokenized into multiple different combina- 21 21 Wikidata identifiers and Wikipedia articles. The paper tions. Each token is extended into the (1) token itself, 22 22 also claims to use descriptions but does not describe (2) the token and the predecessor, (3) the token and 23 23 anywhere in what way. No hyper-relational or graph the successor, and (4) the token with both predecessor 24 24 features are used. As it employs class-dependent fea- and successor. For each token combination, candidates 25 25 tures, it is limited to the entities of classes specified in are searched for by using the BM25 similarity mea- 26 26 the TAC 2017 task. The NPMI weights have to be up- sure. Fifty candidates are used per tokenization combi- 27 27 dated with the addition of new elements in Wikidata nation. Therefore, 200 candidates are found per token. 28 28 and Wikipedia. For each candidate, features are extracted. Those range 29 29 from the simple length of a token to the graph embed- 30KBPearl [70], published in 2020, utilizes EL to pop- 30 dings of the candidate entity. All features are concate- 31ulate incomplete KGs using documents. First, a docu- 31 nated to a large feature vector. Therefore, per token, 32ment is preprocessed via Tokenization, POS tagging, 32 a sequence of 200 such features vectors exist. Finally, 33NER, noun-phrase chunking, and time tagging. Also, 33 the concatenation of those sequences of each token in 34an existing Information Extraction tool is used to ex- 34 the sentence is then fed into a Pointer network. At each 35tract open triples from the document. Open triples are 35 iteration of the Pointer network, it points to one can- 36non-linked triples in unstructured text. The triples are 36 didate in the network or an END token marking no 37processed further by filtering invalid tokens and doing 37 choice. The entity descriptions, labels and aliases are 38canonicalization. Then, a graph of entities, predicates, 38 all used. Additionally, the graph structure is included 39noun phrases, and relation phrases is constructed. The 39 by TransE graph embeddings, but no hyper-relational 40candidates are generated by comparing the noun/re- 40 information was incorporated. E2E models often can 41lation phrases to the labels and aliases of the enti- 41 improve the performance of the ER. Most EL algo- 42ties/predicates. The edges between the entities/rela- 42 rithms employed in industry often use older ER meth- 43tions and between entities and relations are weighted 43 ods decoupled from the EL process. Thus, such an E2E 44by the number of intersecting one-hop statements. The 44 EL approach can be of use. Nevertheless, due to its re- 45next step is the computation of a maximum dense sub- 45 liance on static graph embeddings, complete retraining 46graph. Density is defined by the minimum weighted 46 will be necessary if Wikidata changes. 47degree of all nodes [53]. As this problem is NP-hard, 47 48a greedy algorithm is used for optimization. New enti- The approach designed by Huang et al. [54] is uti- 48 49ties relevant for the task of Knowledge Graph Popula- lizing deep and shallow models together. It special- 49 50tion are identified by thresholding the weighted sum of ized in short texts. The ER is performed via a pre- 50 51an entity’s incident edges. Like used here, global co- trained BERT model [25] with a single classification 51 16 C. Möller et al. / Survey on English Entity Linking on Wikidata

1layer on top, determining if a token belongs to an en- EL, a knowledge base is built from Wikipedia, contain- 1 2tity mention. The candidate search is done via an Elas- ing Wikipedia titles, ids, disambiguation pages, redi- 2 3ticSearch4 index, comparing the entity mention to la- rects and calculating link probability between men- 3 4bels and aliases by exact match and Levenshtein dis- tions and Wikipedia pages. The link probability be- 4 5tance. The candidate ranking uses three similarity mea- tween anchors and Wikipedia pages is used to gather 5 6sures to calculate the final rank. A CNN is used to entity candidates for a mention. The disambiguation 6 7compute a character-based similarity between entity approach follows an already existing method [64]. 7 8mention and candidate label. This results in a similar- Here, the utterance tokens are embedded via a Bi- 8 9ity matrix whose entries are calculated by the cosine LSTM. The token embeddings of a single mention are 9 10similarity between each character embedding of both combined. Then similarity scores between the result- 10 11strings. The context is included in two ways. First, being mention embedding and the entity embeddings of 11 12tween the utterance and the entity description, by em- the candidates are calculated. The entity embeddings 12 13bedding the tokens of each sequence through a BERT are computed according to Ganea and Hofmann [40]. 13 14model. Again, a similarity matrix is built by calculat- These similarity scores are combined with the link 14 15ing the cosine similarity between each token embed- probability and long-range context attention, calcu- 15 16ding of both utterance and description. The KG is also lated by taking the inner product between an addi- 16 17considered by including the triples containing the can- tional context-sensitive mention embedding and an en- 17 18didate as a subject. For each such a triple a similarity tity candidate embedding. The resulting score is a local 18 19matrix is calculated between the label concatenation ranking measure and is again combined with a global 19 20of the triple and the utterance. All measures are then ranking measure considering all other entity mentions 20 21combined and fed into a two-layer perceptron. Wiki- in the text. In the end, additional filtering is applied 21 22data labels, aliases and descriptions are utilized. Addi- by comparing the DBpedia types of the entities to the 22 23tionally, the KG structure is incorporated through the ones classified during the ER. If the type does not 23 24labels of candidate-related triples. This is similar to the match or other inconsistencies apply, the entity candi- 24 25approach by Mulang et al. [81], but only 1-hop triples date gets a lower rank. Here, they also experimented 25 26are used. There are also no hyper-relational informa- with Wikidata types, but this resulted in a performance 26 27tion considered. Due to its reliance on text alone, it is decrease. As can be seen, technically, no Wikidata in- 27 28less susceptible to the changes of Wikidata. formation besides the unsuccessful type inclusion is 28 29used. Thus, the approach resembles more of a Wikifi- 29 CLEF 2020 HIPE challenge 30In connection to the [28], cation algorithm. Yet, they do link to Wikidata as the 30 31multiple approaches for ER and EL of historical news- HIPE task dictates it and therefore, the approach was 31 32papers on Wikidata were developed. Documents were included in the survey. New Wikipedia entity embed- 32 33available in English, French and German. Three ap- dings can be easily added [40] which is an advantage 33 34proaches with a focus on the English language are de- when Wikipedia changes. Also, its robustness against 34 35scribed in the following. The documents are noisy as erroneous texts makes it ideal for real-world use. 35 36the OCR method for transcribing the newspapers pro- 36 37duced errors. The authors often constructed different Labusch and Neudecker [67] also applied a BERT 37 38methods for different languages. From now on, only model for ER. For EL, they used mostly Wikipedia, 38 39the English models are described. Differences in the similar to Boros et al. [13]. They built a knowledge 39 40usage of Wikidata between the languages did not exist. base containing all person, location and organization 40 41Yet, the approaches were not multilingual as different entities from the German Wikipedia. Then it was con- 41 42models were used and/or a retraining was necessary verted to an English knowledge base by mapping from 42 43for different languages. the German Wikipedia Pages via Wikidata to the En- 43 44glish ones. This mapping process resulted in the loss 44 Boros et al. [13] tackled ER by using a BERT model 45of numerous entities. The candidate generation is done 45 with a CRF layer on top, which recognizes the en- 46by embedding all Wikipedia page titles in an Approx- 46 tity mentions and classifies the type. During the train- 47imative Nearest Neighbour index. Using this index, 47 ing, the regular sentences are enriched with misspelled 48the neighboring entities to the mention embedding are 48 words to make the model robust against noise. For the 49found and used as candidates. For ranking, anchor- 49 50contexts of Wikipedia pages are embedded and fed 50 514https://www.elastic.co/elasticsearch/ into a classifier together with the embedded mention- 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 17

1context, which outputs whether both belong to the and then searching for the aliases in a dictionary. The 1 2same entity. This is done for each candidate for around ranking is done using the link probabilities while prun- 2 350 different anchor-contexts. Then, multiple statistics ing all candidates that do not belong to the type pro- 3 4on those similarity scores and candidates are calcu- vided by the Entity Recognizer. It is a relatively sim- 4 5lated, which are used in a Random Forest model to ple approach that does not need to be trained, making 5 6compute the final ranks. Similar to the previous ap- it very suitable for linking entities in tweets. In that 6 7proach, Wikidata was only used as the target knowl- document type, often novel entities with minimal con- 7 8edge base, while information from Wikipedia was used text exist. Regarding features of Wikidata, it uses la- 8 9for all the EL work. Thus, no special characteristics bel, alias and type information. Due to it being unsu- 9 10of Wikidata were used. The approach is less affected pervised, changes to the KG do not affect it. 10 11by a change of Wikidata due to similar reasons as the 11 12previous approach. Also, this approach lacks perfor- 5.2. Evaluation 12 13mance compared to the state of the art in the HIPE task. 13 14The knowledge base creation process produces a dis- Table 8 and Table 9 give an overview of all available 14 15advantageous loss of entities, but this might be easily results for the approaches described in the previous 15 16changed. section. The first gives information for EL only ap- 16 17 17 Provatorov et al. [88] used an ensemble of fine-tuned proaches and the second for approaches evaluating EL 18 18 BERT models for ER. The ensemble is used to com- together with ER. The micro F1 scores are given: 19 19 pensate for the noise of the OCR procedure. The candi- 20p · r 20 dates were generated by using an ElasticSearch index F1 = 2 · 21p + r 21 filled with Wikidata labels. The candidate’s final rank 22 22 is calculated by taking the search score, increasing it 23where p is the precision p = tp and r is the recall 23 if a perfect match applies and finally taking the candi- tp+fp 24tp 24 date with the lowest Wikidata identifier number. They r = tp+fn . tp are here the amount of true positives, 25 25 also created three other methods of the EL approach: fp the amount of false positives and fn the amount of 26 26 (1) The ranking was done by calculating cosine simi- false negatives. Micro F1 means that the scores are cal- 27 27 larity between the embedding of the utterance and the culated over all linked entity mentions and not sepa- 28 28 embedding of the same utterance with the mention re- rately for each document and then averaged. True pos- 29 29 placed by the Wikidata description. Furthermore, the itives are the correctly linked entity mentions, false 30 30 score is increased by the Levenshtein distance between positives incorrectly linked entities which do not oc- 31 31 the entity label and the mention. (2) A variant was used cur in the set of valid entities and false negatives en- 32 32 where the candidate generation is enriched with his- tities which occur in the set of valid entities but are 33 33 torical spellings of Wikidata entities. (3) The last vari- not linked to [19]. The approaches were evaluated on 34 34 ant used an existing tool, which included contextual many different datasets, which makes comparison very 35 35 similarity and co-occurrence probabilities of mentions difficult. Additionally, many approaches are evaluated 36 36 and Wikipedia articles. Also, a global ranking was ap- on datasets designed for knowledge graphs different to 37 37 plied. The approach uses Wikidata labels and descrip- Wikidata and then mapped. Often, the approaches are 38 38 tions in one variant of candidate ranking. Beyond that, evaluated on the same dataset but over different sub- 39 39 no other characteristics specific to Wikidata were con- sets, which complicates a comparison even more. The 40 40 sidered. Overall, the approach is very basic and uses method by Perkins [85] was also evaluated on the Ken- 41 41 mostly pre-existing tools to solve the task. The ap- sho Derived Wikimedia Dataset [58], but it was only 42 42 proach is not susceptible to a change of Wikidata as used to compare different variants of the designed ap- 43 43 it is mainly based on language and does not need re- proach and focussed on different amounts of training 44 44 training. However, its poor performance in the HIPE data. Thus, inclusion in the evaluation table is not rea- 45 45 challenge makes it a less desirable method to employ. sonable. 46 46 47Tweeki [48] is an approach focusing on unsupervised Inferring the utility of a Wikidata characteristic from 47 48EL over tweets. The ER is done by a pre-existing En- the different approaches’ F1-measures is inconclusive 48 49tity Recognizer [41] which also tags the mentions. The due to the sparsity of results. For EL-only, AIDA- 49 50candidates are generated by first calculating the link CoNLL results are available for three of five ap- 50 51probability between Wikidata aliases over Wikipedia proaches, but the results for two are the accuracies in- 51 18 C. Möller et al. / Survey on English Entity Linking on Wikidata

1Table 8: Results: EL only. 1 2 2 LSH-ELMo model [85] NED using on DL 3Graphs [18] 3 Mulang etal. [81] Botha et al. [14] 4DeepType [89] 4 5 5 6 6 72 7 8 8 1 9 9 10 10 3 3,4 11AIDA-CoNLL [53] 0.949 [89] 0.9494 [81] 0.73 [85] - - 11 5 12ISTEX-1000 [23] - 0.9261 [81] --- 12 3 6 13Wikidata-Disamb [18] 0.924 [89] 0.9235 [81] - 0.916 [18] - 13 7 14Mewsli-9 [14] - - - - 0.91 [14] 14

151 4 7 15 Only evaluated on Wikipedia DCA-SL used Recall instead of F1 162 Model with best result 5 XLNet used 16 173 6 17 Accuracy instead of F1 Roberta used 18 18 19stead of the F1-measures. However, considering the information than just the labels of entities and rela- 19 20results of Deeptype [89] for Wikidata-Disamb, it be- tions, they mostly only include simple n-hop triple 20 21comes apparent that the inclusion of type information information. Hyper-relational information like quali- 21 22might help a lot. Still, it was only used with Wikipedia fiers is only used by OpenTapioca but still in a simple 22 23categories. The available labels for each item and prop- manner. This is surprising, as they can provide valu- 23 24erty make language-model-based approaches possible able additional information. As one can see in Fig- 24 25that perform quite well [81]. No approaches are avail- ure 8, around half of the statements on entities occur- 25 26able to compare to the one by Botha et al. [14], but ring in the LC-QuAD 2.0 dataset have one or more 26 27the result demonstrates the promising performance of qualifiers. These percentages differ from the ones in 27 28multilingual EL with Wikidata as the target KG. For all of Wikidata, but when entities are considered, ap- 28 29ER + EL approaches, most results were available for pearing in realistic use cases like QA, qualifiers are 29 30LC-QuAD 2.0. Yet, no conclusion can be drawn as much more abundant. Thus, dismissing the qualifier 30 31many approaches were evaluated on different subsets information might be critical. The inclusion of hyper- 31 32of the dataset. Falcon 2.0 performs well, but it does not relational graph embeddings could improve the perfor- 32 33substantially rely on Wikidata characteristics. The per- mance of many approaches already using non-hyper- 33 34formance is good as it is designed for simple questions relational ones. Rank information of statements might 34 35that follow its rules very closely. Arjun performs well be useful to consider, but choosing the best one will 35 36on T-REx by mainly using label information, but the probably often suffice. 36 37 37 amount of methods tested on the T-REx dataset is too Of all approaches, only two algorithms [6, 54] use de- 38 38 low to be conclusive. Besides that, PNEL and the ap- scriptions explicitly. Others incorporate them through 39 39 proach by Huang et al. also achieve good results; both triples too, but more on the side [81]. Descriptions can 40 40 include a broader scope of Wikidata information in the provide valuable context information and many items 41 41 form of labels, descriptions and graph structure. As do have them; see Figure 6d. Hedwig [61] claims to 42 42 HIPE challenge approaches are using Wikidata only use descriptions but fails to describe how. Three ap- 43 43 marginally and the difference in performance depends proaches [14, 61, 89] demonstrated the usefulness of 44 44 more on the robustness against the OCR-introduced the inherent multilingualism of Wikidata, notably in 45 45 noise, comparing them is not providing information on combination with Wikipedia. 46the relevance of Wikidata characteristics. 46 47As Wikidata is always changing, approaches robust 47 48While some algorithms [80] do try to examine the chal- against change are preferred. A reliance on transduc- 48 49lenges of Wikidata, like more noisy long entity la- tive graph embeddings [6, 18, 85, 104], which need 49 50bels, many fail to use most of the advantages of Wiki- to have all entities available during training, makes 50 51data’s structure. If the approaches are using even more repeated retraining necessary. Alternatively, the used 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 19

1Table 9: Results: ER + EL. 1 2 2 4 3 2 1 CQA . etstue nKPalpaper KBPearl in used set test 2.0 2.0 LC-QuAD LC-QuAD from questions sampled 1000 model L model NN rp-usin 17 .4 14 ------[104] 0.442 - - [107] Graph-Questions irpss21 11 .8 2] .4 4]------028[48] 0.248 ------[48] 0.148 [23], 0.087 [121] 2016 Microposts LFHP 00[9 .3 [28] 0.531 ------[29] 2020 HIPE CLEF 3 [12] Simple-Question 3 AD7WK 13 .7 7]------[70] 0.679 - - - - [113] QALD-7-WIKI nweg e 2]----034[0 ------[70] 0.384 - - - - [22] Net Knowledge CQA . [27] 2.0 LC-QuAD [27] 2.0 LC-QuAD IACNL[3 .8 2]------[23] 0.482 [53] AIDA-CoNLL CQA . 2]031[]045[]-04 6 .8 [6] 0.589 - [6] 0.47 - [6] 0.445 [6] 0.301 [27] 2.0 LC-QuAD Y21 6,7]----055[0 ------[70] 0.575 - - - - 70] [69, NYT2018 SE-00[3 .7[3 ------[23] 0.87 [23] ISTEX-1000 weiod[8 .9 4]------06 [48] 0.65 ------[48] 0.291 [48] TweekiGold ipeQeto .0[]04 6 .8[6] 0.68 - - - [6] 0.41 [6] 0.20 Simple-Question ecysi[4 .4[8 .7 [48] 0.371 ------[48] 0.14 [24] Derczynski eQP[3]--070[,14 .1 [6] 0.712 - - 104] [6, 0.730 - - [133] WebQSP eeb8[0 .5 7]------[70] 0.653 - - - - [70] ReVerb38 A21 5]------052[61] 0.582 ------[56] TAC2017

S-0 9]035[3 - - - - 4 ------[23] 0.335 [93] RSS-500 4 -E [34] T-REx -E 3]059[0 .1 8]------[80] 0.713 - [80] 0.579 [34] T-REx 5 5 6 6 7 7 7

84 3 8 6 9 9

10 10 .5[8 .8[8 - - - 11 ------[98] 0.68 [98] 0.25 [23] OpenTapioca 11

.2 7]------12 ------[70] 0.421 [6] - 0.629 ------[98] 0.63 [6] 0.320 - - - 12 13 13 14 14 15 15

16 [98] 2.0 Falcon 16 17 17 18 18

ru paper Arjun 7 6 5 19 [80] Arjun 19 vlaino usto -E aadfeett h ustue in used subset the to different data T-REx of subset on set Evaluation test and train on evaluated Probably model S 20 20 21 21 22 22

23 [104] VCG 23 24 24

25 25 Ber [70] KBPearl

26 26 271 27 28 28

29 [6] PNEL 29 30 30 5 8 2 2 31 31 .8 5]------[54] 0.780

32 [54] al. et Huang 32 ------33 33

34 34 35 [13] al. et Boros 35 9 8

36 matching mention Strict model W 36

379 37

.0 [28] 0.300 38 [88] al. et Provatorov 38 39 39 40 40 9

41 41 .4 [28] 0.141 edce [67] Neudecker 42 & Labusch 42 43 43

449 44 45 [61] Hedwig 45

46 - - 46 47 47

48 48 wei[48] Tweeki 49 49 50 50 51 51 20 C. Möller et al. / Survey on English Entity Linking on Wikidata

1This summary and evaluation of the existing Wikidata 1 2Entity Linkers answers RQ 1. 2 3 3 45.3. Reproducibility 4 5 5 6Not all algorithms are available as an Web API or 6 7even as source code. An overview can be seen in Ta- 7 8ble 10. The amount of approaches for Wikidata having 8 9 9 10Table 10: Availability of approaches. 10 11 11 12Approach Code Web API 12 13 13 OpenTapioca [23] 14 14 NED using DL on 15 15 Fig. 8. Percentage of statements having the specified number of qual- Graphs [18] 16 16 ifiers for all LC-QuAD 2.0 and Wikidata entities. Falcon 2.0 [98] 17 17 Arjun [80] 18 18 embeddings would need to be replaced with graph DeepType [89] 19 19 embeddings, which are efficiently updatable or induc- Hedwig [61] 20 20 tive [3, 5, 47, 109, 122, 123, 128]. The rule-based VCG [104] 21 21 approach Falcon 2.0 [98] is not affected by a devel- KBPearl [70] 22 22 oping knowledge base but only usable for correctly- PNEL [6] 23 23 stated questions. Methods only working on text in- Mulang et al. [81] 24 24 formation [54, 80, 81] like labels, descriptions or Perkins [85] 25 25 aliases do not need to be updated if Wikidata changes, Huang et al. [54] 26 26 only if the text type or the language itself does. For Boros et al. [13] 27 27 approaches [48, 54, 61] that rely on statistics over Provatorov et al. [88] 28Wikipedia, new entities may in Wikidata may some- 28 Labusch and Neudecker [67] 29times not exist in Wikipedia to a satisfying degree. 29 Botha et al. [14] 30The approaches by Boros et al. [13], and Labusch and 30 Tweeki [48] 31Neudecker [67] are mostly using Wikipedia informa- 31 32tion. They are, therefore, susceptible to changes in 32 33Wikipedia, especially specific statistics calculated over an accessible Web API is meager. While the code for 33 34Wikipedia pages. Botha et al. [14] also mainly depends some methods exists, this is still just the case for less 34 35on Wikipedia and thus on the availability of the de- than half. The effort to set up the different approaches 35 36sired Wikidata entities in Wikipedia itself. But as it also varies significantly due to missing instructions or 36 37uses Wikipedia articles in multiple languages, it en- data. Thus, we refrained from evaluating and filling the 37 38compasses many more entities than the previous ap- missing results for all the datasets in Tables 8 and 9. 38 39proaches that focus on Wikipedia. As it was designed 39 40for the zero- and few-shot setting, it is quite robust 40 41against changes in the underlying knowledge base. 6. Datasets 41 42Deeptype [89] relies on a fine-grained type system. 42 43As the categories of Wikidata are not evolving as fast 6.1. Overview 43 44as novel entities appear, it is relatively robust against 44 45a changing knowledge base. However, it was not yet This section is concerned with analyzing the differ- 45 46tested on Wikidata, which’s type assignments differs ent datasets which are used for Wikidata EL. A com- 46 47vastly from Wikipedia. Statistical approaches [23, 70] parison can be found in Table 11. Here, information 47 48need to update the underlying statistics, but this might about the purpose, release year, domain and more is 48 49be efficiently doable. Overall, the robustness against given. The majority of datasets on which existing En- 49 50change is most negatively affected by static/transduc- tity linkers were evaluated, were originally constructed 50 51tive graph embeddings. for KGs different from Wikidata. Such a mapping 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 21

1Table 11: Comparison of used datasets. 1 2 2 1 aafo 2010 from data weiod[8 wes22 EL EL 2020 EL 2020 3 EL EL ER, Tweets lan- Tweets multiple Annotated in News 2019 2018 2020 Pop- Base Knowledge newspapers Historical [48] TweekiGold [48] Purpose TweekiData [14] Mewsli-9 [29] News 2020 HIPE CLEF EL News Wikimedia [58] Dataset Derived Kensho bi- 2015 abstracts, Wikipedia [82] KORE50DYWC abstracts Wikipedia Year 70] [69, NYT2018 [22] Net Knowledge 2019 ques- complex General Domain articles Research [34] T-REx [27] 2.0 LC-QuAD Wiki-Disamb30[38]) (based on [18] Wikidata-Disamb [23] ISTEX-1000 Dataset 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 iiei 00NtrlLanguage Natural 2020 guages Wikipedia texts ographical 2018 tions articles Wikipedia 14 14 15 15 16 16 17 17 18 18 19 19 20 20 21 21 00Mliiga EL Multilingual 2020 KBP Answering Question 2019 2019 22 22 23 23 1 24 24 25 25 26 26 27 27 2

28 Wikipedia on dataset Original 28 29 29 30 30

31 (NLP) Processing (NLG) neation Ge- Language Natural (RE), Extraction tion Rela- (KBP), ulation (QA) EL 31 32 32 33 33 34 34 35 35 36 36 37 37 38 38

39mentions 39 40 40 41 41 42 42

43Wikidata Wikidata Wikidata Wikidata Wikipedia Wikidata, DBpedia, Crunchbase YAGO, Wikidata, DBpedia Wikidata, Wikidata Wikidata Wikidata DBpedia, Wikidata Wikidata Identiﬁers 43 44 44 45 45

462 46 47 47 48 48 49 49 50 50 51 51 22 C. Möller et al. / Survey on English Entity Linking on Wikidata

1can be problematic as some entities labeled for other This dataset is of great difficulty due to many errors in 1 2KGs could be missing in Wikidata. Or some NIL en- the text, which originates from the OCR method used 2 3tities that do not exist in other KGs could exist in to parse the scanned newspapers. For the English lan- 3 4Wikidata. Eleven datasets were found for which Wiki- guage, only a development and test set exist. In the 4 5data [14, 22, 23, 27, 29, 34, 48, 58, 70, 82] identifiers other two languages, a training set is also available. It 5 6were available from the start. was manually annotated. 6 7 7 LC-QuAD 2.0 [27] is a dataset semi-automatically Mewsli-9 [14] is a multilingual dataset automatically 8 8 created for Complex Questions Answering providing constructed from WikiNews. It includes nine different 9 9 complex natural language questions. For each ques- languages. A high percentage of entity mentions in the 10 10 tion, Wikidata and DBpedia identifiers are provided. dataset do not have corresponding English Wikipedia 11 11 The questions are generated from subgraphs of the pages, and thus, cross-lingual linking is necessary. 12 12 Wikidata KG. The dataset does not provide annotated 13TweekiData and TweekiGold [48] are an automatically 13 mentions. 14annotated corpus and a manually annotated dataset 14 15T-REx [34] was constructed automatically over Wiki- for EL over tweets. TweekiData was created by using 15 16pedia abstracts. Its main purpose is Knowledge Base other existing tweet-based datasets and linking them to 16 17Population. According to Mulang et al. [80], this Wikidata data via the Tweeki EL. TweekiGold was cre- 17 18dataset describes the challenges of Wikidata, at least ated by an expert, manually annotating tweets from an- 18 19in the form of long, noisy labels, the best. other dataset with Wikidata identifiers and Wikipedia 19 20page-titles. 20 ISTEX-1000 [23] is a research-focused dataset con- 21 21 taining 1000 author affiliation strings. It was manually Table 13 shows the number of documents, the num- 22 22 annotated to evaluate the OpenTapioca entity linker. ber of mentions, emerging entities and unique entities, 23 23 and the mentioned ratio. What classifies as a document 24KnowledgeNet [22] is a Knowledge Base Population 24 in a dataset depends on the dataset itself. For exam- 25dataset with 9073 manually annotated sentences. The 25 ple, for T-REx, a document is a whole paragraph of 26text was extracted from biographical documents from 26 a Wikipedia article, while for LC-QuAD 2.0, a docu- 27the web or Wikipedia articles. 27 ment is just a single question. Due to this, the average 28 28 NYT2018 [69, 70] consists of 30 news documents that amount of entities in a document also varies, e.g., LC- 29 29 were manually annotated on Wikidata and DBpedia. It QuAD 2.0 with 1.47 entities per document and T-REx 30 30 was constructed for KBPearl, so its main focus is also with 11.03. If a dataset was not available, information 31 31 KBP which is a downstream task of EL. from the original paper was included. If dataset splits 32 32 were available, the statistics are also shown separately. 33One dataset, KORE 50 DYWC [82], was found, which 33 The majority of datasets do not contain emerging enti- 34was not used by any of the approach papers. It is an 34 ties. For the Tweeki datasets, it is not mentioned which 35annotated EL dataset based on the KORE50 dataset, 35 Wikidata dump was used to annotate. For a dataset that 36a manually annotated subset of the AIDA corpus. All 36 contains emerging entities, this is problematic. On the 37sentences are reannotated with DBpedia, Yago, Wiki- 37 other hand, the dump is specified for the CLEF HIPE 38data and Crunchbase entities. 38 2020 dataset, making it possible to work on the Wiki- 39 39 The Kensho Derived Wikimedia Dataset [58] is an data version with the correct entities missing. 40 40 automatically created condensed subset of Wikimedia 41To get an overview how widespread they datasets are 41 data. It consists of three levels: Wikipedia text, an- 42in use, see the section 5.2. Thus, RQ 3 is answered. 42 notations with Wikipedia pages and links to Wikidata 43 43 items. Thus, mentions in Wikipedia articles are anno- 446.2. Evaluation 44 tated with Wikidata items. However, as some Wikidata 45 45 items do not have a corresponding Wikipedia page, 46The difficulty of the different datasets was measured 46 the annotation is not exhaustive. It was constructed for 47by the accuracy of a simple EL method (Table 14) and 47 NLP in general. 48the ambiguity of mentions (Table 12). The simple EL 48 49CLEF HIPE 2020 [29] is a dataset based on histori- method searches for entity candidates via an Elastic- 49 50cal newspapers in English, French and German. Only Search index, including all English labels and aliases. 50 51the English dataset will be analyzed in the following. It then disambiguates by taking the one with the largest 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 23

1Table 12: Ambiguity of mentions (existence of a match does not correspond to a correct match). 1 2 2 3Dataset Average num- No match Exact match More than one 3 4ber of matches match 4 5ISTEX-1000 (train) 23.23 8.06% 26.34% 65.61% 5 6ISTEX-1000 (test) 25.85 10.30% 23.88% 65.82% 6 7Wiki-Disamb30 (train) 25.06 0.36% 1.26% 98.38% 7 8Wiki-Disamb30 (dev) 30.39 0.40% 1.18% 98.42% 8 9Wiki-Disamb30 (test) 30.18 0.30% 1.44% 98.26% 9 10Knowledge Net (train) 21.90 10.41% 22.29% 67.3% 10 11T-REx 4.79 31.36% 32.98% 35.65% 11 12KORE50DYWC 28.31 3.93% 7.49% 88.60% 12 13Kensho Derived Wikimedia Dataset 8.16 35.18% 30.94% 33.88% 13 14CLEF HIPE 2020 (en, dev) 24.02 35.71% 11.51% 52.78% 14 15CLEF HIPE 2020 (en, test) 17.78 43.82% 6.74% 49.44% 15 16Mewsli-9 (en) 11.09 16.80% 34.90% 47.30% 16 17TweekiData 19.61 19.98% 12.01% 68.01% 17 18TweekiGold 16.02 7.41% 20.25% 72.34% 18 19 19 20 20 21tf-idf based BM25 similarity measure score and the no exact match. The reason for that is the noise created 21 22lowest Q-identifier number resembling the popularity. by OCR. Only the English dataset was examined. The 22 5 23Nothing was done to handle inflections. Here, only second column of Table 14 specifies the accuracy with 23 24datasets were included which were accessible. As one all unique exact matches removed. This is based on 24 25can see, is the accuracy positively correlated with the the intuition that exact matches without any competi- 25 26number of exact matches. The more ambiguous the tors are usually correct. In general, the removal does 26 27underlying entity mentions are, the more inaccurate decrease the accuracy with one exception. The Wiki- 27 28a simple similarity measure between label and men- Disamb30 datasets constantly achieve better accuracy 28 29tion becomes. In this case, more context information as a large percentage of the unique exact matches ap- 29 30is necessary. The simple Entity Linker was only appear to point to wrong entities. Thus, the true entity 30 31plied to datasets that were feasible to disambiguate does not have the label it is referenced by. 31 32in that way. T-REx and the Kensho Derived Wikime- 32 Two main characteristics of Wikidata may affect the 33dia Dataset were too large. According to the EL per- 33 design of Wikidata EL datasets. First, multilingual- 34formance, ISTEX-1000 is the easiest dataset. Many 34 35of the ambiguous mentions reference the most pop- ism is the main focus of Wikidata, and thus, multi- 35 36ular one, while also many exact unique matches ex- lingual datasets should also be a focus. Unfortunately, 36 37ist. T-REx, the Kensho Derived Wikimedia Dataset only two datasets [14, 29] focus on the multilingual- 37 38and the Mewsli-9 training dataset have the largest per- ism of Wikidata. The CLEF HIPE 2020 dataset is de- 38 39centage of exact matches for labels. The largest num- signed for Wikidata and has documents for the lan- 39 40ber of ambiguous mentions have the Wiki-Disamb30 guages English, French and German, but each lan- 40 41datasets, resulting in a low EL but not the lowest ac- guage has a different corpus of documents. The same 41 42curacy. Deciding on the most prominent entity appears is the case for the Mewsli-9 dataset, while here, docu- 42 43to produce good EL results. This is also the case for ments in nine languages are available. A dataset sim- 43 44the TweekiGold dataset. While the KORE50DYWC ilar to VoxEL [94], which is defined for Wikipedia, 44 45dataset is less ambiguous than Wiki-Disamb30, it per- would be helpful. Here, each utterance is translated 45 46forms the worst due to references to unpopular entities. into multiple languages, which eases the comparison 46 47The CLEF HIPE 2020 dataset also has a low EL accu- of the multilingual EL performance. Having the same 47 48racy but not due to ambiguity but many mentions with corpus of documents in different languages would al- 48 49low a better comparison of a method’s performance 49 505All source code, plots and results can be found on https://github. in various languages. Of course, such translations will 50 51com/cedricm-research/ELEnglishWD never be perfectly comparable. 51 24 C. Möller et al. / Survey on English Entity Linking on Wikidata

1Table 13: Comparison of the datasets with focus on the number of documents and Wikidata entities. 1 2 2

weiod[8 0 5 11 89 66 1.92 - 66.6% 2.91 6.14 - 88.9% 5.88 42.5% 72.0% 11.1% 31.9% - 66.4% 100% 53.6% 2.68 3 958 2.76 33.6% - 46.4% 65.8% 0% 500 53.7% 5 134 12 100% 470 - 307 100% 46 [48] TweekiGold 44 0% 13 4 80 [48] 30 TweekiData 50 [14] (en) [29] Mewsli-9 test) 0% (en, 2020 HIPE [29] CLEF Wikimedia dev) (en, 2020 HIPE CLEF 6046 Derived 3977 [58] Dataset 670 Kensho 2073 [82] KORE50DYWC 70] [69, NYT2018 10 (train) [22] Net Knowledge 10 250 [34] T-REx 100 [27] 2.0 750 LC-QuAD (dev) [18] Wikidata-Disamb (test) [18] Wikidata-Disamb (train) [18] Wikidata-Disamb (test) [23] ISTEX-1000 (train) [23] ISTEX-1000 Dataset 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 15 15

1614 # 16 , , 000 650 documents , , , , 7 80 679 255 10 10 000 000

17, 17 0 100 000 , , 0 5 000 51 000

18, 18 5 121 258 19 19 20 20 21 21 # ,

22038 en- Emerging mentions 22 , , , , , , 4 %10 82 6.33 48.2% 3.28 100% 1.47 1.0 30% 1.0 0% 51.2% 56.2% 57.3% 100% 242 100% 100% 100% 0% 0% 0% 039 0% 297 529 000 000 , , 23 1.0 27.2% 835 100% 0% 000 23 , 7 12 88 .%1.01 5.4% 38.8% 61.2% 870 ,

24 11.03 9.1% 100% 0% 484 24 , 5 %10 .%8.55 3.7% 100% 0% 453 25 25 26 26 27 27 tities 28 28 29 29 30 30 31 31 32 32 33 33 tities en- Wikidata 34 34 35 35 36 36 37 37 38 38 39 39 aaentities data Wiki- Unique 40 40 41 41 42 42 43 43 44 44 45 45 document per Mentions 46 46 47 47 48 48 49 49 50 50 51 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 25

1Table 14: EL accuracy - Kensho Derived Wikimedia cused on Wikidata. The survey also discusses the cur- 1 2Dataset, T-REx and TweekiData are not included due rent state of the art of domain-independent and multi- 2 3to size, Acc. filtered has all exact matches removed. lingual neural EL approaches. However, the influence 3 4of the underlying KG was not of concern to the au- 4 5Dataset Acc. Acc. filtered thors. It is not described in detail how they found the 5 6ISTEX-1000 (train) 0.744 0.716 considered approaches. 6 7 7 ISTEX-1000 (test) 0.716 0.678 In the survey by Al-Moslmi et al. [2], the focus lies on 8 8 Wiki-Disamb30 (train) 0.597 0.600 ER and EL approaches over KGs in general. It consid- 9 9 Wiki-Disamb30 (dev) 0.580 0.584 ers approaches from 2014 to 2019. It gives an overview 10 10 Wiki-Disamb30 (test) 0.576 0.580 of the different approaches of ER, Entity Disambigua- 11 11 Knowledge Net (train) 0.371 0.285 tion, and EL. A distinction between Entity Disam- 12 12 KORE50DYWC 0.225 0.187 biguation and EL is made, while our survey sees En- 13 13 CLEF HIPE 2020 (en, dev) 0.333 0.287 tity Disambiguation as a part of EL. The roles of dif- 14 14 CLEF HIPE 2020 (en, test) 0.258 0.241 ferent domains, text types, or languages are discussed. 15 15 TweekiGold 0.565 0.520 The authors considered 89 different approaches and 16 16 Mewsli-9 (en) 0.602 0.490 tools. Most approaches were designed for DBpedia or 17 17 Wikipedia, some for Freebase or YAGO, and some to 18 18 be KG-agnostic. Again, the only Wikidata contender 19The second characteristic is the large rate of change 19 was Deeptype. F1 scores were gathered on 17 different 20of Wikidata. Thus, it would also be advisable that the 20 datasets. Fifteen algorithms, for which an implementa- 21datasets specify the Wikidata dumps they were cre- 21 tion or a WebAPI was available, were evaluated using 22ated, similar to Petroni et al. [87]. Many of the existing 22 GERBIL [92]. 23datasets do that, yet not all. In current dumps, entities, 23 which were available while the dataset was created, 24Another survey [83] examines recent approaches, 24 could have been removed. It is even more probable that 25which employ holistic strategies. Holism in the context 25 emerging entities could now have a corresponding en- 26of EL is defined as the usage of domain-specific inputs 26 tity in an updated Wikidata dump version. If the EL ap- 27and metadata, joint ER-EL approaches and collective 27 proach now would detect it as an emerging entity, it is 28disambiguation methods. Thirty-six research articles 28 evaluated as correct, but in reality, this is false and vice 29were found which had any holistic aspect - none of the 29 versa. Concerning emerging entities, another variant of 30designed approaches linked explicitly to Wikidata. 30 31an EL dataset could be useful. Two Wikidata dumps 31 A comparison of the number of approaches and 32from different time points could be used to label the ut- 32 datasets included in the different surveys can be found 33terances. Such a dataset would be valuable in the con- 33 in Table 15. 34text of Knowledge Graph Population when emerging 34 entities are inserted into the KG. With the true emerg- 35If we go further into the past, the existing surveys [72, 35 ing entity available, one could measure the quality of 36102] are not considering Wikidata at all or only in a 36 the insertion. Also, constraining that the method needs 37small amount as it is still a rather recent KG in compar- 37 to perform well on both KG dumps would force EL 38ison to the other established ones like DBpedia, Free- 38 approaches to be less reliant on a fixed graph structure. 39base or YAGO. For an overview on different KGs on 39 This answers RQ 4. 40the web, we refer the interested reader to the one by 40 41Heist et al. [50]. 41 42 42 7. Related work No found survey focused on the differences of EL over 43 43 different knowledge graphs, respectively, on the par- 44 44 While there are multiple recent surveys on EL, none ticularities of EL over Wikidata. 45 45 of those are specialized in analyzing the area of EL on 46 46 Wikidata. 47 47 48The extensive survey by Sevgili et al. [99] is giving an 48 49overview of all neural approaches from 2015 to 2020. 49 50It compares 30 different approaches on nine different 50 51datasets. Of those, only Deeptype can be seen as fo- 51 26 C. Möller et al. / Survey on English Entity Linking on Wikidata

1Table 15: Survey Comparison 1 2 2 3Survey # Approaches # Wikidata Approaches # Datasets # Wikidata Datasets 3 4Sevgili et al. [99] 30 1 9 0 4 5Al-Moslmi et al. [2] 39 1 17 0 5 6Oliveira et al. [83] 36 0 32 0 6 7This survey 17 17 21 11 7 8 8 9 9 108. Discussion data as the target KG with its language-agnostic iden- 10 11tifiers and the easily connectable Wikipedia with its 11 128.1. Current Approaches, Datasets and their multilingual textual information are the perfect pair. 12 Drawbacks 13Most of the approaches tried to use Wikidata due to it 13 14 14 Approaches. The number of algorithms using Wiki- being up to date while not utilizing its structure. With 15 15 data is small; the number of algorithms using Wikidata small adjustments, many would also work on any other 16 16 solely is even smaller. Most algorithms employ labels KG. None of the investigated approaches tried to ex- 17 17 and alias information contained in Wikidata. Some amine the performance between different versions of 18 18 deep learning-based algorithms leverage the underly- Wikidata. As continuous evolution is a central charac- 19 19 ing graph structure, but the inclusion of that informa- teristic of Wikidata, a temporal analysis would be rea- 20 20 tion is often superficial. The same information is also sonable. 21 21 available in other KGs. Additional statement specific 22This survey aimed to identify the extent to which the 22 information like qualifiers is used by only one algo- 23current state of the art in Wikidata EL is utilizing the 23 rithm (OpenTapioca), and even then, it only interprets 24characteristics of Wikidata. As only a few are using 24 qualifiers as extra edges to the item. Thus, there is no 25more information than on other established KGs, there 25 inclusion of the actual structure of a hyper-relation. In- 26is still much potential for future research. 26 formation like the descriptions of items which are pro- 27 27 viding valuable context information is also used sel- 28Datasets. Only a limited amount of datasets were 28 dom. Wikidata includes type information, but almost 29created entirely with Wikidata in mind exist. Many 29 none of the existing algorithms utilize it to do more 30datasets used are still only mapped versions of datasets 30 than to filter out entities that are not desired to link in 31created for other knowledge bases. Multilingualism is 31 general. An exception is Tweeki, which uses it together 32present so far that some datasets contain documents 32 with ER, and perhaps DeepType, though the evaluated 33in different languages. However, only different docu- 33 model used Wikipedia categories. 34ments for different languages are available. Having the 34 35One could claim that the current algorithms are mostly same documents in multiple languages would be more 35 36trying to map algorithms also usable on other KGs to helpful for an evaluation of multilingual Entity Link- 36 37Wikidata. Besides utilizing specific characteristics of ers. The fact that the Wikidata is ever-changing is also 37 38Wikidata, it is also notable that there is no clear focus not genuinely considered in any datasets. Always pro- 38 39on one of the essential characteristics of Wikidata, the viding the dump version on which the dataset was cre- 39 40continual growth. Many approaches use static graph ated is advisable. Great is that datasets from very dif- 40 41embeddings, which need to be retrained if the KG ferent domains like news, forums, research, tweets ex- 41 42changes. EL algorithms working on Wikidata, which ist. The utterances can also vary from shorter texts with 42 43are not usable on future versions, seem unintuitive. only a few entities to large documents with many enti- 43 44But there also exist some approaches which can han- ties. The difficulty of the datasets significantly differs 44 45dle change. They often rely on more extensive tex- in the ambiguity of the entity mentions. The datasets 45 46tual information, which is again challenging due to also differ in quality. Some were automatically created 46 47the limited amount of such data in Wikidata. Wiki- and others annotated manually by experts. There are no 47 48data descriptions do exist, but only short paragraphs unanimously agreed upon datasets used for Wikidata 48 49are provided, in general, insufficient to train a language EL. Of course, a single dataset can not exist as differ- 49 50model. To compensate, Wikipedia is included, which ent domains and text types make different approaches, 50 51provides this textual information. It seems like Wiki- and hence datasets necessary. 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 27

18.2. Future Research Avenues Affairs and Energy (BMWi) project SPEAKER (FKZ 1 201MK20011A), the German Federal Ministry of Edu- 2 3In general, Wikidata EL could be improved by includ- cation and Research (BMBF) projects and excellence 3 4ing: clusters ML2R (FKZ 01 15 18038 A/B/C), MLwin 4 5(01S18050 D/F), ScaDS.AI (01/S18026A) as well as 5 6– Hyper-relational statements which provide addi- the Fraunhofer Zukunftsstiftung project JOSEPH. 6 7tional information 7 8– Type information for more than limiting the candi- 8 date space 9References 9 10– Inductive or efficently trainable knowledge graph em- 10 11beddings [1] P. Agarwal, J. Strötgen, L.D. Corro, J. Hoffart and 11 G. Weikum, diaNED: Time-Aware Named Entity Disam- 12– Item label and description information in multiple 12 biguation for Diachronic Corpora, in: Proceedings of the 13languages for multilingual EL 13 56th Annual Meeting of the Association for Computa- 14tional Linguistics, ACL 2018, Melbourne, Australia, July 14 15The qualifier and rank information of Wikidata could 15-20, 2018, Volume 2: Short Papers, I. Gurevych and 15 16be also suitable to do EL on time-sensitive utter- Y. Miyao, eds, Association for Computational Linguistics, 16 17ances [1]. The problem evolves around utterances 2018, pp. 686–693. doi:10.18653/v1/P18-2109. https://www. 17 aclweb.org/anthology/P18-2109/. 18which talk about entities from different time points and 18 [2] T. Al-Moslmi, M.G. Ocaña, A.L. Opdahl and C. Veres, 19spans and thus, the referred entity can significantly di- 19 verge. Named Entity Extraction for Knowledge Graphs: A Lit- 20erature Overview, IEEE Access 8 (2020), 32862–32881. 20 21The usefulness of other characteristics of Wikidata, doi:10.1109/ACCESS.2020.2973928. 21 22e.g., references, may be limited but could make EL [3] M. Albooyeh, R. Goel and S.M. Kazemi, Out-of-Sample 22 23more challenging due to the inclusion of contradic- Representation Learning for Knowledge Graphs, in: Pro- 23 ceedings of the 2020 Conference on Empirical Methods in 24 24 tory information. Therefore, research into the con- Natural Language Processing: Findings, EMNLP 2020, On- 25 25 sequences and solutions of conflicting information line Event, 16-20 November 2020, T. Cohn, Y. He and 26would be advisable. Y. Liu, eds, Association for Computational Linguistics, 26 272020, pp. 2657–2666. https://www.aclweb.org/anthology/ 27 28To reiterate, due to the fast rate of change of Wikidata, 2020.findings-emnlp.241/. 28 29approaches are necessary, which are more robust to [4] P.D. Almeida, J.G. Rocha, A. Ballatore and A. Zipf, Where 29 the Streets Have Known Names, in: Computational Science 30such a dynamic KG. Continuously retraining transduc- 30 and Its Applications - ICCSA 2016 - 16th International Con- 31tive embeddings is intractable, so more sophisticated 31 ference, Beijing, China, July 4-7, 2016, Proceedings, Part 32methods like inductive or efficiently retrainable graph 32 embeddings are a necessity. IV, O. Gervasi, B. Murgante, S. Misra, A.M.A.C. Rocha, 33C.M. Torre, D. Taniar, B.O. Apduhan, E.N. Stankova and 33 34Multilingual or cross-lingual EL is already tackled S. Wang, eds, Lecture Notes in Computer Science, Vol. 9789, 34 35with Wikidata but currently mainly by depending on Springer, 2016, pp. 1–12. doi:10.1007/978-3-319-42089-9_- 35 1. 36Wikipedia. Using the available multilingual label/de- 36 [5] J. Baek, D.B. Lee and S.J. Hwang, Learning to Extrapolate 37 37 scription information in a structured form together Knowledge: Transductive Few-shot Out-of-Graph Link Pre- 38with the rich textual information in Wikipedia could diction, in: Advances in Neural Information Processing Sys- 38 39move the field forward. tems 33: Annual Conference on Neural Information Process- 39 40ing Systems 2020, NeurIPS 2020, December 6-12, 2020, vir- 40 It seems like there exist no commonly agreed on Wiki- 41tual, H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan and 41 data EL datasets as shown by a large number of dif- H. Lin, eds, 2020. https://proceedings.neurips.cc/paper/2020/ 42 42 ferent datasets the approaches were tested on. Such hash/0663a4ddceacb40b095eda264a85f15c-Abstract.html. 43 43 datasets should try to represent the challenges of Wiki- [6] D. Banerjee, D. Chaudhuri, M. Dubey and J. Lehmann, 44 44 data like the time-variance, contradictory triple infor- PNEL: Pointer Network Based End-To-End Entity Link- 45ing over Knowledge Graphs, in: The Semantic Web - 45 mation, noisy labels, and multilingualism. 46ISWC 2020 - 19th International Semantic Web Confer- 46 47ence, Athens, Greece, November 2-6, 2020, Proceedings, 47 Part I, J.Z. Pan, V.A.M. Tamma, C. d’Amato, K. Janow- 48 48 Acknowledgments icz, B. Fu, A. Polleres, O. Seneviratne and L. Kagal, eds, 49 49 Lecture Notes in Computer Science, Vol. 12506, Springer, 50We acknowledge the support of the EU project TAI- 2020, pp. 21–38. doi:10.1007/978-3-030-62419-4_2. https: 50 51LOR (GA 952215), the Federal Ministry for Economic //doi.org/10.1007/978-3-030-62419-4_2. 51 28 C. Möller et al. / Survey on English Entity Linking on Wikidata

1[7] A. Bastos, A. Nadgeri, K. Singh, I.O. Mulang, S. Shekarpour [19] M. Cornolti, P. Ferragina and M. Ciaramita, A frame- 1 2and J. Hoffart, RECON: Relation Extraction using Knowl- work for benchmarking entity-annotation systems, 2 3edge Graph Context in a Graph Neural Network, CoRR in: 22nd International World Wide Web Conference, 3 abs/2009.08694 (2020). https://arxiv.org/abs/2009.08694. WWW ’13, Rio de Janeiro, Brazil, May 13-17, 2013, 4 4 [8] J. Berant, A. Chou, R. Frostig and P. Liang, Semantic D. Schwabe, V.A.F. Almeida, H. Glaser, R. Baeza-Yates 5 5 Parsing on Freebase from Question-Answer Pairs, in: Pro- and S.B. Moon, eds, International World Wide Web Con- 6ceedings of the 2013 Conference on Empirical Methods in ferences Steering Committee / ACM, 2013, pp. 249–260. 6 7Natural Language Processing, EMNLP 2013, 18-21 Octo- doi:10.1145/2488388.2488411. 7 8ber 2013, Grand Hyatt Seattle, Seattle, Washington, USA, [20] H.-J. Dai, C.-Y. Wu, R.T.-H. Tsai and W.-L. Hsu, From 8 9A meeting of SIGDAT, a Special Interest Group of the entity recognition to entity linking: a survey of advanced 9 ACL, ACL, 2013, pp. 1533–1544. https://www.aclweb.org/ entity linking techniques, in: 26 (2012), , 2012, 10 10 anthology/D13-1160/. pp. 3M2IOS3b1–3M2IOS3b1. 11 11 [9] P. Bhargava, N. Spasojevic, S. Ellinger, A. Rao, A. Menon, [21] DBpedia, DBpedia Live. https://wiki.dbpedia.org/online- 12S. Fuhrmann and G. Hu, Learning to Map Wikidata En- access/DBpediaLive. 12 13tities To Predeﬁned Topics, in: Companion of The 2019 [22] F. de Sá Mesquita, M. Cannaviccio, J. Schmidek, P. Mirza 13 14World Wide Web Conference, WWW 2019, San Francisco, and D. Barbosa, KnowledgeNet: A Benchmark Dataset for 14 CA, USA, May 13-17, 2019, S. Amer-Yahia, M. Mah- 15Knowledge Base Population, in: Proceedings of the 2019 15 dian, A. Goel, G. Houben, K. Lerman, J.J. McAuley, 16Conference on Empirical Methods in Natural Language Pro- 16 R. Baeza-Yates and L. Zia, eds, ACM, 2019, pp. 1194–1202. cessing and the 9th International Joint Conference on Natural 17 17 doi:10.1145/3308560.3316749. Language Processing, EMNLP-IJCNLP 2019, Hong Kong, 18[10] K.D. Bollacker, C. Evans, P. Paritosh, T. Sturge and J. Tay- China, November 3-7, 2019, K. Inui, J. Jiang, V. Ng and 18 19lor, Freebase: a collaboratively created graph database X. Wan, eds, Association for Computational Linguistics, 19 Proceedings of the 20for structuring human knowledge, in: 2019, pp. 749–758. doi:10.18653/v1/D19-1069. 20 ACM SIGMOD International Conference on Management 21[23] A. Delpeuch, OpenTapioca: Lightweight Entity Linking for 21 of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10- Wikidata, CoRR abs/1904.09131 (2019). http://arxiv.org/abs/ 2212, 2008, J.T. Wang, ed., ACM, 2008, pp. 1247–1250. 22 1904.09131. 23doi:10.1145/1376616.1376746. 23 [24] L. Derczynski, D. Maynard, G. Rizzo, M. van Erp, G. Gorrell, 24[11] A. Bordes, N. Usunier, A. García-Durán, J. Weston 24 R. Troncy, J. Petrak and K. Bontcheva, Analysis of named and O. Yakhnenko, Translating Embeddings for Model- 25entity recognition and linking for tweets, Inf. Process. Manag. 25 ing Multi-relational Data, in: Advances in Neural Infor- 2651(2) (2015), 32–49. doi:10.1016/j.ipm.2014.10.006. 26 mation Processing Systems 26: 27th Annual Conference [25] J. Devlin, M. Chang, K. Lee and K. Toutanova, BERT: Pre- 27on Neural Information Processing Systems 2013. Pro- 27 training of Deep Bidirectional Transformers for Language 28ceedings of a meeting held December 5-8, 2013, Lake 28 Understanding, in: Proceedings of the 2019 Conference of 29Tahoe, Nevada, United States, C.J.C. Burges, L. Bot- 29 the North American Chapter of the Association for Computa- 30tou, Z. Ghahramani and K.Q. Weinberger, eds, 2013, 30 tional Linguistics: Human Language Technologies, NAACL- 31pp. 2787–2795. http://papers.nips.cc/paper/5071-translating- 31 embeddings-for-modeling-multi-relational-data. HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Vol- 32 32 [12] A. Bordes, N. Usunier, S. Chopra and J. Weston, Large-scale ume 1 (Long and Short Papers), J. Burstein, C. Doran and 33Simple Question Answering with Memory Networks, CoRR T. Solorio, eds, Association for Computational Linguistics, 33 34abs/1506.02075 (2015). http://arxiv.org/abs/1506.02075. 2019, pp. 4171–4186. doi:10.18653/v1/n19-1423. 34 35[13] E. Boros, E.L. Pontes, L.A. Cabrera-Diego, A. Hamdi, [26] C. Dogan, A. Dutra, A. Gara, A. Gemma, L. Shi, M. Sigamani 35 and E. Walters, Fine-Grained Named Entity Recognition us- 36N. Sidère and A. Doucet, Robust Named Entity Recognition 36 and Linking on Historical Multilingual Documents (2020). ing ELMo and Wikidata, CoRR abs/1904.10503 (2019). http: 37 37 [14] J.A. Botha, Z. Shan and D. Gillick, Entity Linking in 100 Lan- //arxiv.org/abs/1904.10503. 38 38 guages, CoRR abs/2011.02690 (2020). https://arxiv.org/abs/ [27] M. Dubey, D. Banerjee, A. Abdelkawi and J. Lehmann, LC- 392011.02690. QuAD 2.0: A Large Dataset for Complex Question Answer- 39 40[15] D.G. Brizan and A.U. Tansel, A. survey of entity resolution ing over Wikidata and DBpedia, in: The Semantic Web - 40 41and record linkage methodologies, Communications of the ISWC 2019 - 18th International Semantic Web Conference, 41 Auckland, New Zealand, October 26-30, 2019, Proceedings, 42IIMA 6(3) (2006), 5. 42 [16] M. Burtsev, A. Seliverstov, R. Airapetyan, M. Arkhipov, Part II, C. Ghidini, O. Hartig, M. Maleshkova, V. Svátek, 43 43 D. Baymurzina, N. Bushkov, O. Gureenkova, T. Khakhulin, I.F. Cruz, A. Hogan, J. Song, M. Lefrançois and F. Gan- 44Y. Kuratov, D. Kuznetsov et al., Deeppavlov: Open-source li- don, eds, Lecture Notes in Computer Science, Vol. 11779, 44 45brary for dialogue systems, in: Proceedings of ACL 2018, Sys- Springer, 2019, pp. 69–78. doi:10.1007/978-3-030-30796-7_- 45 46tem Demonstrations, 2018, pp. 122–127. 5. https://doi.org/10.1007/978-3-030-30796-7_5. 46 47[17] J. Callan, M. Hoy, C. Yoo and L. Zhao, Clueweb09 data set, [28] M. Ehrmann, M. Romanello, A. Flückiger and S. Clematide, 47 Extended overview of CLEF HIPE 2020: named entity pro- 482009. 48 [18] A. Cetoli, M. Akbari, S. Bragaglia, A.D. O’Harney and cessing on historical newspapers, in: CLEF, 2020a. 49 49 M. Sloan, Named Entity Disambiguation using Deep Learn- [29] M. Ehrmann, M. Romanello, A. Flückiger and S. Clematide, 50ing on Graphs, CoRR abs/1810.09164 (2018). http://arxiv. Extended overview of CLEF HIPE 2020: named entity pro- 50 51org/abs/1810.09164. cessing on historical newspapers, in: CLEF, 2020b. 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 29

1[30] C.B. El Vaigh, G. Le Noé-Bienvenu, G. Gravier and P. Sébil- [42] J. Geiß, A. Spitz and M. Gertz, NECKAr: A Named Entity 1 2lot, IRISA System for Entity Detection and Linking at CLEF Classifier for Wikidata, in: Language Technologies for the 2 3HIPE 2020, in: CEUR Workshop Proceedings, 2020. Challenges of the Digital Age - 27th International Confer- 3 [31] R. Ellgren, Exploring Emerging Entities and Named Entity ence, GSCL 2017, Berlin, Germany, September 13-14, 2017, 4 4 Disambiguation in News Articles, 2020a. Proceedings, G. Rehm and T. Declerck, eds, Lecture Notes in 5 5 [32] R. Ellgren, Exploring Emerging Entities and Named Entity Computer Science, Vol. 10713, Springer, 2017, pp. 115–129. 6Disambiguation in News Articles, 2020b. doi:10.1007/978-3-319-73706-5_10. https://doi.org/10.1007/ 6 7[33] J. Ellis, S. Strassel and J. Getman, TAC KBP Evaluation 978-3-319-73706-5_10. 7 8Source Corpora 2016-2017, Linguistic Data Consortium, [43] G.A. Gesese, M. Alam and H. Sack, Semantic Entity Enrich- 8 University of Pennsylvania, 2019. 9ment by Leveraging Multilingual Descriptions for Link Pre- 9 [34] H. ElSahar, P. Vougiouklis, A. Remaci, C. Gravier, J.S. Hare, CoRR abs/2004.10640 10diction, (2020). https://arxiv.org/abs/ 10 F. Laforest and E. Simperl, T-REx: A Large Scale Align- 2004.10640. 11ment of Natural Language with Knowledge Base Triples, 11 [44] M.R. Glass and A. Gliozzo, A Dataset for Web-Scale Knowl- in: Proceedings of the Eleventh International Conference on 12edge Base Population, in: The Semantic Web - 15th Inter- 12 Language Resources and Evaluation, LREC 2018, Miyazaki, 13national Conference, ESWC 2018, Heraklion, Crete, Greece, 13 Japan, May 7-12, 2018, N. Calzolari, K. Choukri, C. Cieri, 14June 3-7, 2018, Proceedings, A. Gangemi, R. Navigli, M. Vi- 14 T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, dal, P. Hitzler, R. Troncy, L. Hollink, A. Tordai and M. Alam, 15J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis and 15 eds, Lecture Notes in Computer Science, Vol. 10843, 16T. Tokunaga, eds, European Language Resources Associ- 16 Springer, 2018, pp. 256–271. doi:10.1007/978-3-319-93417- 17ation (ELRA), 2018. http://www.lrec-conf.org/proceedings/ 17 4_17. https://doi.org/10.1007/978-3-319-93417-4_17. 18lrec2018/summaries/632.html. 18 [35] F. Erxleben, M. Günther, M. Krötzsch, J. Mendez and [45] A. Graves, S. Fernández and J. Schmidhuber, Bidirec- 19 19 D. Vrandecic, Introducing Wikidata to the Linked Data tional LSTM Networks for Improved Phoneme Classifica- 20Web, in: The Semantic Web - ISWC 2014 - 13th Interna- tion and Recognition, in: Artificial Neural Networks: For- 20 21tional Semantic Web Conference, Riva del Garda, Italy, Oc- mal Models and Their Applications - ICANN 2005, 15th 21 22tober 19-23, 2014. Proceedings, Part I, P. Mika, T. Tudo- International Conference, Warsaw, Poland, September 11- 22 15, 2005, Proceedings, Part II, W. Duch, J. Kacprzyk, 23rache, A. Bernstein, C. Welty, C.A. Knoblock, D. Vrande- 23 cic, P. Groth, N.F. Noy, K. Janowicz and C.A. Goble, eds, E. Oja and S. Zadrozny, eds, Lecture Notes in Com- 24 24 Lecture Notes in Computer Science, Vol. 8796, Springer, puter Science, Vol. 3697, Springer, 2005, pp. 799–804. 252014, pp. 50–65. doi:10.1007/978-3-319-11964-9_4. https: doi:10.1007/11550907_126. 25 26//doi.org/10.1007/978-3-319-11964-9_4. [46] S. Guan, X. Jin, Y. Wang and X. Cheng, Link Prediction on 26 27[36] M. Färber, B. Ell, C. Menne and A. Rettinger, A comparative N-ary Relational Data, in: The World Wide Web Conference, 27 28survey of dbpedia, freebase, opencyc, wikidata, and yago, Se- WWW 2019, San Francisco, CA, USA, May 13-17, 2019, 28 mantic Web Journal 1(1) (2015), 1–5. L. Liu, R.W. White, A. Mantrach, F. Silvestri, J.J. McAuley, 29 29 [37] M. Färber, F. Bartscherer, C. Menne and A. Rettinger, Linked R. Baeza-Yates and L. Zia, eds, ACM, 2019, pp. 583–593. 30 30 data quality of DBpedia, Freebase, OpenCyc, Wikidata, and doi:10.1145/3308558.3313414. 31YAGO, Semantic Web 9(1) (2018), 77–129. doi:10.3233/SW- [47] T. Hamaguchi, H. Oiwa, M. Shimbo and Y. Matsumoto, 31 32170275. Knowledge Transfer for Out-of-Knowledge-Base Entities : 32 33[38] P. Ferragina and U. Scaiella, TAGME: on-the-fly annotation A Graph Neural Network Approach, in: Proceedings of the 33 of short text fragments (by wikipedia entities), in: Proceed- 34Twenty-Sixth International Joint Conference on Artificial In- 34 ings of the 19th ACM Conference on Information and Knowl- telligence, IJCAI 2017, Melbourne, Australia, August 19- 35 35 edge Management, CIKM 2010, Toronto, Ontario, Canada, 25, 2017, C. Sierra, ed., ijcai.org, 2017, pp. 1802–1808. 36October 26-30, 2010 36 , J. Huang, N. Koudas, G.J.F. Jones, doi:10.24963/ijcai.2017/250. X. Wu, K. Collins-Thompson and A. An, eds, ACM, 2010, 37[48] B. Haradizadeh and S. Singh, Tweeki: Linking Named Enti- 37 pp. 1625–1628. doi:10.1145/1871437.1871689. 38ties on Twitter to a Knowledge Graph, in: Workshop on Noisy 38 [39] M. Galkin, P. Trivedi, G. Maheshwari, R. Usbeck and 39User-generated Text (W-NUT), 2020. 39 J. Lehmann, Message Passing for Hyper-Relational Knowl- [49] S. Heindorf, M. Potthast, B. Stein and G. Engels, Van- 40edge Graphs, CoRR abs/2009.10847 (2020). https://arxiv.org/ 40 Proceedings of the 25th 41abs/2009.10847. dalism Detection in Wikidata, in: 41 ACM International Conference on Information and Knowl- 42[40] O. Ganea and T. Hofmann, Deep Joint Entity Disambigua- 42 edge Management, CIKM 2016, Indianapolis, IN, USA, Oc- 43tion with Local Neural Attention, in: Proceedings of the 2017 43 Conference on Empirical Methods in Natural Language Pro- tober 24-28, 2016, S. Mukhopadhyay, C. Zhai, E. Bertino, 44 44 cessing, EMNLP 2017, Copenhagen, Denmark, September 9- F. Crestani, J. Mostafa, J. Tang, L. Si, X. Zhou, Y. Chang, 4511, 2017, M. Palmer, R. Hwa and S. Riedel, eds, Associ- Y. Li and P. Sondhi, eds, ACM, 2016, pp. 327–336. 45 46ation for Computational Linguistics, 2017, pp. 2619–2629. doi:10.1145/2983323.2983740. 46 47doi:10.18653/v1/d17-1277. [50] N. Heist, S. Hertling, D. Ringler and H. Paulheim, Knowledge 47 Graphs on the Web - An Overview, in: Knowledge Graphs for 48[41] M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi, 48 N.F. Liu, M.E. Peters, M. Schmitz and L. Zettlemoyer, Al- eXplainable Artificial Intelligence: Foundations, Applications 49 49 lenNLP: A Deep Semantic Natural Language Processing and Challenges, I. Tiddi, F. Lécué and P. Hitzler, eds, Studies 50Platform, CoRR abs/1803.07640 (2018). http://arxiv.org/abs/ on the Semantic Web, Vol. 47, IOS Press, 2020, pp. 3–22. 50 511803.07640. doi:10.3233/SSW200009. 51 30 C. Möller et al. / Survey on English Entity Linking on Wikidata

1[51] S. Hochreiter and J. Schmidhuber, Long Short-Term Linguistics, 2020, pp. 6982–6993. https://www.aclweb.org/ 1 2Memory, Neural Comput. 9(8) (1997), 1735–1780. anthology/2020.acl-main.624/. 2 3doi:10.1162/neco.1997.9.8.1735. [64] N. Kolitsas, O. Ganea and T. Hofmann, End-to-End Neu- 3 [52] J. Hoffart, Y. Altun and G. Weikum, Discovering emerg- ral Entity Linking, in: Proceedings of the 22nd Conference 4 4 ing entities with ambiguous names, in: 23rd International on Computational Natural Language Learning, CoNLL 2018, 5World Wide Web Conference, WWW ’14, Seoul, Repub- Brussels, Belgium, October 31 - November 1, 2018, A. Ko- 5 6lic of Korea, April 7-11, 2014, C. Chung, A.Z. Broder, rhonen and I. Titov, eds, Association for Computational Lin- 6 7K. Shim and T. Suel, eds, ACM, 2014, pp. 385–396. guistics, 2018, pp. 519–529. doi:10.18653/v1/k18-1050. 7 8doi:10.1145/2566486.2568003. [65] T. Kristanti and L. Romary, DeLFT and entity-fishing: 8 Tools for CLEF HIPE 2020 Shared Task, in: CLEF 2020- 9[53] J. Hoffart, M.A. Yosef, I. Bordino, H. Fürstenau, M. Pinkal, 9 M. Spaniol, B. Taneva, S. Thater and G. Weikum, Robust Dis- Conference and Labs of the Evaluation Forum, Vol. 2696, 10 10 ambiguation of Named Entities in Text, in: Proceedings of the CEUR, 2020a. 112011 Conference on Empirical Methods in Natural Language [66] T. Kristanti and L. Romary, DeLFT and entity-fishing: 11 12Processing, EMNLP 2011, 27-31 July 2011, John McIntyre Tools for CLEF HIPE 2020 Shared Task, in: CLEF 2020- 12 13Conference Centre, Edinburgh, UK, A meeting of SIGDAT, a Conference and Labs of the Evaluation Forum, Vol. 2696, 13 CEUR, 2020b. 14Special Interest Group of the ACL, ACL, 2011, pp. 782–792. 14 https://www.aclweb.org/anthology/D11-1072/. [67] K. Labusch and C. Neudecker, Named Entity Disambiguation 15 15 [54] B. Huang, H. Wang, T. Wang, Y. Liu and Y. Liu, Entity Link- and Linking on Historic Newspaper OCR with BERT (2020). 16ing for Short Text Using Structured Knowledge Graph via [68] J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, 16 17Multi-grained Text Matching, Amazon Science (2020). P.N. Mendes, S. Hellmann, M. Morsey, P. van Kleef, S. Auer 17 18[55] A. Ismayilov, D. Kontokostas, S. Auer, J. Lehmann and and C. Bizer, DBpedia - A large-scale, multilingual knowl- 18 edge base extracted from Wikipedia, Semantic Web 6(2) 19S. Hellmann, Wikidata through the eyes of DBpedia, Seman- 19 (2015), 167–195. doi:10.3233/SW-140134. 20tic Web 9(4) (2018), 493–503. doi:10.3233/SW-170277. 20 [56] H. Ji, X. Pan, B. Zhang, J. Nothman, J. Mayfield, P. Mc- [69] X. Lin and L. Chen, Canonicalization of Open Knowl- 21edge Bases with Side Information from the Source Text, in: 21 Namee and C. Costello, Overview of TAC-KBP2017 13 35th IEEE International Conference on Data Engineering, 22Languages Entity Discovery and Linking, in: Proceedings 22 ICDE 2019, Macao, China, April 8-11, 2019, IEEE, 2019, 23of the 2017 Text Analysis Conference, TAC 2017, Gaithers- 23 pp. 950–961. doi:10.1109/ICDE.2019.00089. 24burg, Maryland, USA, November 13-14, 2017, NIST, 2017. 24 [70] X. Lin, H. Li, H. Xin, Z. Li and L. Chen, KBPearl: A https://tac.nist.gov/publications/2017/additional.papers/ 25Knowledge Base Population System Supported by Joint En- 25 TAC2017.KBP_Entity_Discovery_and_Linking_overview. 26tity and Relation Linking, Proc. VLDB Endow. 13(7) (2020), 26 proceedings.pdf. 1035–1049. http://www.vldb.org/pvldb/vol13/p1035-lin.pdf. 27[57] S.M. Kazemi, R. Goel, K. Jain, I. Kobyzev, A. Sethi, 27 [71] Y. Lin, Z. Liu and M. Sun, Neural Relation Extraction with 28P. Forsyth and P. Poupart, Representation Learning for Dy- 28 Multi-lingual Attention, in: Proceedings of the 55th Annual 29namic Graphs: A Survey, J. Mach. Learn. Res. 21 (2020), 29 Meeting of the Association for Computational Linguistics, 70:1–70:73. http://jmlr.org/papers/v21/19-447.html. 30ACL 2017, Vancouver, Canada, July 30 - August 4, Vol- 30 [58] Kensho R&D group, Kensho Derived Wikimedia Dataset. 31ume 1: Long Papers, R. Barzilay and M. Kan, eds, As- 31 https://www.kaggle.com/kenshoresearch/kensho-derived- 32sociation for Computational Linguistics, 2017, pp. 34–43. 32 wikimedia-data. 33doi:10.18653/v1/P17-1004. 33 [59] B. Kitchenham, Procedures for performing systematic re- [72] X. Ling, S. Singh and D.S. Weld, Design Challenges 34 34 views, Keele, UK, Keele University 33(2004) (2004), 1–26. for Entity Linking, Trans. Assoc. Comput. Linguistics 3 35[60] M. Klang and P. Nugues, Named entity disambiguation in a (2015), 315–328. https://tacl2013.cs.columbia.edu/ojs/index. 35 36question answering system, in: TheFifth Swedish Language php/tacl/article/view/528. 36 Technology Conference (SLTC2014), 2014. 37[73] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, 37 [61] M. Klang and P. Nugues, Hedwig: A Named Entity Linker, 38M. Lewis, L. Zettlemoyer and V. Stoyanov, RoBERTa: 38 in: Proceedings of The 12th Language Resources and Evalua- A Robustly Optimized BERT Pretraining Approach, CoRR 39 39 tion Conference, LREC 2020, Marseille, France, May 11-16, abs/1907.11692 (2019). http://arxiv.org/abs/1907.11692. 402020, N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, [74] L. Logeswaran, M. Chang, K. Lee, K. Toutanova, J. De- 40 41T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, vlin and H. Lee, Zero-Shot Entity Linking by Reading Entity 41 42H. Mazo, A. Moreno, J. Odijk and S. Piperidis, eds, Euro- Descriptions, in: Proceedings of the 57th Conference of the 42 pean Language Resources Association, 2020, pp. 4501–4508. 43Association for Computational Linguistics, ACL 2019, Flo- 43 https://www.aclweb.org/anthology/2020.lrec-1.554/. rence, Italy, July 28- August 2, 2019, Volume 1: Long Pa- 44 44 [62] M. Klang, F. Dib and P. Nugues, Overview of the Ugglan En- pers, A. Korhonen, D.R. Traum and L. Màrquez, eds, Asso- 45tity Discovery and Linking System, CoRR abs/1903.05498 ciation for Computational Linguistics, 2019, pp. 3449–3460. 45 46(2019). http://arxiv.org/abs/1903.05498. doi:10.18653/v1/p19-1335. 46 47[63] J. Klie, R.E. de Castilho and I. Gurevych, From Zero to [75] M. Manske, Wikidata Stats, 2020-07-20. https://wikidata- 47 48Hero: Human-In-The-Loop Entity Linking in Low Resource todo.toolforge.org/stats.php. 48 Domains, in: Proceedings of the 58th Annual Meeting of [76] D.N. Milne and I.H. Witten, Learning to link with wikipedia, 49 49 the Association for Computational Linguistics, ACL 2020, in: Proceedings of the 17th ACM Conference on Informa- 50Online, July 5-10, 2020, D. Jurafsky, J. Chai, N. Schluter tion and Knowledge Management, CIKM 2008, Napa Val- 50 51and J.R. Tetreault, eds, Association for Computational ley, California, USA, October 26-30, 2008, J.G. Shana- 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 31

1han, S. Amer-Yahia, I. Manolescu, Y. Zhang, D.A. Evans, A. Moschitti, B. Pang and W. Daelemans, eds, ACL, 2014, 1 2A. Kolcz, K. Choi and A. Chowdhury, eds, ACM, 2008, pp. 1532–1543. doi:10.3115/v1/d14-1162. 2 3pp. 509–518. doi:10.1145/1458082.1458150. [85] D. Perkins, Separating the Signal from the Noise: Predict- 3 [77] D. Moussallem, R. Usbeck, M. Röder and A.N. Ngomo, ing the Correct Entities in Named-Entity Linking, Master’s 4 4 MAG: A Multilingual, Knowledge-base Agnostic and De- thesis, Uppsala University, 2020. http://uu.diva-portal.org/ 5terministic Entity Linking Approach, in: Proceedings of smash/record.jsf?pid=diva2:1437921&dswid=6063. 5 6the Knowledge Capture Conference, K-CAP 2017, Austin, [86] M.E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, 6 7TX, USA, December 4-6, 2017, Ó. Corcho, K. Janowicz, K. Lee and L. Zettlemoyer, Deep Contextualized Word 7 8G. Rizzo, I. Tiddi and D. Garijo, eds, ACM, 2017, pp. 9:1–9:8. Representations, in: Proceedings of the 2018 Conference 8 of the North American Chapter of the Association for 9doi:10.1145/3148011.3148024. 9 [78] D. Moussallem, R. Usbeck, M. Röder and A.N. Ngomo, En- Computational Linguistics: Human Language Technologies, 10 10 tity Linking in 40 Languages Using MAG, in: The Seman- NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1- 11tic Web: ESWC 2018 Satellite Events - ESWC 2018 Satel- 6, 2018, Volume 1 (Long Papers), M.A. Walker, H. Ji and 11 12lite Events, Heraklion, Crete, Greece, June 3-7, 2018, Re- A. Stent, eds, Association for Computational Linguistics, 12 13vised Selected Papers, A. Gangemi, A.L. Gentile, A.G. Nuz- 2018, pp. 2227–2237. doi:10.18653/v1/n18-1202. 13 [87] F. Petroni, A. Piktus, A. Fan, P. Lewis, M. Yazdani, N.D. Cao, 14zolese, S. Rudolph, M. Maleshkova, H. Paulheim, J.Z. Pan 14 and M. Alam, eds, Lecture Notes in Computer Science, J. Thorne, Y. Jernite, V. Plachouras, T. Rocktäschel and 15 15 Vol. 11155, Springer, 2018, pp. 176–181. doi:10.1007/978-3- S. Riedel, KILT: a Benchmark for Knowledge Intensive Lan- 16319-98192-5_33. guage Tasks, CoRR abs/2009.02252 (2020). https://arxiv.org/ 16 17[79] I.O. Mulang, K. Singh, A. Vyas, S. Shekarpour, A. Sakor, abs/2009.02252. 17 18M. Vidal, S. Auer and J. Lehmann, Context-aware Entity [88] V. Provatorova, S. Vakulenko, E. Kanoulas, K. Dercksen and 18 J.M. van Hulst, Named Entity Recognition and Linking on 19Linking with Attentive Neural Networks on Wikidata Knowl- 19 Historical Newspapers: UvA.ILPS & REL at CLEF HIPE 20edge Graph, CoRR abs/1912.06214 (2019). http://arxiv.org/ 20 abs/1912.06214. 2020 (2020). 21[89] J. Raiman and O. Raiman, DeepType: Multilingual En- 21 [80] I.O. Mulang, K. Singh, A. Vyas, S. Shekarpour, M. Vidal tity Linking by Neural Type System Evolution, in: Pro- 22and S. Auer, Encoding Knowledge Graph Entity Aliases in 22 ceedings of the Thirty-Second AAAI Conference on Artifi- 23Attentive Neural Network for Wikidata Entity Linking, in: 23 cial Intelligence, (AAAI-18), the 30th innovative Applica- 24Web Information Systems Engineering - WISE 2020 - 21st In- 24 tions of Artificial Intelligence (IAAI-18), and the 8th AAAI ternational Conference, Amsterdam, The Netherlands, Octo- 25Symposium on Educational Advances in Artificial Intelli- 25 ber 20-24, 2020, Proceedings, Part I, Z. Huang, W. Beek, 26gence (EAAI-18), New Orleans, Louisiana, USA, February 26 H. Wang, R. Zhou and Y. Zhang, eds, Lecture Notes in Com- 2-7, 2018, S.A. McIlraith and K.Q. Weinberger, eds, AAAI 27puter Science, Vol. 12342, Springer, 2020a, pp. 328–342. 27 Press, 2018, pp. 5406–5413. https://www.aaai.org/ocs/index. 28doi:10.1007/978-3-030-62005-9_24. https://doi.org/10.1007/ 28 php/AAAI/AAAI18/paper/view/17148. 29978-3-030-62005-9_24. 29 [90] L. Ratinov, D. Roth, D. Downey and M. Anderson, Local and [81] I.O. Mulang, K. Singh, C. Prabhu, A. Nadgeri, J. Hof- 30Global Algorithms for Disambiguation to Wikipedia, in: The 30 31fart and J. Lehmann, Evaluating the Impact of Knowledge 49th Annual Meeting of the Association for Computational 31 Graph Context on Entity Disambiguation Models, in: CIKM 32Linguistics: Human Language Technologies, Proceedings of 32 ’20: The 29th ACM International Conference on Information 33the Conference, 19-24 June, 2011, Portland, Oregon, USA, 33 and Knowledge Management, Virtual Event, Ireland, Octo- D. Lin, Y. Matsumoto and R. Mihalcea, eds, The Association 34 34 ber 19-23, 2020, M. d’Aquin, S. Dietze, C. Hauff, E. Curry for Computer Linguistics, 2011, pp. 1375–1384. https://www. 35and P. Cudré-Mauroux, eds, ACM, 2020b, pp. 2157–2160. aclweb.org/anthology/P11-1138/. 35 36doi:10.1145/3340531.3412159. [91] D. Ringler and H. Paulheim, One Knowledge Graph to Rule 36 [82] K. Noullet, R. Mix and M. Färber, KORE 50DYWC: An 37Them All? Analyzing the Differences Between DBpedia, 37 Evaluation Data Set for Entity Linking Based on DBpe- 38YAGO, Wikidata & co, in: KI 2017: Advances in Artifi- 38 dia, YAGO, Wikidata, and Crunchbase, in: Proceedings of cial Intelligence - 40th Annual German Conference on AI, 39 39 The 12th Language Resources and Evaluation Conference, Dortmund, Germany, September 25-29, 2017, Proceedings, 40LREC 2020, Marseille, France, May 11-16, 2020, N. Calzo- G. Kern-Isberner, J. Fürnkranz and M. Thimm, eds, Lec- 40 41lari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, ture Notes in Computer Science, Vol. 10505, Springer, 2017, 41 42S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, pp. 366–372. doi:10.1007/978-3-319-67190-1_33. https:// 42 A. Moreno, J. Odijk and S. Piperidis, eds, European Lan- 43doi.org/10.1007/978-3-319-67190-1_33. 43 guage Resources Association, 2020, pp. 2389–2395. https: [92] M. Röder, R. Usbeck and A.N. Ngomo, GERBIL - Bench- 44 44 //www.aclweb.org/anthology/2020.lrec-1.291/. marking Named Entity Recognition and Linking consis- 45[83] I.L. Oliveira, R. Fileto, R. Speck, L.P. Garcia, D. Moussallem tently, Semantic Web 9(5) (2018), 605–625. doi:10.3233/SW- 45 46and J. Lehmann, Towards holistic Entity Linking: Survey and 170286. 46 47directions, Information Systems (2020), 101624. [93] M. Röder, R. Usbeck, S. Hellmann, D. Gerber and A. Both, 47 3 48[84] J. Pennington, R. Socher and C.D. Manning, Glove: Global N - A Collection of Datasets for Named Entity Recogni- 48 Vectors for Word Representation, in: Proceedings of the 2014 tion and Disambiguation in the NLP Interchange Format, in: 49 49 Conference on Empirical Methods in Natural Language Pro- Proceedings of the Ninth International Conference on Lan- 50cessing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, guage Resources and Evaluation, LREC 2014, Reykjavik, Ice- 50 51A meeting of SIGDAT, a Special Interest Group of the ACL, land, May 26-31, 2014, N. Calzolari, K. Choukri, T. De- 51 32 C. Möller et al. / Survey on English Entity Linking on Wikidata

1clerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, [103] D. Sorokin and I. Gurevych, Context-Aware Representa- 1 2J. Odijk and S. Piperidis, eds, European Language Resources tions for Knowledge Base Relation Extraction, in: Pro- 2 3Association (ELRA), 2014, pp. 3529–3533. http://www.lrec- ceedings of the 2017 Conference on Empirical Methods in 3 conf.org/proceedings/lrec2014/summaries/856.html. Natural Language Processing, EMNLP 2017, Copenhagen, 4 4 [94] H. Rosales-Méndez, A. Hogan and B. Poblete, VoxEL: A Denmark, September 9-11, 2017, M. Palmer, R. Hwa and 5Benchmark Dataset for Multilingual Entity Linking, in: The S. Riedel, eds, Association for Computational Linguistics, 5 6Semantic Web - ISWC 2018 - 17th International Seman- 2017, pp. 1784–1789. doi:10.18653/v1/d17-1188. 6 7tic Web Conference, Monterey, CA, USA, October 8-12, [104] D. Sorokin and I. Gurevych, Mixing Context Granularities 7 82018, Proceedings, Part II, D. Vrandecic, K. Bontcheva, for Improved Entity Linking on Question Answering Data 8 across Entity Categories, in: Proceedings of the Seventh 9M.C. Suárez-Figueroa, V. Presutti, I. Celino, M. Sabou, 9 L. Kaffee and E. Simperl, eds, Lecture Notes in Com- Joint Conference on Lexical and Computational Semantics, 10 10 puter Science, Vol. 11137, Springer, 2018, pp. 170–186. *SEM@NAACL-HLT 2018, New Orleans, Louisiana, USA, 11doi:10.1007/978-3-030-00668-6_11. https://doi.org/10.1007/ June 5-6, 2018, M. Nissim, J. Berant and A. Lenci, eds, As- 11 12978-3-030-00668-6_11. sociation for Computational Linguistics, 2018, pp. 65–75. 12 13[95] H. Rosales-Méndez, A. Hogan and B. Poblete, Fine-Grained doi:10.18653/v1/s18-2007. 13 [105] A. Sperduti and A. Starita, Supervised neural networks for 14Entity Linking, Journal of Web Semantics (2020), 100600. 14 [96] P. Rosso, D. Yang and P. Cudré-Mauroux, Beyond Triplets: the classiﬁcation of structures, IEEE Trans. Neural Networks 15 15 Hyper-Relational Knowledge Graph Embedding for Link 8(3) (1997), 714–735. doi:10.1109/72.572108. 16Prediction, in: WWW ’20: The Web Conference 2020, Taipei, [106] A. Spitz, J. Geiß and M. Gertz, So far away and yet so 16 17Taiwan, April 20-24, 2020, Y. Huang, I. King, T. Liu and close: augmenting toponym disambiguation and similarity 17 18M. van Steen, eds, ACM / IW3C2, 2020, pp. 1885–1896. with text-based networks, in: Proceedings of the Third Inter- 18 national ACM SIGMOD Workshop on Managing and Min- 19doi:10.1145/3366423.3380257. 19 [97] A. Sakor, I.O. Mulang, K. Singh, S. Shekarpour, M. Vidal, ing Enriched Geo-Spatial Data, GeoRich@SIGMOD 2016, 20San Francisco, California, USA, June 26 - July 1, 2016 20 J. Lehmann and S. Auer, Old is Gold: Linguistic Driven Ap- , 21A. Züﬂe, B. Adams and D. Wu, eds, ACM, 2016, pp. 2:1–2:6. 21 proach for Entity and Relation Linking of Short Text, in: doi:10.1145/2948649.2948651. 22Proceedings of the 2019 Conference of the North American 22 [107] Y. Su, H. Sun, B.M. Sadler, M. Srivatsa, I. Gur, Z. Yan and 23Chapter of the Association for Computational Linguistics: 23 X. Yan, On Generating Characteristic-rich Question Sets for 24Human Language Technologies, NAACL-HLT 2019, Min- 24 QA Evaluation, in: Proceedings of the 2016 Conference on neapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and 25Empirical Methods in Natural Language Processing, EMNLP 25 Short Papers), J. Burstein, C. Doran and T. Solorio, eds, Asso- 262016, Austin, Texas, USA, November 1-4, 2016, J. Su, X. Car- 26 ciation for Computational Linguistics, 2019, pp. 2336–2346. reras and K. Duh, eds, The Association for Computational 27doi:10.18653/v1/n19-1243. 27 Linguistics, 2016, pp. 562–572. doi:10.18653/v1/d16-1054. 28[98] A. Sakor, K. Singh, A. Patel and M. Vidal, Falcon 2.0: An 28 [108] T.P. Tanon, G. Weikum and F.M. Suchanek, YAGO 4: A 29Entity and Relation Linking Tool over Wikidata, in: CIKM 29 Reason-able Knowledge Base, in: The Semantic Web - 17th ’20: The 29th ACM International Conference on Information 30International Conference, ESWC 2020, Heraklion, Crete, 30 and Knowledge Management, Virtual Event, Ireland, Octo- 31Greece, May 31-June 4, 2020, Proceedings, A. Harth, S. Kir- 31 ber 19-23, 2020, M. d’Aquin, S. Dietze, C. Hauff, E. Curry 32rane, A.N. Ngomo, H. Paulheim, A. Rula, A.L. Gentile, 32 and P. Cudré-Mauroux, eds, ACM, 2020, pp. 3141–3148. 33P. Haase and M. Cochez, eds, Lecture Notes in Com- 33 doi:10.1145/3340531.3412777. puter Science, Vol. 12123, Springer, 2020, pp. 583–596. 34 34 [99] Ö. Sevgili, A. Panchenko and C. Biemann, Improving Neu- doi:10.1007/978-3-030-49461-2_34. https://doi.org/10.1007/ 35ral Entity Disambiguation with Graph Embeddings, in: Pro- 978-3-030-49461-2_34. 35 36ceedings of the 57th Conference of the Association for Com- [109] K. Teru, E. Denis and W. Hamilton, Inductive Relation Pre- 36 putational Linguistics, ACL 2019, Florence, Italy, July 28 37diction by Subgraph Reasoning, in: Proceedings of the 37th 37 - August 2, 2019, Volume 2: Student Research Workshop, 38International Conference on Machine Learning, ICML 2020, 38 F.E. Alva-Manchego, E. Choi and D. Khashabi, eds, Asso- 13-18 July 2020, Virtual Event, Proceedings of Machine 39 39 ciation for Computational Linguistics, 2019, pp. 315–322. Learning Research, Vol. 119, PMLR, 2020, pp. 9448–9457. 40doi:10.18653/v1/p19-2044. http://proceedings.mlr.press/v119/teru20a.html. 40 41[100] Ö. Sevgili, A. Shelmanov, M. Arkhipov, A. Panchenko and [110] A. Thawani, M. Hu, E. Hu, H. Zafar, N.T. Divvala, A. Singh, 41 42C. Biemann, Neural Entity Linking: A Survey of Mod- E. Qasemi, P.A. Szekely and J. Pujara, Entity Linking to 42 els based on Deep Learning, CoRR abs/2006.00575 (2020). 43Knowledge Graphs to Infer Column Types and Properties, in: 43 https://arxiv.org/abs/2006.00575. Proceedings of the Semantic Web Challenge on Tabular Data 44 44 [101] F. Shanaz and R.G. Ragel, Wikidata based Location Entity to Knowledge Graph Matching co-located with the 18th In- 45Linking, in: Proceedings of the 9th International Confer- ternational Semantic Web Conference, SemTab@ISWC 2019, 45 46ence on Software and Computer Applications, ICSCA 2020, Auckland, New Zealand, October 30, 2019, E. Jiménez-Ruiz, 46 47Langkawi, Malaysia, February 18-21, 2020, ACM, 2020, O. Hassanzadeh, K. Srinivas, V. Efthymiou and J. Chen, eds, 47 pp. 307–312. doi:10.1145/3384544.3384592. 48CEUR Workshop Proceedings, Vol. 2553, CEUR-WS.org, 48 [102] W. Shen, J. Wang and J. Han, Entity Linking with 2019, pp. 25–32. http://ceur-ws.org/Vol-2553/paper4.pdf. 49 49 a Knowledge Base: Issues, Techniques, and Solutions, [111] E.F. Tjong Kim Sang and F. De Meulder, Introduction to the 50IEEE Trans. Knowl. Data Eng. 27(2) (2015), 443–460. CoNLL-2003 Shared Task: Language-Independent Named 50 51doi:10.1109/TKDE.2014.2327028. Entity Recognition, in: Proceedings of the Seventh Confer- 51 C. Möller et al. / Survey on English Entity Linking on Wikidata 33

1ence on Natural Language Learning at HLT-NAACL 2003, [121] W3C Microposts Community Group, Making Sense of Mi- 1 22003, pp. 142–147. https://www.aclweb.org/anthology/W03- croposts (#Microposts2016), 2016. http://microposts2016. 2 30419. seas.upenn.edu/challenge.html. 3 [112] R. Usbeck, A.N. Ngomo, M. Röder, D. Gerber, S.A. Coelho, [122] P. Wang, J. Han, C. Li and R. Pan, Logic Attention Based 4 4 S. Auer and A. Both, AGDISTIS - Agnostic Disambiguation Neighborhood Aggregation for Inductive Knowledge Graph 5of Named Entities Using Linked Open Data, in: ECAI 2014 Embedding, in: The Thirty-Third AAAI Conference on Ar- 5 6- 21st European Conference on Artificial Intelligence, 18-22 tificial Intelligence, AAAI 2019, The Thirty-First Innova- 6 7August 2014, Prague, Czech Republic - Including Prestigious tive Applications of Artificial Intelligence Conference, IAAI 7 8Applications of Intelligent Systems (PAIS 2014), T. Schaub, 2019, The Ninth AAAI Symposium on Educational Advances 8 G. Friedrich and B. O’Sullivan, eds, Frontiers in Artificial in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, 9 9 Intelligence and Applications, Vol. 263, IOS Press, 2014, USA, January 27 - February 1, 2019, AAAI Press, 2019a, 10pp. 1113–1114. doi:10.3233/978-1-61499-419-0-1113. pp. 7152–7159. doi:10.1609/aaai.v33i01.33017152. 10 11[113] R. Usbeck, A.N. Ngomo, B. Haarmann, A. Krithara, [123] X. Wang, T. Gao, Z. Zhu, Z. Liu, J. Li and J. Tang, KEPLER: 11 12M. Röder and G. Napolitano, 7th Open Challenge on Ques- A Unified Model for Knowledge Embedding and Pre-trained 12 13tion Answering over Linked Data (QALD-7), in: Semantic Language Representation, CoRR abs/1911.06136 (2019b). 13 Web Challenges - 4th SemWebEval Challenge at ESWC 2017, http://arxiv.org/abs/1911.06136. 14 14 Portoroz, Slovenia, May 28 - June 1, 2017, Revised Selected [124] G. Weikum, L. Dong, S. Razniewski and F.M. Suchanek, 15Papers, M. Dragoni, M. Solanki and E. Blomqvist, eds, Com- Machine Knowledge: Creation and Curation of Comprehen- 15 16munications in Computer and Information Science, Vol. 769, sive Knowledge Bases, CoRR abs/2009.11564 (2020). https: 16 17Springer, 2017, pp. 59–69. doi:10.1007/978-3-319-69146-6_- //arxiv.org/abs/2009.11564. 17 186. https://doi.org/10.1007/978-3-319-69146-6_6. [125] Wikimedia Foundation, Index of /wikidatawiki/entities/, 18 [114] T. van Veen, J. Lonij and W.J. Faber, Linking Named Enti- 2020-08-21. https://dumps.wikimedia.org/wikidatawiki/ 19 19 ties in Dutch Historical Newspapers, in: Metadata and Se- entities/. 20mantics Research - 10th International Conference, MTSR [126] Wikimedia Foundation, Wikistats, 2020-10-09. https://stats. 20 212016, Göttingen, Germany, November 22-25, 2016, Proceed- wikimedia.org/#/metrics/wikidata.org. 21 22ings, E. Garoufallou, I.S. Coll, A. Stellato and J. Greenberg, [127] L. Wu, F. Petroni, M. Josifoski, S. Riedel and L. Zettlemoyer, 22 23eds, Communications in Computer and Information Science, Zero-shot Entity Linking with Dense Entity Retrieval, CoRR 23 Vol. 672, 2016, pp. 205–210. doi:10.1007/978-3-319-49157- abs/1911.03814 (2019a). http://arxiv.org/abs/1911.03814. 24 24 8_18. https://doi.org/10.1007/978-3-319-49157-8_18. [128] T. Wu, A. Khan, H. Gao and C. Li, Efficiently Embedding Dy- 25[115] S. Vashishth, P. Jain and P.P. Talukdar, CESI: Canonical- namic Knowledge Graphs, CoRR abs/1910.06708 (2019b). 25 26izing Open Knowledge Bases using Embeddings and Side http://arxiv.org/abs/1910.06708. 26 27Information, in: Proceedings of the 2018 World Wide Web [129] Z. Wu, S. Pan, F. Chen, G. Long, C. Zhang and S.Y. Philip, A 27 28Conference on World Wide Web, WWW 2018, Lyon, France, comprehensive survey on graph neural networks, IEEE Trans- 28 April 23-27, 2018, P. Champin, F.L. Gandon, M. Lal- actions on Neural Networks and Learning Systems (2020). 29 29 mas and P.G. Ipeirotis, eds, ACM, 2018, pp. 1317–1327. [130] X. Yang, X. Gu, S. Lin, S. Tang, Y. Zhuang, F. Wu, Z. Chen, 30doi:10.1145/3178876.3186030. G. Hu and X. Ren, Learning Dynamic Context Augmenta- 30 31[116] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, tion for Global Entity Linking, in: Proceedings of the 2019 31 32A.N. Gomez, L. Kaiser and I. Polosukhin, Attention is All Conference on Empirical Methods in Natural Language Pro- 32 33you Need, in: Advances in Neural Information Processing cessing and the 9th International Joint Conference on Natural 33 Systems 30: Annual Conference on Neural Information Pro- Language Processing, EMNLP-IJCNLP 2019, Hong Kong, 34 34 cessing Systems 2017, December 4-9, 2017, Long Beach, CA, China, November 3-7, 2019, K. Inui, J. Jiang, V. Ng and 35USA, I. Guyon, U. von Luxburg, S. Bengio, H.M. Wallach, X. Wan, eds, Association for Computational Linguistics, 35 36R. Fergus, S.V.N. Vishwanathan and R. Garnett, eds, 2017, 2019, pp. 271–281. doi:10.18653/v1/D19-1026. 36 37pp. 5998–6008. http://papers.nips.cc/paper/7181-attention-is- [131] Y. Yang and M. Chang, S-MART: Novel Tree-based Struc- 37 38all-you-need. tured Learning Algorithms Applied to Tweet Entity Link- 38 [117] O. Cerbaˇ Ing and M. Otakar Cerba,ˇ 7th International Confer- ing, CoRR abs/1609.08075 (2016). http://arxiv.org/abs/1609. 39 39 ence on Cartography and GIS, Technical Report, 29, 2018. 08075. 40ISSN 1848-0713. http://id.loc.gov/authorities. [132] Z. Yang, Z. Dai, Y. Yang, J.G. Carbonell, R. Salakhutdinov 40 41[118] O. Vinyals, M. Fortunato and N. Jaitly, Pointer Networks, in: and Q.V. Le, XLNet: Generalized Autoregressive Pretrain- 41 42Advances in Neural Information Processing Systems 28: An- ing for Language Understanding, in: Advances in Neural 42 43nual Conference on Neural Information Processing Systems Information Processing Systems 32: Annual Conference 43 2015, December 7-12, 2015, Montreal, Quebec, Canada, on Neural Information Processing Systems 2019, NeurIPS 44 44 C. Cortes, N.D. Lawrence, D.D. Lee, M. Sugiyama and 2019, 8-14 December 2019, Vancouver, BC, Canada, 45R. Garnett, eds, 2015, pp. 2692–2700. http://papers.nips.cc/ H.M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché- 45 46paper/5866-pointer-networks. Buc, E.B. Fox and R. Garnett, eds, 2019, pp. 5754–5764. 46 47[119] D. Vrandecic, Architecture for a multilingual Wikipedia, http://papers.nips.cc/paper/8812-xlnet-generalized- 47 48CoRR abs/2004.04733 (2020). https://arxiv.org/abs/2004. autoregressive-pretraining-for-language-understanding. 48 04733. [133] W. Yih, M. Richardson, C. Meek, M. Chang and J. Suh, The 49 49 [120] D. Vrandeciˇ c´ and M. Krötzsch, Wikidata: a free collabora- Value of Semantic Parse Labeling for Knowledge Base Ques- 50tive knowledgebase, Commun. ACM 57(10) (2014), 78–85. tion Answering, in: Proceedings of the 54th Annual Meet- 50 51doi:10.1145/2629489. ing of the Association for Computational Linguistics, ACL 51 34 C. Möller et al. / Survey on English Entity Linking on Wikidata

12016, August 7-12, 2016, Berlin, Germany, Volume 2: Short tics 8 (2020), 109–124. https://transacl.org/ojs/index.php/tacl/ 1 2Papers, The Association for Computer Linguistics, 2016. article/view/1906. 2 3doi:10.18653/v1/p16-2033. [136] S. Zwicklbauer, C. Seifert and M. Granitzer, DoSeR - 3 [134] F. Yu and V. Koltun, Multi-Scale Context Aggregation by A Knowledge-Base-Agnostic Framework for Entity Disam- 4 4 Dilated Convolutions, in: 4th International Conference on biguation Using Semantic Embeddings, in: The Semantic 5Learning Representations, ICLR 2016, San Juan, Puerto Web. Latest Advances and New Domains - 13th Interna- 5 6Rico, May 2-4, 2016, Conference Track Proceedings, Y. Ben- tional Conference, ESWC 2016, Heraklion, Crete, Greece, 6 7gio and Y. LeCun, eds, 2016. http://arxiv.org/abs/1511. May 29 - June 2, 2016, Proceedings, H. Sack, E. Blomqvist, 7 807122. M. d’Aquin, C. Ghidini, S.P. Ponzetto and C. Lange, eds, Lec- 8 [135] S. Zhou, S. Rijhwani, J. Wieting, J.G. Carbonell and G. Neu- ture Notes in Computer Science, Vol. 9678, Springer, 2016, 9 9 big, Improving Candidate Generation for Low-resource pp. 182–198. doi:10.1007/978-3-319-34129-3_12. 10Cross-lingual Entity Linking, Trans. Assoc. Comput. Linguis- [137] entity-ﬁshing, GitHub, 2016–2020. 10 11 11 12 12 13 13 14 14 15 15 16 16 17 17 18 18 19 19 20 20 21 21 22 22 23 23 24 24 25 25 26 26 27 27 28 28 29 29 30 30 31 31 32 32 33 33 34 34 35 35 36 36 37 37 38 38 39 39 40 40 41 41 42 42 43 43 44 44 45 45 46 46 47 47 48 48 49 49 50 50 51 51