Semantic Web 0 (2017) 1–0 1 IOS Press
Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Editor(s): Amrapali Zaveri, University of Leipzig Solicited review(s): Zhigang Wang, Beijing Normal University, China; Anonymous; Sebastian Mellor, Newcastle University, U.K.
Michael Färber ∗,∗∗, Frederic Bartscherer, Carsten Menne, and Achim Rettinger ∗∗∗ Karlsruhe Institute of Technology (KIT), Institute AIFB, 76131 Karlsruhe, Germany
Abstract. In recent years, several noteworthy large, cross-domain, and openly available knowledge graphs (KGs) have been created. These include DBpedia, Freebase, OpenCyc, Wikidata, and YAGO. Although extensively in use, these KGs have not been subject to an in-depth comparison so far. In this survey, we provide data quality criteria according to which KGs can be analyzed and analyze and compare the above mentioned KGs. Furthermore, we propose a framework for finding the most suitable KG for a given setting.
Keywords: Knowledge Graph, Linked Data Quality, Data Quality Metrics, Comparison, DBpedia, Freebase, OpenCyc, Wikidata, YAGO
1. Introduction The idea of a Semantic Web was introduced to a wider audience by Berners-Lee in 2001 [10]. Accord- The vision of the Semantic Web is to publish and ing to his vision, the traditional Web as a Web of Docu- query knowledge on the Web in a semantically struc- ments should be extended to a Web of Data where not tured way. According to Guns [23], the term “Seman- only documents and links between documents, but any tic Web” had already been used in fields such as Ed- entity (e.g., a person or organization) and any relation ucational Psychology, before it became prominent in between entities (e.g., isSpouseOf ) can be represented Computer Science. Freedman and Reynolds [21], for on the Web. instance, describe “semantic webbing” as organizing in- When it comes to realizing the idea of the Semantic formation and relationships in a visual display. Berners- Web, knowledge graphs (KGs) are currently seen as one Lee has mentioned his idea of using typed links as ve- of the most essential components. The term "knowledge hicle of semantics already since 1989 and proposed it graph" was reintroduced by Google in 2012 [42] and under the term Semantic Web for the first time at the is intended for any graph-based knowledge repository. INET conference in 1995 [23]. Since in the Semantic Web RDF graphs are used we use the term knowledge graph for any RDF graph. An RDF graph consists of a finite set of RDF triples *Corresponding author. E-mail: [email protected]. **This work was carried out with the support of the German Fed- where each RDF triple (s, p, o) is an ordered set of the eral Ministry of Education and Research (BMBF) within the Software following RDF terms: a subject s ∈ U ∪ B, a predicate Campus project SUITE (Grant 01IS12051). p ∈ U, and an object o ∈ U ∪ B ∪ L. An RDF term *** The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7/2007- is either a URI u ∈ U, a blank node b ∈ B, or a 2013) under grant agreement no. 611346. literal l ∈ L. U, B, and L are infinite sets and pairwise
1570-0844/17/$27.50 c 2017 – IOS Press and the authors. All rights reserved 2 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO disjoint. We denote the system that hosts a KG g with lected DBpedia, Freebase, OpenCyc, Wikidata, and hg. YAGO as KGs for our comparison. In this survey, we focus on those KGs having the In this paper, we give a systematic overview of these following aspects: KGs in their current versions (as of April 2016) and discuss how the knowledge in these KGs is modeled, 1. The KGs are freely accessible and freely usable stored, and queried. To the best of our knowledge, such within the Linked Open Data (LOD) cloud. a comparison between these widely used KGs has not Linked Data refers to a set of best practices1 for been presented before. Note that the focus of this survey publishing and interlinking structured data on the is not the life cycle of KGs on the Web or in enterprises. Web, defined by Berners-Lee [8] in 2006. Linked We can refer in this respect to [5]. Instead, the focus of Open Data refers to the Linked Data which "can our KG comparison is on data quality, as this is one of be freely used, modified, and shared by anyone for the most crucial aspects when it comes to considering any purpose."2 The aim of the Linking Open Data which KG to use in a specific setting. community project3 is to publish RDF datasets on Furthermore, we provide a KG recommendation the Web and to interlink these datasets. 2. The KGs should cover general knowledge (often framework for users who are interested in using one of also called cross-domain or encyclopedic knowl- the mentioned KGs in a research or industrial setting, edge) instead of knowledge about special domains but who are inexperienced in which KG to choose for such as biomedicine. their concrete settings. The main contributions of this survey are: Thus, out of scope are KGs which are not openly available such as the Google Knowledge Graph4 and 1. Based on existing literature on data quality, we the Google Knowledge Vault [13]. Excluded are also provide 34 data quality criteria according to which KGs which are only accessible via an API, but which KGs can be analyzed. are not provided as dump files (see WolframAlpha5 2. We calculate key statistics for the KGs DBpedia, and the Facebook Graph6) as well as KGs which are Freebase, OpenCyc, Wikidata, and YAGO. not based on Semantic Web standards at all or which 3. We analyze DBpedia, Freebase, OpenCyc, Wiki- data, and YAGO along the mentioned data quality are only unstructured or weakly structured knowledge 9 collections (e.g., The World Factbook of the CIA7). criteria. For selecting the KGs for analysis, we regarded 4. We propose a framework which enables users to all datasets which had been registered at the online find the most suitable KG for their needs. dataset catalog http://datahub.io8 and which The survey is organized as follows: were tagged as “crossdomain”. Besides that, we took Wikidata into consideration, since it also fulfilled the – In Section 2 we introduce formal definitions used above mentioned requirements. Based on that, we se- throughout the article. – In Section 3 we describe the data quality dimen- sions which we later use for the KG comparison, 1 See http://www.w3.org/TR/ld-bp/, requested on April including their subordinated data quality criteria 5, 2016. 2See http://opendefinition.org/, requested on Apr 5, and corresponding data quality metrics. 2016. – In Section 4 we describe the selected KGs. 3See http://www.w3.org/wiki/SweoIG/ – In Section 5 we analyze the KGs using several TaskForces/CommunityProjects/LinkingOpenData, key statistics and using the data quality metrics requested on Apr 5, 2016. introduced in Section 3. 4See http://www.google.com/insidesearch/ features/search/knowledge.html, requested on Apr 3, – In Section 6 we present our framework for assess- 2016. ing and rating KGs according to the user’s setting. 5See http://products.wolframalpha.com/api/, re- – In Section 7 we present related work on (linked) quested on Aug 30, 2016. data quality criteria and on key statistics for KGs. 6 See https://developers.facebook.com/docs/ – In Section 8 we conclude the survey. graph-api, requested on Aug 30, 2016. 7See https://www.cia.gov/library/ publications/the-world-factbook/, requested on Aug 9The data and detailed evaluation results for both the 30, 2016 key statistics and the metric evaluations are online avail- 8This catalog is also used for registering Linked Open Data able at http://km.aifb.kit.edu/sites/knowledge- datasets. graph-comparison/ (requested on Jan 31, 2017). M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 3
imp 2. Important Definitions We also call them predicates. Pg denotes the set of all implicitly defined relations in g: imp We define the following sets that are used in formal- Pg := {p | (s, p, o) ∈ g} izations throughout the article. If not otherwise stated, – Ug denotes the set of all URIs used in g: we use the prefixes listed in Listing 1 for indicating Ug := {x | ((x, p, o) ∈ g ∨ (s, x, o) ∈ g ∨ namespaces throughout the article. (s, p, x) ∈ g) ∧ x ∈ U} local – Ug denotes the set of all URIs in g with local – Cg denotes the set of classes in g: namespace; i.e., those URIs start with the KG g Cg := {x | (x, rdfs:subClassOf, o) ∈ dedicated prefix (cf. Listing 1). ext g ∨ (s, rdfs:subClassOf, x) ∈ g ∨ (x, – Complementary, Ug consists of all URIs in Ug wdt:P279, o) ∈ g ∨ (s, wdt:P279, x) ∈ g ∨ which are external to the KG g which means that (x, rdf:type, rdfs:Class) ∈ g} hg is not responsible for resolving those URIs. – An instance of a class is a resource which is mem- ber of that class. This membership is given by a Note that knowledge about the KGs which were ana- 10 lyzed for this survey was taken into account when defin- corresponding instantiation assignment. Ig de- notes the set of instances in g: ing these sets. These definitions may not be appropriate I := {s | (s, rdf:type, o) ∈ g ∨ (s, wdt:P31, for other KGs. g Furthermore, the sets’ extensions would be different o) ∈ g} when assuming a certain semantic (e.g., RDF, RDFS, or – Entities are defined as instances which represent OWL-LD). Under the assumption that all entailments real world objects. E denotes the set of entities g under one of these semantics were added to a KG, the in g: definition of each set could be simplified and the exten- E := {s | (s, rdf:type, owl:Thing) ∈ g sions would be of larger cardinality. However, for this g ∨ (s, rdf:type, wdo:Item) ∈ g ∨ article we did not derive entailments. (s, rdf:type, freebase:common.topic) ∈ g ∨ (s, rdf:type, cych:Individual) ∈ g} 3. Data Quality Assessment w.r.t. KGs – Relations (interchangeably used with "proper- ties") are links between RDF terms11 defined on Everybody on the Web can publish information. the schema level (i.e., T-Box). To emphasize this Therefore, a data consumer does not only face the chal- characterization, we also call them explicitly de- lenge to find a suitable data source, but is also con- fined relations. Pg denotes the set of all those fronted with the issue that data on the Web can dif- relations in g: fer very much regarding its quality. Data quality can Pg := {s | (s, rdf:type, rdf:Property) ∈ thereby be viewed not only in terms of accuracy, but in g ∨ (s, rdf:type, rdfs:Property) multiple other dimensions. In the following, we intro- ∈ g ∨ (s, rdf:type, wdo:Property) ∈ duce concepts regarding the data quality of KGs in the g ∨ (s, rdf:type, owl:Functional Linked Data context, which are used in the following Property) ∈ g ∨ (s, rdf:type, owl: sections. The data quality dimensions are then exposed InverseFunctionalProperty) ∈ g ∨ in Sections 3.2 – 3.5. (s, rdf:type, owl:DatatypeProperty) ∈ Data quality (DQ) – in the following interchange- g ∨ (s, rdf:type, owl:Object ably used with information quality12 – is defined by Property) ∈ g ∨ (s, rdf:type, owl: Juran et al. [32] as fitness for use. This means that data SymmetricProperty) ∈ g ∨ quality is dependent on the actual use case. (s, rdf:type, owl:TransitiveProperty) One of the most important and foundational works on ∈ g} data quality is that of Wang et al. [47]. They developed – Implicitly defined relations embrace all links a framework for assessing the data quality of datasets used in the KG, i.e., on instance and schema level. in the database context. In this framework, Wang et al.
10See https://www.w3.org/TR/rdf-schema/, re- 12As soon as data is considered w.r.t. usefulness, the data is seen quested on Aug 29, 2016. in a specific context. It can, thus, already be regarded as information, 11RDF terms comprise URIs, blank nodes, and literals. leading to the term “information quality” instead of “data quality.” 4 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Listing 1: Default prefixes for namespaces used throughout this article. @prefix cc:
distinguish between data quality criteria, data quality criteria Syntactic validity of RDF documents, Syntactic dimensions, and data quality categories.13 In the follow- validity of literals and Semantic validity of triples form ing, we reuse these concepts for our own framework, the Accuracy dimension. which has the particular focus on the data quality of Data quality dimensions and their respective data KGs in the context of Linked Open Data. quality criteria are further grouped into data quality A data quality criterion (Wang et al. also call it categories. Based on empirical studies, Wang et al. “data quality attribute”) is a particular characteristic of specified four categories: data w.r.t. its quality and can be either subjective or category of the intrinsic data quality objective. An example of a subjectively measurable – Criteria of the data quality criterion is Trustworthiness on KG level. focus on the fact that data has quality in its own An example of an objective data quality criterion is the right. Syntactic validity of RDF documents (see Section 3.2 – Criteria of the category of the contextual data qual- and [46]). ity cannot be considered in general, but must be In order to measure the degree to which a certain assessed depending on the application context of data quality criterion is fulfilled for a given KG, each the data consumer. criterion is formalized and expressed in terms of a func- – Criteria of the category of the representational tion with the value range of [0, 1]. We call this function data quality reveal in which form the data is avail- the data quality metric of the respective data quality able. criterion. – Criteria of the category of the accessibility data A data quality dimension – in the following just quality determine how the data can be accessed. called dimension – is a main aspect how data quality Since its publication, the presented framework of can be viewed. A data quality dimension comprises one Wang et al. has been extensively used, either in its or several data quality criteria [47]. For instance, the original version or in an adapted or extended version. Bizer [11] and Zaveri [49] worked on data quality in the 13The quality dimensions are defined in [47], the sub-classification Linked Data context. They make the following adapta- into parameters/indicators in [46, p. 354]. tions on Wang et al.’s framework: M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 5
– Bizer [11] compared the work of Wang et al. [47] Based on the quality dimensions introduced by Wang with other works in the area of data quality. He et al. [47], we now present the DQ criteria and met- thereby complements the framework with the di- rics as used in our KG comparison. Note that some of mensions consistency, verifiability, and offensive- the criteria have already been introduced by others as ness. outlined in Section 7. – Zaveri et al. [49] follow Wang et al. [47], but intro- Note also that our metrics are to be understood as duce licensing and interlinking as new dimensions possible ways of how to evaluate the DQ dimensions. in the linked data context. Other definitions of the DQ metrics might be possible and reasonable. We defined the metrics along the char- In this article, we use the DQ dimensions as defined acteristics of the KGs DBpedia, Freebase, OpenCyc, by Wang et al. [47] and as extended by Bizer [11] and Wikidata, and YAGO, but kept the definitions as generic Zaveri [49]. More precisely, we make the following as possible. In the evaluations, we then used those met- adaptations on Wang et al.’s framework: ric definitions and applied them, e.g., on the basis of 1. Consistency is treated by us as separate DQ dimen- own-created gold standards. sion. 3.2. Intrinsic Category 2. Verifiability is incorporated within the DQ dimen- sion Trustworthiness as criterion Trustworthiness “Intrinsic data quality denotes that data have quality on statement level. in their own right.” [47] This kind of data quality can 3. The Offensiveness of KG facts is not considered therefore be assessed independently from the context. by us, as it is hard to make an objective evaluation The intrinsic category embraces the three dimensions in this regard. Accuracy, Trustworthiness, and Consistency, which are category of the accessibility data 4. We extend the defined in the following subsections. The dimensions quality License Interlinking by the dimension and , Believability, Objectivity, and Reputation, which are as those data quality dimensions get in addition separate dimensions in Wang et al.’s classification sys- relevant in the Linked Data context. tem [47], are subsumed by us under the dimension Trustworthiness. 3.1. Criteria Weighting 3.2.1. Accuracy When applying our framework to compare KGs, the Definition of dimension. Accuracy is “the extent to single DQ metrics can be weighted differently so that which data are correct, reliable, and certified free of the needs and requirements of the users can be taken error” [47]. into account. In the following, we first formalize the Discussion. Accuracy is intuitively an important di- idea of weighting the different metrics. We then present mension of data quality. Previous work on data quality the criteria and the corresponding metrics of our frame- has mainly analyzed only this aspect [47]. Hence, accu- work. racy has often been used as synonym for data quality Accuracy Given are a KG g, a set of criteria C = {c , ..., c }, a [39]. Bizer [11] highlights in this context that 1 n is an objective dimension and can only be applied on set of metrics M = {m , ..., m }, and a set of weights 1 n verifiable statements. W = {w , ..., w }. Each metric m corresponds to the 1 n i Batini et al. [6] distinguish between syntactic and criterion c and m (g) ∈ [0, 1] where a value of 0 de- i i semantic accuracy: Syntactic accuracy describes the fines the minimum fulfillment degree of a KG regarding formal compliance to syntactic rules without review- a quality criterion and a value of 1 the maximum fulfill- ing whether the value reflects the reality. The semantic ment degree. Furthermore, each criterion c is weighted i accuracy determines whether the value is semantically by w . i valid, i.e., whether the value is true. Based on the clas- The fulfillment degree h(g) ∈ [0, 1] of a KG g is sification of Batini et al., we can define the metric for then the weighted normalized sum of the fulfillment Accuracy as follows: degrees w.r.t. the criteria c , ..., c : 1 n Definition of metric. The dimension Accuracy is determined by the criteria Pn i=1 wi mi(g) – Syntactic validity of RDF documents, h(g) = Pn j=1 wj – Syntactic validity of literals, and 6 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
– Semantic validity of triples. is common sense, or if the statement can be measured or perceived by the user directly. Wikidata has similar The fulfillment degree of a KG g w.r.t. the dimen- guidelines implemented to determine whether a fact sion Accuracy is measured by the metrics m , synRDF needs to be sourced.15 m , and m , which are defined as fol- synLit semT riple We measure the Semantic validity of triples based on lows. empirical evidence, i.e., based on a reference data set Syntactic validity of RDF documents The syntactic serving as gold standard. We determine the fulfillment validity of RDF documents is an important require- degree as the precision that the triples which are in the ment for machines to interpret an RDF document com- KG g and in the gold standard GS have the same values. pletely and correctly. Hogan et al. [29] suggest using Note that this measurement is heavily depending on the standardized tools for creating RDF data. The authors truthfulness of the reference data set. state that in this way normally only little syntax errors Formally, let nog,GS = |{(s, p, o) | (s, p, o) ∈ g ∧ occur, despite the complex syntactic representation of (x, y, z) ∈ GS∧equi(s, x)∧equi(p, y)∧equi(o, z))}| RDF/XML. be the number of triples in g to which semantically RDF data can be validated by an RDF validator such corresponding triples in the gold standard GS exist. Let 14 as the W3C RDF validator. nog = |{(s, p, o) | (s, p, o) ∈ g ∧ (x, y, z) ∈ GS ∧ equi(s, x) ∧ equi(p, y)}| be the number of triples in g ( where the subject-relation-pairs (s, p) are semantically 1 if all RDF documents are valid equivalent to subject-relation-pairs (x, y) in the gold msynRDF (g) = 0 otherwise standard. Then we can state:
Syntactic validity of literals Assessing the syntactic nog,GS validity of literals means to determine to which degree msemT riple(g) = literal values stored in the KG are syntactically valid. nog The syntactic validity of literal values depends on the data types of the literals and can be automatically as- In case of an empty set in the denominator of the sessed via rules [22,34]. Syntactic rules can be writ- fraction, the metric should evaluate to 1. ten in the form of regular expressions. For instance, 3.2.2. Trustworthiness it can be verified whether a literal representing a date Definition of dimension. Trustworthiness is defined follows the ISO 8601 specification. Assuming that L is as "the degree to which the information is accepted to be the infinite set of literals, we can state: correct, true, real, and credible" [49]. We define it as a collective term for believability, reputation, objectivity, and verifiability. These aspects were defined by Wang |{(s, p, o) ∈ g | o ∈ L ∧ synV alid(o)}| m (g) = et al. [47] and Naumann [39] as follows: synLit |{(s, p, o) ∈ g | o ∈ L}| – Believability: Believability is “the extent to which In case of an empty set in the denominator of the data are accepted or regarded as true, real, and fraction, the metric should evaluate to 1. credible” [47]. Semantic validity of triples The criterion Semantic – Reputation: Reputation is “the extent to which validity of triples is introduced to evaluate whether the data are trusted or highly regarded in terms of their statements expressed by the triples (with or without source or content” [47]. literals) hold true. Determining whether a statement – Objectivity: Objectivity is “the extent to which is true or false is strictly speaking impossible (see the data are unbiased (unprejudiced) and impartial” field of epistemology in philosophy). For evaluating the [47]. Semantic validity of statements, Bizer et al. [11] note – Verifiability: Verifiability is “the degree and ease that a triple is semantically correct if it is also available with which the data can be checked for correctness” from a trusted source (e.g., Name Authority File), if it [39].
14See http://www.w3.org/RDF/Validator, requested 15See https://www.wikidata.org/wiki/Help: on Feb 29, 2016. Sources, requested on Sep 8, 2016. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 7
Discussion. In summary, believability considers the tem, where data is curated by a community. Therefore, subject (data consumer) side; reputation takes the gen- we assign the values of the metric for Trustworthiness eral, social view on trustworthiness; objectivity consid- on KG level as follows: ers the object (data provider) side, while verifiability focuses on the possibility of verification. Trustworthiness has been discussed as follows: 1 manual data curation, man- ual data insertion in a – Believability: According to Naumann [39], believ- closed system ability is the “expected accuracy” of a data source. 0.75 manual data curation and in- – Reputation: The essential difference of believ- sertion, both by a commu- ability to accuracy is that for believability, data is nity trusted without verification [11]. Thus, believabil- 0.5 manual data curation, data ity is closely related to the reputation of a dataset. insertion by community or – Objectivity: According to Naumann [39], the ob- jectivity of a data source is strongly related to the mgraph(hg) = data insertion by automated verifiability: The more verifiable a data source or knowledge extraction statement is, the more objective it is. The authors 0.25 automated data curation, of this article would not go so far, since also biased data insertion by automated statements could be verifiable. knowledge extraction from – Verifiability: Heath et al. [26] emphasize that it is structured data sources 0 automated data curation, essential for trustworthy applications to be able to data insertion by automated verify the origin of data. knowledge extraction from Definition of metric. We define the metric for the unstructured data sources data quality dimension Trustworthiness as a combina- tion of trustworthiness metrics on both KG and state- Note that all proposed DQ metrics should be seen as ment level. Believability and reputation are thereby cov- suggestions of how to formulate DQ metrics. Hence, ered by the DQ criterion Trustworthiness on KG level other numerical values and other classification schemes (metric mgraph(hg)), while objectivity and verifiability (e.g., for mgraph(hg)) might be taken for defining the are covered by the DQ criteria Trustworthiness on state- DQ metrics. ment level (metric mfact(g)) and Indicating unknown Trustworthiness on statement level The fulfillment of and empty values (metric mNoV al(g)). Hence, the ful- fillment degree of a KG g w.r.t. the dimension Trust- Trustworthiness on statement level is determined by an assessment whether a provenance vocabulary is used. worthiness is measured by the metrics mgraph, mfact, and mNoV al, which are defined as follows. By means of a provenance vocabulary, the source of statements can be stored. Storing source information is Trustworthiness on KG level The measure of Trust- an important precondition to assess statements easily worthiness on KG level exposes a basic indication about w.r.t. semantic validity. We distinguish between prove- the trustworthiness of the KG. In this assessment, the nance information provided for triples and provenance method of data curation as well as the method of data information provided for resources. insertion is taken into account. Regarding the method The most widely used ontologies for storing prove- of data curation, we distinguish between manual and automated methods. Regarding the data insertion, we nance information are the Dublin Core Metadata 16 can differentiate between: 1. whether the data is entered terms with properties such as dcterms:prove by experts (of a specific domain), 2. whether the knowl- nance and dcterms:source and the W3C PROV 17 edge comes from volunteers contributing in a commu- ontology with properties such as prov:wasDe nity, and 3. whether the knowledge is extracted automat- rivedFrom. ically from a data source. This data source can itself be either structured, semi-structured, or un-structured. We 16See http://purl.org/dc/terms/, requested on Feb 4, assume that a closed system, where experts or other reg- 2017. istered users feed knowledge into a system, is less vul- 17See https://www.w3.org/TR/prov-o/, requested on nerable to harmful behavior of users than an open sys- Dec 27, 2016. 8 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
erty, global cardinality constraints can be specified. Functional relations permit at most one value per re- 1 provenance on statement source (e.g., the birth date of a person). Inverse func- level is used tional relations specify that a value should only occur m (g) = fact 0.5 provenance on resource once per resource. This means that the subject is the level is used 0 otherwise only resource linked to the given object via the given relation. Indicating unknown and empty values If the data Definition of metric. We can measure the data qual- model of the considered KG supports the representa- ity dimension Consistency by means of (i) whether tion of unknown and empty values, more complex state- schema constraints are checked during the insertion of ments can be represented. For instance, empty values new statements into the KG and (ii) whether already allow to represent that a person has no children and existing statements in the KG are consistent to specified unknown values allow to represent that the birth date of class and relation constraints. The fulfillment degree of a person in not known. This kind of higher explanatory a KG g w.r.t. the dimension consistency is measured power of a KG increases the trustworthiness of the KG. by the metrics mcheckRestr, mconClass, and mconRelat, which are defined as follows. 1 unknown and empty values Check of schema restrictions during insertion of new are used statements Checking the schema restrictions during mNoV al(g) = 0.5 either unknown or empty the insertion of new statements can help to reject facts values are used 0 otherwise that would render the KG inconsistent. Such simple checks are often done on the client side in the user inter- 3.2.3. Consistency face. For instance, the application checks whether data Definition of dimension. Consistency implies that with the right data type is inserted. Due to the depen- “two or more values [in a dataset] do not conflict each dency to the actual inserted data, the check needs to be other” [37]. custom-designed. Simple rules are applicable, however, Discussion. Due to the high variety of data providers inconsistencies can still appear if no suitable rules are in the Web of Data, a user must expect data inconsisten- available. Examples of consistency checks are: check- cies. Data inconsistencies may be caused by (i) differ- ing the expected data types of literals; checking whether ent information providers, (ii) different levels of knowl- the entity to be inserted has a valid entity type (i.e., edge, and (iii) different views of the world [11]. checking the rdf:type relation); checking whether In OWL, restrictions can be introduced to ensure the assigned classes of the entity are disjoint, i.e., con- consistent modeling of knowledge to some degree. The tradicting each other (utilizing owl:disjointWith OWL schema restrictions can be divided into class re- relations). strictions and relation restrictions [7]. Class restrictions refer to classes. For instance, one can specify via owl:disjointWith that two 1 schema restrictions are m (h ) = classes have no common instance. checkRestr g checked 0 otherwise Relation restrictions refer to the usage of relations. They can be classified into value constraints and cardi- nality constraints. Consistency of statements w.r.t. class constraints This Value constraints determine the range of relations. metric is intended to measure the degree to which the owl:someValuesFrom, for instance, specifies that instance data is consistent with the class restrictions at least one value of a relation belongs to a certain (e.g., owl:disjointWith) specified on the schema class. If the expected data type of a relation is specified level. via rdfs:range, we also consider this as relation In the following, we limit ourselves to the class restriction. constraints given by all owl:disjointWith state- Cardinality constraints limit the number of times a re- ments defined on the schema level of the consid- lation may exist per resource. Via owl:Functional ered KG. I.e., let CC be the set of all class con- property and owl:InverseFunctionalProp straints, defined as CC := {(c1, c2) | (c1, owl:dis- M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 9
18 jointWith, c2) ∈ g} . Furthermore, let cg(e) be and Rf be the set of all owl:FunctionalPro- the set of all classes of instance e in g, defined as perty constraints, cg(e) = {c | (e, rdf:type, c) ∈ g}. Then we define mconClass(g) as follows: Rf := {(p, d) | (p, rdf:type, owl:Func tionalProperty) ∈ g ∧ (p, rdfs:range, d) ∈ g ∧ isDatatype(d)} mconClass(g) = Then we can define the metrics mconRelatRg(g) and |{(c1, c2) ∈ CC | ¬∃e :(c1 ∈ cg(e) ∧ c2 ∈ cg(e))}| mconRelatF ct(g) as follows: |{(c1, c2) ∈ CC}| m (g) = In case of an empty set of class constraints CC, the conRelatRg metric should evaluate to 1. |{(s, p, o) ∈ g | ∃(p, d) ∈ Rr : datatype(o) 6= d}| |{(s, p, o) ∈ g | ∃(p, d) ∈ R }| Consistency of statements w.r.t. relation constraints r The metric for this criterion is intended for measur- ing the degree to which the instance data is consis- tent with the relation restrictions (e.g., indicated via mconRelatF ct(g) = rdfs:range owl:FunctionalProperty and ) |{(s, p, o) ∈ g|∃(p, d) ∈ Rf : ¬∃(s, p, o2) ∈ g : o 6= o2}| specified on the schema level. We evaluate this crite- |{(s, p, o) ∈ g | ∃(p, d) ∈ Rf }| rion by averaging over the scores obtained from sin- gle metrics mconRelat,i indicating the consistency of In case of an empty set of relation constraints (Rr or statements w.r.t. different relation constraints: Rf ), the respective metric should evaluate to 1.
n 3.3. Contextual Category 1 X m (g) = m (g) conRelat n conRelat,i Contextual data quality “highlights the requirement i=1 that data quality must be considered within the context of the task at hand” [47]. This category contains the In case of evaluating the consistency of instance data three dimensions (i) Relevancy, (ii) Completeness, and concretely w.r.t. given rdfs:range and owl:Func (iii) Timeliness. Wang et al.’s further dimensions in this 19 tionalProperty statements, we can state category, appropriate amount of data and value-added, are considered by us as being part of the dimension Completeness. mconRelatRg(g) + mconRelatF ct(g) mconRelat(g) = 3.3.1. Relevancy 2 Definition of dimension. Relevancy is “the extent Let R be the set of all rdfs:range constraints, to which data are applicable and helpful for the task at r hand” [47]. Discussion. According to Bizer [11], Relevancy is an important quality dimension, since the user is con- Rr := {(p, d) | (p, rdfs:range, d) ∈ g fronted with a variety of potentially relevant informa- ∧ isDatatype(d)} tion on the Web. Definition of metric. The dimension Relevancy is determined by the criterion Creating a ranking of 20 18Implicit restrictions which can be deducted from the class hi- statements. The fulfillment degree of a KG g w.r.t. erarchy, e.g., that a restriction for dbo:Animal counts also for the dimension Relevancy is measured by the metric dbo:Mammal dbo:Animal , a subclass of , are not considered by mRanking, which is defined as follows. us here. 19We chose those relations (and, for instance, not owl:InverseFunctionalProperty), as only those relations 20We do not consider the relevancy of literals, as there is no ranking are used by more than half of the considered KGs. of literals provided for the considered KGs. 10 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Creating a ranking of statements By means of this an investor who is looking for an overview of European criterion one can determine whether the KG supports stocks. The completeness is, hence, only assessable by a ranking of statements by which the relative rele- means of a concrete use case at hand or with the help vance of statements among other statements can be of a defined gold standard. expressed. For instance, given the Wikidata entity Definition of metric. We follow the above-mentioned "Barack Obama" (wdt:Q76) and the relation "posi- distinction of Pipino et al. [40] and determine Com- tion held" (wdt:P39), "President of the United States pleteness by means of the criteria Schema completeness, of America" (wdt:Q11696) has a "preferred rank" Column completeness, and Population completeness. (wdo:PreferredRank) (until 2017), while older The fulfillment degree of a KG g w.r.t. the dimension positions which he holds no more are ranked as "normal Completeness is measured by the metrics mcSchema, rank" (wdo:NormalRank). mcCol, and mcP op, which are defined as follows. Schema completeness By means of the criterion ( Schema completeness 1 ranking of statements supported , one can determine the complete- mRanking(g) = ness of the schema w.r.t. classes and relations [40]. The 0 otherwise schema is assessed by means of a gold standard. This gold standard consists of classes and relations which are Note that this criterion refers to a characteristic of relevant for the use case. For evaluating cross-domain the KG and not to a characteristic of the system that KGs, we use as gold standard a typical set of cross- hosts the KG. domain classes and relations. It comprises (i) basic 3.3.2. Completeness classes such as people and locations in different gran- Definition of dimension. Completeness is “the ex- ularities and (ii) basic relations such as birth date and tent to which data are of sufficient breadth, depth, and number of inhabitants. We define the schema complete- scope for the task at hand” [47]. ness mcSchema as the ratio of the number of classes We include the following two aspects in this dimen- and relations of the gold standard existing in g, noclatg, sion, which are separate dimensions in Wang et al.’s and the number of classes and relations in the gold framework: standard, noclat.
– Appropriate amount of data: Appropriate amount noclatg of data is “the extent to which the quantity or mcSchema(g) = volume of available data is appropriate” [47]. noclat – Value-added: Value-added is “the extent to which Column completeness In the traditional database area data are beneficial and provide advantages from (with fixed schema), by means of the Column complete- their use” [47]. ness criterion one can determine the degree by which Discussion. Pipino et al. [40] divide Completeness the relations of a class, which are defined on the schema into level (each relation has one column), exist on the in- stance level [40]. In the Semantic Web and Linked Data Schema completeness 1. , i.e., the extent to which context, however, we cannot presume any fixed rela- classes and relations are not missing, tional schema on the schema level. The set of possible 2. Column completeness, i.e., the extent to which relations for the instances of a class is given "at run- values of relations on instance level – i.e., facts – time" by the set of used relations for the instances of are not missing, and this class. Therefore, we need to modify this criterion 3. Population completeness, i.e., the extent to which as already proposed by Pipino et al. [40]. In the updated entities are not missing. version, by means of the criterion Column completeness The Completeness dimension is context-dependent and one can determine the degree by which the instances of therefore belongs to the contextual category, because a class use the same relations, averaged over all classes. the fact that a KG is seen as complete depends on the Formally, we define the Column completeness met- use case scenario, i.e., on the given KG and on the infor- ric mcCol(g) as the ratio of the number of instances mation need of the user. As exemplified by Bizer [11], having class k and a value for the relation r, nokp, to a list of German stocks is complete for an investor who the number of all instances having class k, nok. By is interested in German stocks, but it is not complete for averaging over all class-relation-pairs which occur on M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 11 instance level, we obtain a fulfillment degree regarding measured depends on the application context: For some the whole KG: situations years are sufficient, while in other situations one may need days [39]. Definition of metric. The dimension timeliness is 1 X nokp Timeliness frequency of the mcCol(g) = determined by the criteria |H| nok (k,p)∈H KG, Specification of the validity period, and Specifica- tion of the modification date of statements. We thereby let H = {(k, p) ∈ (K × P ) | ∃k ∈ The fulfillment degree of a KG g w.r.t. the dimen- imp sion Timeliness is measured by the metrics mF req, Cg ∧ ∃(x, p, o) | p ∈ Pg ∧ (x, rdf:type, k)} be the set of all combinations of the considered classes, mV alidity, and mChange, which are defined as follows. K = {k1, ..., kn}, and considered relations, P = Timeliness frequency of the KG The criterion Time- {p1, ..., pm}. liness frequency of the KG indicates how fast the KG Note that there are also relations which are dedicated is updated. We consider the KG RDF export here and to the instances of a specific class, but which do not differentiate between continuous updates, where the up- need to exist for all instances of that class. For instance, dates are always performed immediately, and discrete not all people need to have a relation :hasChild or KG updates, where the updates take place in discrete 21 :deathDate. For measuring the Column complete- time intervals. In case the KG edits are available online ness, we selected only those relations for an assessment immediately but the RDF export files are available in where a value of the relation typically exists for all discrete, varying updating intervals, we consider the given instances. online version of the KG, since in the context of Linked Population completeness The Population complete- Data it is sufficient that URIs are dereferenceable. ness metric determines the extent to which the consid- 1 ered KG covers a basic population [40]. The assess- continuous updates ment of the KG completeness w.r.t. a basic population 0.5 discrete periodic updates mF req(g) = is performed by means of a gold standard, which covers 0.25 discrete non-periodic updates both well-known entities (called “short head”, e.g., the 0 otherwise n largest cities in the world according to the number of inhabitants) and little-known entities (called “long tail”; Specification of the validity period of statements Spec- e.g., municipalities in Germany). We take all entities ifying the validity period of statements enables to tem- contained in our gold standard equally into account. porally limit the validity of statements. By using this cri- Let GS be the set of entities in the gold standard. terion, we measure whether the KG supports the speci- Then we can define: fication of starting and maybe end dates of statements by means of providing suitable forms of representation. |{e|e ∈ GS ∧ e ∈ E }| m (g) = g cP op |{e|e ∈ GS}| 1 specification of validity pe- 3.3.3. Timeliness mV alidity(g) = riod supported Definition of dimension. Timeliness is “the extent 0 otherwise to which the age of the data is appropriate for the task at hand” [47]. Specification of the modification date of statements Discussion. Timeliness does not describe the creation The modification date discloses the point in time date of a statement, but instead the time range since the of the last verification of a statement. The modifi- last update or the last verification of the statement [39]. cation date is typically represented via the relations Due to the easy way of publishing data on the Web, schema:dateModified and dcterms:modi data sources can be kept easier up-to-date than tradi- fied. tional isolated data sources. This results in advantages to the consumer of Web data [39]. How Timeliness is 1 specification of modifica- tion dates for statements mChange(g) = 21 supported For an evaluation about the prediction which relations are of this nature, see [1]. 0 otherwise 12 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
3.4. Representational Data Quality (Intermediate-Node-Mapping) keine rdfs:label haben. Folglich würde ich die Definition der Metrik allgemein Representational data quality “contains aspects re- halten (beschränkt auf proprietäre Ressourcen, d.h. im lated to the format of the data [...] and meaning of selben Namespace), die Evaluation jedoch nur anhand data” [47]. This category contains the two dimensions der Entitäten machen. (i) Ease of understanding (i.e., regarding the human- Labels in multiple languages Resources in the KG are readability) and (ii) Interoperability (i.e., regarding the described in a human-readable way via labels, e.g., via machine-readability). The dimensions Interpretability, rdfs:label or skos:prefLabel.22 The charac- Representational consistency and Concise representa- teristic feature of skos:prefLabel is that this kind tion as in addition proposed by Wang et al. [47] are of label should be used per resource at most once; in considered by us as being a part of the dimension Inter- contrast, rdfs:label has no cardinality restrictions, operability. i.e., it can be used several times for a given resource. 3.4.1. Ease of Understanding Labels are usually provided in English as the “basic Definition of dimension. The ease of understanding language.” The now introduced metric for the criterion is “the extent to which data are clear without ambiguity Labels in multiple languages determines whether labels and easily comprehended” [47]. in other languages than English are provided in the KG. Discussion. This dimension focuses on the under- standability of a data source by a human data con- 1 Labels provided in English sumer. In contrast, the dimension Interoperability fo- and at least one other lan- m (g) = cuses on technical aspects. The understandability of a Lang guage data source (here: KG) can be improved by things such 0 otherwise as descriptive labels and literals in multiple languages. Definition of metric. The dimension understand- Understandable RDF serialization RDF/XML is the ability is determined by the criteria Description of re- recommended RDF serialization format of the W3C. sources, Labels in multiple languages, Understandable However, due to its syntax RDF/XML documents are RDF serialization, and Self-describing URIs. The ful- hard to read for humans. The understandability of RDF fillment degree of a KG g w.r.t. the dimension Con- data by humans can be increased by providing RDF sistency is measured by the metrics mDescr, mLang, in other, more human-understandable serialization for- muSer, and muURI , which are defined as follows. mats such as N3, N-Triple, and Turtle. We measure this criterion by measuring the supported serialization Description of resources Heath et al. [26,30] suggest formats during the dereferencing of resources. to describe resources in a human-understandable way, e.g., via rdfs:label or rdfs:comment. Within 1 our framework, the criterion is measured as follows: Other RDF serializations m (h ) = Given a sample of resources, we divide the number uSer g than RDF/XML available 0 otherwise of resources in the KG for which at least one label or rdfs:label one description is provided, (e.g., via , Note that conversions from one RDF serialization rdfs:comment schema:description , or ) by format into another are easy to perform. the number of all considered resources in the local namespace: Self-describing URIs Descriptive URIs contribute to a better human-readability of KG data. Sauermann et 23 local al. recommend to use short, memorable URIs in the mDescr(g) = |{u|u ∈ Ug ∧ ∃(u, p, o) ∈ g : Semantic Web context, which are easier understandable local and memorable by humans compared to opaque URIs24 p ∈ PlDesc}|/|{u|u ∈ Ug }|
22 PlDesc is the set of implicitly used relations in g in- Using the namespace http://www.w3.org/2004/02/ dicating that the value is a label or description (e.g., skos/core#. 23See https://www.w3.org/TR/cooluris, requested on PlDesc = {rdfs:label, rdfs:comment}). Mar 1, 2016. Beschreibung). Darüber hinaus ist das Ergebnis 24For an overview of URI patterns see https://www.w3. der Evaluation auf Basis der Entitäten interessant - org/community/bpmlod/wiki/Best_practises_- > DBpedia weicht deutlich ab, da manche Entitäten _previous_notes, requested on Dec 27, 2016. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 13 such as wdt:Q1040. The criterion Self-describing container, and (iii) blank nodes are not very widely URIs is dedicated to evaluate whether self-describing used in the Linked Open Data context. Those features URIs or generic IDs are used for the identification of should be avoided according to Heath et al. in order resources. to simplify the processing of data on the client side. Even the querying of the data via SPARQL may get complicated if RDF reification, RDF collections, and 1 self-describing URIs always used RDF container are used. We agree on that, but also muURI (g) = 0.5 self-describing URIs partly used point out that reification (implemented via RDF stan- 0 otherwise dard reification, n-ary relations, singleton properties, or named graphs) is inevitably necessary for making 3.4.2. Interoperability statements about statements. Interoperability is another dimension of the repre- Definition of metric. The dimension Interoperabil- sentational data quality category and subsumes Wang ity is determined via the following criteria: et al.’s aspects interpretability, representational consis- – Avoiding blank nodes and RDF reification tency, and concise representation. Definition of dimension. We define Interoperability – Provisioning of several serialization formats along the subsumed dimensions of Wang et al.: – Using external vocabulary – Interoperability of proprietary vocabulary – Interpretability: Interpretability is “the extent to which data are in appropriate language and units The fulfillment degree of a KG g w.r.t. the dimen- and the data definitions are clear” [47]. sion Interoperability is measured by the metrics mReif , – Representational consistency: Representational miSerial, mexV oc, and mpropV oc, which are defined as consistency is “the extent to which data are always follows. presented in the same format and are compatible Avoiding blank nodes and RDF reification Using RDF with previous data” [47]. blank nodes, RDF reification, RDF container, and RDF – Concise representation: Concise representation lists is often considered as ambivalent: On the one hand, is “the extent to which data are compactly repre- these RDF features are not very common and they sented without being overwhelming” [47]. complicate the processing and querying of RDF data Discussion regarding interpretability. In contrast [30,26]. On the other hand, they are necessary in cer- to the dimension understandability, which focuses on tain situations, e.g., when statements about statements the understandability of RDF KG data towards the user should be made. We measure the criterion by evaluating as data consumer, interpretability focuses on the rep- whether blank nodes and RDF reification are used. resentation forms of information in the KG from a technical perspective. An example is the consideration 1 no blank nodes and no RDF whether blank nodes are used. According to Heath et reification al. [26], blank nodes should be avoided in the Linked mReif (g) = 0.5 either blank nodes or RDF Data context, since they complicate the integration of reification multiple data sources and since they cannot be linked 0 otherwise by resources of other data sources. Discussion regarding representational consistency. Provisioning of several serialization formats The in- In the context of Linked Data, it is best practice to reuse terpretability of RDF data of a KG is increased if be- existing vocabulary for the creation of own RDF data. In this way, less data needs to be prepared for being proposal described in the RDF standard (Brickley, D., Guha, R. (eds.): published as Linked Data [26]. RDF Vocabulary Description Language 1.0: RDF Schema. W3C Discussion regarding concise representation. Heath Recommendation, online available at http://www.w3.org/TR/ et al. [26] made the observation that the RDF features rdf-schema/, requested on Sep 2, 2016.). For more information (i) RDF reification,25 (ii) RDF collections and RDF about reification and its implementation possibilities, we can refer the reader to [27]. In this article, we use the term "reification" by default for the general sense and "standard reification" or "RDF reification" 25In the literature, it is often not differentiated between "reification" for referring to the modeling of reification according to the RDF in the general sense and "reification" in the sense of the specific standard. 14 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO sides the serialization standard RDF/XML further seri- – Accessibility, alization formats are supported for URI dereferencing. – Licensing, and – Interlinking. 1 RDF/XML and further for- Wang’s dimension access security is considered by us mats are supported as being not relevant in the Linked Open Data context, miSerial(hg) = 0.5 only RDF/XML is supported as we only take open data sources into account. 0 otherwise In the following, we go into details of the mentioned data quality dimensions: Using external vocabulary Using a common vocabu- 3.5.1. Accessibility lary for representing and describing the KG data allows Definition of dimension. Accessibility is “the ex- to represent resources and relations between resources tent to which data are available or easily and quickly in the Web of Data in a unified way. This increases the retrievable” [47]. interoperability of data [30,26] and allows a comfort- Discussion. Wang et al.’s definition of Accessibility able data integration. We measure the criterion of using contains the aspects availability, response time, and an external vocabulary by setting the number of triples data request. They are defined as follows: with external vocabulary in predicate position to the number of all triples in the KG: 1. Availability “of a data source is the probability that a feasible query is correctly answered in a given time range” [39]. external |{(s, p, o)|(s, p, o) ∈ g ∧ p ∈ Pg }| According to Naumann [39], the availability is an m (g) = extV oc |{(s, p, o) ∈ g}| important quality aspect for data sources on the Web, since in case of integrated systems (with fed- Interoperability of proprietary vocabulary Linking erated queries) usually all data sources need to on schema level means to link the proprietary vo- be available in order to execute the query. There cabulary to external vocabulary. Proprietary vocab- can be different influencing factors regarding the ulary are classes and relations which were defined availability of data sources, such as the day time, in the KG itself. The interlinking to external vo- the worldwide distribution of servers, the planed cabulary guarantees a high degree of interoperabil- maintenance work, and the caching of data. Linked ity [26]. We measure the interlinking on schema Data sources can be available as SPARQL end- level by calculating the ratio to which classes and points (for performing complex queries on the relations have at least one equivalency link (e.g., data) and via HTTP URI dereferencing. We need owl:sameAs, owl:equivalentProperty, or to consider both possibilities for this DQ dimen- owl:equivalentClass) to classes and relations, sion. respectively, of other data sources. 2. Response time characterizes the delay between the point in time when the query was submitted and the point in time when the query response is mpropV oc(g) = |{x ∈ Pg ∪ Cg|∃(x, p, o) ∈ g : received [11]. Note that the response time is dependent on em- (p ∈ P ∧ (o ∈ U ∧ o ∈ U ext))}|/|P ∪ C | eq g g g pirical factors such as the query, the size of the in- dexed data, the data structure, the used triple store, where P = {owl:sameAs, owl:equivalent- eq the hardware, and so on. We do not consider the Property, owl:equivalenClass} and U ext con- g response time in our evaluations, since obtaining sists of all URIs in U which are external to the KG g g a comprehensive result here is hard. which means that h is not responsible for resolving g 3. In the context of Linked Data, data requests can these URIs. be made (i) on SPARQL endpoints, (ii) on RDF dumps (export files), and (iii) on Linked Data 3.5. Accessibility Category APIs. Accessibility data quality refers to aspects on how Definition of metric. We define the metric for the data can be accessed. This category contains the three dimension Accessibility by means of metrics for the dimensions following criteria: M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 15
– Dereferencing possibility of resources we do not measure these restrictions here. – Availability of the KG – Provisioning of public SPARQL endpoint 1 SPARQL endpoint publicly – Provisioning of an RDF export mSP ARQL(hg) = available 0 otherwise – Support of content negotiation Linking HTML sites to RDF serializations – Provisioning of an RDF export If there is no pub- – Provisioning of KG metadata lic SPARQL endpoint available or the restrictions of The fulfillment degree of a KG g w.r.t. the dimen- this endpoint are so strict that the user does not use it, an RDF export dataset (RDF dump) can often be sion Accessibility is measured by the metrics mDeref , used. This dataset can be used to set up a local, pri- m , m , m , m , m , Avai SP ARQL Export Negot HT MLRDF vate SPARQL endpoint. The criterion here indicates and m , which are defined as follows. Meta whether an RDF export dataset is officially available: Dereferencing possibility of resources One of the ( Linked Data principles [9] is the dereferencing possi- 1 RDF export available mExport(hg) = bility of resources: URIs must be resolvable via HTTP 0 otherwise requests and useful information should be returned thereby. We assess the dereferencing possibility of re- Support of content negotiation Content negotiation sources in the KG by analyzing for each URI in the sam- (CN) allows that the server returns RDF documents ple set (here: all URIs Ug) the HTTP response status during the dereferencing of resources in the desired code and by evaluating whether RDF data is returned. A RDF serialization format. The HTTP protocol allows successful dereferencing of resources is given if HTTP the client to specify the desired content type (e.g., RDF/ status code 200 and an RDF document is returned. XML) in the HTTP request and the server to specify the returned content type in the HTTP response header |dereferencable(Ug)| (e.g., application/rdf+xml). In this way, the de- mDeref (hg) = |Ug| sired and the provided content type are matched as far as possible. It can happen that the server does not pro- Availability of the KG The Availability of the KG cri- vide the desired content type. Moreover, it may hap- terion indicates the uptime of the KG. It is an essential pen that the server returns an incorrect content type. criterion in the context of Linked Data, since in case of This may lead to the fact that serialized RDF data is an integrated or federated query mostly all data sources not processed further. An example is RDF data which need to be available [39]. We measure the availabil- is declared as text/plain [26]. Hogan et al. [29] ity of a KG by monitoring the ability of dereferencing therefore propose to let KGs return the most specific URIs over a period of time. This monitoring process content type as possible. We measure the Support of can be done with the help of a monitoring tool such as content negotiation by dereferencing resources with Pingdom.26 different RDF serialization formats as desired content type and by comparing the accept header of the HTTP Number of successful requests request with the content type of the HTTP response. mAvai(hg) = Number of all requests 1 CN supported and correct Provisioning of public SPARQL endpoint SPARQL content types returned endpoints allow the user to perform complex queries mNegot(hg) = 0.5 CN supported but wrong (including potentially many instances, classes, and rela- content types returned tions) on the KG. This criterion here indicates whether 0 otherwise an official SPARQL endpoint is publicly available. There might be additional restrictions of this SPARQL Linking HTML sites to RDF serializations Heath et endpoint such as a maximum number of requests per al. [26] suggest linking any HTML description of a time slice or a maximum runtime of a query. However, resource to RDF serializations of this resource in or- der to make the discovery of corresponding RDF data easier (for Linked Data aware applications). For that 26See http://pingdom.com/, requested on Mar 1, 2016. reason, in the HTML header the so-called Autodiscov- 16 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
ery pattern can be included. This pattern consists of contracts which define rights and obligations. These the phrase link rel=alternate, the indication contracts are also in the Linked Data context popular. about the provided RDF content type, and a link to the The most frequent licenses for Linked Data are CC-BY, RDF document.27 We measure the linking of HTML CC-BY-SA, and CC0 [31]. CC-BY30 requires specify- pages to RDF documents (i.e., resource representations) ing the source of the data, CC-BY-SA31 requires in ad- by evaluating whether the HTML representations of the dition that if the data is published, it is published under resources contain links as described: the same legal conditions; CC032 defines the respective data as public domain and without any restrictions. Noteworthy is that most data sources in the Linked 1 Autodiscovery pattern used Open Data cloud do not provide any licensing infor- mHT MLRDF (hg) = at least once mation [31] which makes it difficult to use the data 0 otherwise in commercial settings. Even if data is published un- der CC-BY or CC-BY-SA, the data is often not used Provisioning of KG metadata In the light of the Se- since companies refer to uncertainties regarding these mantic Web vision where agents select and make use contracts. of appropriate data sources on the Web, also the meta- Definition of metric. The dimension License is information about KGs needs to be available in a determined by the criterion Provisioning machine- machine-readable format. The two important mech- readable licensing information. anisms to specify metadata about KGs are (i) using The fulfillment degree of a KG g w.r.t. the dimension semantic sitemaps and (ii) using the VoID vocabu- License is measured by the metric mmacLicense, which lary28 [26]. For instance, the URI of the SPARQL end- is defined as follows. point can be assigned via void:sparqlEndpoint Provisioning machine-readable licensing information and the RDF export URL can be specified with Licenses define the legal frameworks under which the void:dataDump. Such metadata can be added as ad- KG data may be used. Providing machine-readable li- ditional facts to the KG or it can be provided as separate censing information allows users and applications to be VoID file. We measure the Provisioning of KG meta- aware of the license and to use the data of the KG in data by evaluating whether machine-readable metadata accordance with the legal possibilities [30,26]. about the KG is available. Note that the provisioning Licenses can be specified in RDF via relations of licensing information in a machine-readable format such as cc:licence,33 dcterms:licence, or (which is also a meta-information about the KG) is dcterms:rights. The licensing information can be considered in the data quality dimension License later specified either in the KG as additional facts or sepa- on. rately in a VoID file. We measure the criterion by eval- uating whether licensing information is available in a 1 Machine-readable metadata machine-readable format: mMeta(g) = about g available 0 otherwise 1 machine-readable licensing information m (g) = 3.5.2. License macLicense available Definition of dimension. Licensing is defined as 0 otherwise “the granting of permission for a consumer to re-use a dataset under defined conditions” [49]. 3.5.3. Interlinking Discussion. The publication of licensing information Definition of dimension. Interlinking is the extent about KGs is important for using KGs without legal “to which entities that represent the same concept are concerns, especially in commercial settings. Creative Commons (CC)29 publishes several standard licensing 30See https://creativecommons.org/licenses/ by/4.0/, requestedon Mar 1, 2016. 31See https://creativecommons.org/licenses/ 27An example is
∗ Trustworthiness 4. Selection of KGs ∗ Trustworthiness on KG level ∗ Trustworthiness on statement level We consider the following KGs for our comparative ∗ Using unknown and empty values evaluation: ∗ Consistency – DBpedia: DBpedia39 is the most prominent KG ∗ Check of schema restrictions during inser- in the LOD cloud [4]. The project was initiated tion of new statements by researchers from the Free University of Berlin ∗ Consistency of statements w.r.t. class con- and the University of Leipzig, in collaboration straints with OpenLink Software. Since the first public re- ∗ Consistency of statements w.r.t. relation con- lease in 2007, DBpedia is updated roughly once a straints year.40 By means of a dedicated open source ex- – Contextual category traction framework, DBpedia is created from infor- ∗ Relevancy mation contained in Wikipedia, such as infobox ta- ∗ Creating a ranking of statements bles, categorization information, geo-coordinates, ∗ Completeness and external links. Due to its role as the hub of ∗ Schema completeness the LOD cloud, DBpedia contains many links to ∗ Column completeness other datasets in the LOD cloud such as Freebase, ∗ Population completeness OpenCyc, UMBEL,41 GeoNames, Musicbrainz,42 ∗ Timeliness CIA World Factbook,43 DBLP,44 Project Guten- ∗ Timeliness frequency of the KG berg,45 DBtune Jamendo,46 Eurostat,47 Uniprot,48 ∗ Specification of the validity period of state- and Bio2RDF.49,50 DBpedia has been used exten- ments sively in the Semantic Web research community, ∗ Specification of the modification date of but has become also relevant in commercial set- statements tings: for instance, companies such as the BBC – Representational data quality [33] and the New York Times [41] use DBpedia ∗ Ease of understanding to organize their content. The version of DBpedia ∗ Description of resources we analyzed is 2015-04. ∗ Labels in multiple languages ∗ Understandable RDF serialization 39See http://dbpedia.org, requested on Nov 1, 2016. 40 ∗ Self-describing URIs There is also DBpedia live which started in 2009 and which gets updated when Wikipedia is updated. See http://live. ∗ Interoperability dbpedia.org/, requested on Nov 1, 2016. Note, however, that ∗ Avoiding blank nodes and RDF reification DBpedia live only provides a restricted set of relations compared to ∗ Provisioning of several serialization formats DBpedia. Also, the provisioning of data varies a lot: While for some ∗ Using external vocabulary time ranges DBpedia live provides data for each hour, for other time ∗ Interoperability of proprietary vocabulary ranges DBpedia live data is only available once a month. 41See http://umbel.org/, requested on Dec 31, 2016. – Accessibility category 42See http://musicbrainz.org/, requested on Dec 31, ∗ Accessibility 2016. 43 ∗ Dereferencing possibility of resources See https://www.cia.gov/library/ publications/the-world-factbook/, requested on Dec ∗ Availability of the KG 31, 2016. ∗ Provisioning of public SPARQL endpoint 44See http://www.dblp.org, requested on Dec 31, 2016. ∗ Provisioning of an RDF export 45See https://www.gutenberg.org/, requested on Dec ∗ Support of content negotiation 31, 2016. 46See http://dbtune.org/jamendo/, requested on Dec ∗ Linking HTML sites to RDF serializations 31, 2016. ∗ Provisioning of KG metadata 47See http://eurostat.linked-statistics.org/, ∗ License requested on Dec 31, 2016. 48 ∗ Provisioning machine-readable licensing in- See http://www.uniprot.org/, requested on Dec 31, formation 2016. 49See http://bio2rdf.org/, requested on Dec 31, 2016. ∗ Interlinking 50See a complete list of the links on the websites describing the sin- ∗ Interlinking via owl:sameAs gle DBpedia versions such as http://downloads.dbpedia. ∗ Validity of external URIs org/2016-04/links/ (requested on Nov 1, 2016). M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 19
– Freebase: Freebase51 is a KG announced by Wikipedia. Wikidata does not only store facts, but Metaweb Technologies, Inc. in 2007 and was ac- also the corresponding sources, so that the valid- quired by Google Inc. on July 16, 2010. In con- ity of facts can be checked. Labels, aliases, and trast to DBpedia, Freebase had provided an in- descriptions of entities in Wikidata are provided terface that allowed end-users to contribute to in almost 400 languages. Wikidata is a commu- the KG by editing structured data. Besides user- nity effort, i.e., users collaboratively add and edit contributed data, Freebase integrated data from information. Also, the schema is maintained and Wikipedia, NNDB,52 FMD,53 and MusicBrainz.54 extended based on community agreements. Wiki- Freebase uses a proprietary graph model for stor- data is currently growing considerably due to the ing also complex statements. Freebase shut down integration of Freebase data [44]. The version of its services completely on August 31, 2016. Only Wikidata we analyzed is 2015-10. the latest data dump is still available. Wikimedia – YAGO: YAGO60 – Yet Another Great Ontol- Deutschland and Google integrate Freebase data ogy – has been developed at the Max Planck into Wikidata via the Primary Sources Tool.55 Fur- Institute for Computer Science in Saarbrücken ther information about the migration from Free- since 2007. YAGO comprises information ex- base to Wikidata is provided in [44]. We analyzed tracted from Wikipedia (such as information from the latest Freebase version as of March 2015. the categories, redirects, and infoboxes), Word- – OpenCyc: The Cyc56 project started in 1984 by Net [19] (such as information about synsets and 61 the industry research and development consortium hyponomies), and GeoNames. The version of Microelectronics and Computer Technology Cor- YAGO we analyzed is YAGO3, which was pub- poration. The aim of Cyc is to store – in a machine- lished in March 2015. processable way – millions of common sense facts such as “Every tree is a plant.” The main focus of 5. Comparison of KGs Cyc has been on inferencing and reasoning. Since Cyc is proprietary, a smaller version of the KG 5.1. Key Statistics called OpenCyc57 was released under the open source Apache license Version 2. In July 2006, Re- In the following, we present statistical commonal- searchCyc58 was published for the research com- ities and differences of the KGs DBpedia, Freebase, munity, containing more facts than OpenCyc. We OpenCyc, Wikidata, and YAGO. We thereby use the did not consider Cyc and ResearchCyc, since those following key statistics: KGs do not meet the chosen requirements, namely, – Number of triples that the KGs are freely available and freely us- – Number of classes able in any context. The version of OpenCyc we – Number of relations analyzed is 2012-05-10. – Distribution of classes w.r.t. the number of their – Wikidata: Wikidata59 is a project of Wikimedia corresponding instances Deutschland which started on October 30, 2012. – Coverage of classes with at least one instance per The aim of the project is to provide data which class can be used by any Wikimedia project, including – Covered domains w.r.t. entities – Number of entities – Number of instances 51See http://freebase.com/, requested on Nov 1, 2016. – Number of entities per class 52See http://www.nndb.com, requested on Dec 31, 2016. – Number of unique subjects 53 See http://www.fashionmodeldirectory.com/, re- – Number of unique predicates quested on Dec 31, 2016. – Number of unique objects 54See http://musicbrainz.org/, requested on Dec 31, 2016. In Section 7.2, we provide an overview of related 55See https://www.wikidata.org/wiki/Wikidata: work w.r.t. those key statistics. Primary_sources_tool, requested on Apr 8, 2016. 56See http://www.cyc.com/, requested on Dec 31, 2016. 57See http://www.opencyc.org/, accessed on Nov 1, 60See http://www.mpi-inf.mpg.de/departments/ 2016. databases-and-information-systems/research/ 58See http://research.cyc.com/, requested on Dec 31, yago-naga/yago/downloads/, accessed on Nov 1, 2016. 2016. 61See http://www.geonames.org/, requested on Dec 31, 59See http://wikidata.org/, accessed on Nov 1, 2016. 2016. 20 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
5.1.1. Triples high number of unique subjects concerning the set of Ranking of KGs w.r.t. number of triples. The num- all triples. ber of triples (see Table 2) differs considerably between In case of DBpedia, Freebase, and Wikidata, reifica- the KGs: Freebase is the largest KG with over 3.1B tion is implemented by means of n-ary relations. An triples, while OpenCyc resides the smallest KG with n-ary relation denotes the relation between more than only 2.4M triples. The large size of Freebase can be two resources and is implemented via additional, inter- traced back to the fact that large data sets such as Mu- mediate nodes, since in RDF only binary statements sicBrainz have been integrated into this KG. OpenCyc, can be modeled [16,27]. In Freebase and DBpedia, data in contrast, has been built purely manually by experts. is mostly provided in the form of plain N-Triples and In general, this indicates a correlation between the way n-ary relations are only used for data from higher ar- of building up a KG and its size. ity.63 Wikidata, in contrast, has the peculiarity that not Size differences between DBpedia and YAGO. As only every statement is expressed with the help of an both DBpedia and YAGO were created automatically n-ary relation, but that in addition each statement is in- by extracting semantically-structured information from stantiated with wdo:Statement. This leads to about Wikipedia, the significant difference between their sizes 74M additional instances, which is about one tenth of – in terms of triples – is in particular noteworthy. We all triples in Wikidata. can mention here the following reasons: YAGO inte- 5.1.2. Classes grates the statements from different language versions Methods for counting classes. The number of of Wikipedia in one single KG while for the canon- classes can be calculated in different ways: Classes can ical DBpedia dataset (which is used in our evalua- be identified via rdfs:Class and owl:Class re- tions) solely the English Wikipedia was used as in- lations, or via rdfs:subClassOf relations.64 Since formation source. Besides that, YAGO contains con- Freebase does not provide any class hierarchy with textual information and detailed provenance informa- rdfs:subClassOf relations and since Wikidata tion. Contextual information is for instance the an- does not instantiate classes explicitly as classes, but chor texts of all links within Wikipedia. For repre- uses instead only “subclass of” (wdt:P279) relations, senting the anchor texts, the relation yago:hasWiki the method of calculating the number of classes de- pediaAnchorText (330M triples in total) is used. pends on the considered KG. The provenance information of single statements is Ranking of KG w.r.t. number of classes. Our eval- stored in a reified form. In particular, the relations uations revealed that YAGO contains the highest num- yago:extractionSource (161.2M triples) and ber of classes of all considered KGs; DBpedia, in con- yago:extractionTechnique (176.2M triples) trast, has the fewest (see Table 2). are applied therefore. Number of classes in YAGO and DBpedia. How 3n does it come to this gap between DBpedia and YAGO Influence of reification on the number of triples. with respect to the number of classes, although both DBpedia, Freebase, Wikidata, and YAGO use some KGs were created automatically based on Wikipedia? form of reification. Reification in general describes For YAGO, the classes are extracted from the categories the possibility of making statements about statements. in Wikipedia, while the hierarchy of the classes is de- While reification has an influence on the number of ployed with the help of WordNet synset relations. The triples for DBpedia, Freebase, and Wikidata, the num- DBpedia ontology, in contrast, is very small, since it ber of triples in YAGO is not influenced by reification is created manually, based on the mostly used infobox since data is here provided in N-Quads.62 This style of reification is called Named Graph [27]: The additional column (in comparison to triples) contains a unique ID 63In Freebase Compound Value Types are used for reifi- of the statement by which the triple becomes identified. cation [44]. In DBpedia it is named Intermedia Node Map- ping, see http://mappings.dbpedia.org/index.php/ For backward compatibility the ID is commented and Template:IntermediateNodeMapping (requested on Dec therefore not imported into the triple store. Note, how- 31, 2016). ever, that transforming N-Quads to N-Triples leads to a 64The number of classes in a KG may also be calculated by taking all entity type relations (rdf:type and “instance of” (wdt:P31) in case of Wikidata) on the instance level into account. However, this 62The idea of N-Quads is based on the assignment of triples to would result only in a lower bound estimation, as here those classes different graphs. YAGO uses N-Quads to identify statements per ID. are not considered which have no instances. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 21
100 Table 1 Percentage of considered entities per KG for covered domains
80 DB FB OC WD YA
Reach of method 88% 92% 81% 41% 82% 60
40 creases consistently for the first 250 classes, before it decreases more than exponentially beyond class 250. Coverage in % 20 5.1.3. Domains All considered KGs are cross-domain, meaning that a variety of domains are covered in those KGs. However, the KGs often cover the single domains to a different YAGO degree. Tartir [45] proposed to measure the covered do- DBpediaFreebaseOpenCyc Wikidata mains of ontologies by determining the usage degree of corresponding classes: the number of instances belong- Fig. 1. Coverage of classes having at least one instance. ing to one or more subclasses of the respective domain is compared to the number of all instances. In our work, templates in Wikipedia. Besides those 736 classes, the however, we decided to evaluate the coverage of do- DBpedia KG contains further 444,895 classes which mains concerning the classes per KG via manual assign- originate from the imported YAGO classes and which ments of the mostly used classes to the domains people, are published in the namespace yago:. Those YAGO media, organizations, geography, and biology.65 This classes are – like the DBpedia ontology classes – inter- list of domains was created by aggregating the most connected via rdfs:subClassOf to form a taxon- frequent domains in Freebase. omy. In the evaluation of DBpedia, the YAGO classes The manual assignment of classes to domains are ignored, as they do not belong to the DBpedia on- is necessary in order to obtain a consistent assign- tology given as OWL file. ment of the classes to the domains across all con- Coverage of classes with at least one instance. sidered KGs. Otherwise, the same classes in differ- Fig. 1 shows for each KG the extent to which classes are ent KGs may be assigned to different domains. More- instantiated, that is, for how many classes at least one over, in some KGs classes may otherwise appear in instance exists. YAGO exhibits the highest coverage various domains simultaneously. For instance, the rate (82.6%), although it contains the highest number Freebase classes freebase:music.artist and of classes among the KGs. This can be traced back to freebase:people.person overlap in terms of the fact that YAGO classes are chosen by a heuristic their instances and multiple domains (such as music that considers Wikipedia leaf categories which tend to and people) might be assigned to them. have instances [43]. OpenCyc (with 6.5%) and Wiki- As the reader can see in Table 1, our method to de- data (5.4%) come last in the ranking. Wikidata has the termine the coverage of domains, and, hence, the reach second highest number of classes in total (see Table 2), of our evaluation, includes about 80% of all entities of out of which relatively little are used on instance level. each KG, except Wikidata. It is calculated as the ratio of Note, however, that in some scenarios solely the schema the number of unique entities of all considered domains level information (including classes) of KGs is neces- of a given KG divided by the number of all entities of 66 sary, so that the low coverage of instances by classes is this KG. If the ratio was at 100% we were able to not necessarily an issue. assign all entities of a KG to the chosen domains. Correlation between number of classes and num- Fig. 3 shows the number of entities per domain in the ber of instances. In Fig. 2, we can see a histogram different KGs with a logarithmic scale. Fig. 4 presents of the classes with respect to the number of instances per class. That is, for each KG we can spot how many 65See our website for examples of classes per domain and classes have a high number of instances and how many per KG http://km.aifb.kit.edu/sites/knowledge- graph-comparison/ classes have a low number of instances. Note the log- (requested on Dec 31, 2016). 66We used the number of unique entities of all domains and not arithmic scale on both axes. The curves seem to fol- the sum of the entities measured per domain, since entities may be in low power law distributions. For DBpedia, the line de- several domains at the same time. 22 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
10 8
DBpedia Freebase OpenCyc Wikidata 10 6 YAGO
10 4 Number of instances
10 2
10 0 10 0 10 1 10 2 10 3 Classes
Fig. 2. Distribution of classes w.r.t. the number of instances per KG.
10 10 DBpedia Freebase OpenCyc 10 8 Wikidata YAGO
10 6
10 4 Number of entities
10 2
10 0 persons media organizations geography biology
Fig. 3. Number of entities per domain. the relative coverage of each domain in each KG. It is domain. This fact can be traced back to large-scale calculated as the ratio of the number of entities in each data imports, such as from MusicBrainz. The class domain to the total number of entities of the KG. A freebase:music.release_track is account- value of 100% means that all instances reside in one single domain. able for 42% of the media entities. As shown in Fig. 3, The case of Freebase is especially outstanding here: Freebase provides the most entities in four out of the 77% of all entities here are located in the media five domains when considering all KGs. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 23
80
DBpedia 70 Freebase OpenCyc 60 Wikidata YAGO 50
40
30
20
Relative number of entities in percent 10
persons media organizations geography biology
Fig. 4. Relative number of entities per domain.
In DBpedia and YAGO, the domain of people is the are explicitly defined as such via assignments (for largest domain (50% and 34%, respectively). Peculiar is instance, with rdfs:Property) to classes. In the higher coverage of YAGO regarding the geography Section 2 we used Pg to denote this set. domain compared to DBpedia. As one reason for that – In contrast, we use predicates to denote links used we can point out the data import of GeoNames into in the KG independently of their introduction on YAGO. the schema level. The set of unique predicates per Wikidata contains around 150K entities in the do- KG, denoted as P imp, is nothing else than the set main organization. This is relativly few considering g of unique RDF terms on the predicate position of the total amount of entities being around 18.7M and considering the number of organizations in other KGs. all triples in the KG. Note that even DBpedia provides more organization entities than Wikidata. The reason why Wikidata has It is important to distinguish the key statistics for rela- not so many organization entities is not fully compre- tions from the key statistics for predicates, since they hensible to us. However, we can point out that for our can differ considerably, depending on to which degree analysis we only considered Wikidata classes which relations are only defined on schema level, but not used appeared more than 6,000 times67 and that about 16K on instance level. classes were therefore not considered. It is possible that Evaluation results. entities of the domain organization are belonging to Relations those rather rarely occurring classes. Ranking regarding relations. As presented in Ta- 5.1.4. Relations and Predicates ble 2, Freebase exhibits by far the highest number of Evaluation method. In this article, we differentiate unique relations (around 785K) among the KGs. YAGO between relations and predicates (see also Section 2): shows only 106 relations, which is the lowest value in this comparison. In the following, we point out further – Relations – as short term for explicitly defined re- findings regarding the relations of the single KGs. lations – refers to (proprietary) vocabulary defined DBpedia Regarding DBpedia relations we need to on the schema level of a KG. We identify the set distinguish between so-called mapping-based prop- of relations of a KG as the set of those links which erties and non-mapping-based properties. Mapping- based properties are created by extracing the informa- 67This number is based on heuristics. We focused on the 150 most instantiated classes and cut the long tail of classes having only few tion from infoboxes in Wikipedia using manually cre- instances. ated mappings. These mappings are specified in the DB- 24 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO pedia Mappings Wiki.68 Mapping-based properties are criteria are met.71 One of those criteria is that each new contained in the DBpedia ontology and located in the relation is presumably used at least 100 times. This namespace http://dbpedia.org/ontology/. relation proposal process can be mentioned as likely We count 2,819 such relations for the considered DB- reason why in Wikidata in relative terms more relations pedia version 2015-04. Non-mapping-based properties are actually used than in Freebase. (also called “raw infobox properties”) are extracted YAGO For YAGO we measure the small set of 106 from Wikipedia without the help of manually created unique relations. Although relations are curated man- mappings and, hence, without any manual adjustments. ually for YAGO and DBpedia, the size of the relation Therefore, they are generally of lower quality. We count set differs significantly between those KGs. Hoffart et 58,776 such unique relations. They reside in the names- al. [28] mention the following reasons for that: pace http://dbpedia.org/property/. Both 1. Peculiarity of relations: The DBpedia ontology mapping-based and non-mapping-based properties are provides quite many special relations. For in- instantiated in DBpedia with rdf:Property. We ig- stance, there exists the relation dbo:aircraft nore the non-mapping based properties for the calcu- Fighter between dbo:MilitaryUnit and lation of the number of relations, |Pg|, (see Table 2), dbo:MeanOfTransportation. since, in contrast to DBpedia, in YAGO non-mapping 2. Granularity of relations: Relations in the DB- based properties are not instantiated. Note that the pedia ontology are more fine-grained than rela- mapping-based properties and the non-mapping based tions in YAGO. For instance, DBpedia contains the properties in DBpedia are not aligned69 and may over- relations dbo:author and dbo:director, lap until DBpedia version 2016-04.70 whereas in YAGO there is only the generic relation Freebase The high number or Freebase relations can yago:created. be explained by two facts: 1. About a third of all rela- 3. Date specification: The DBpedia ontology intro- tions in Freebase are duplicates in the sense that they are duces several relations for dates. For instance, DB- declared by means of the owl:inverseOf relation pedia contains the relations dbo:birthDate as being inverse of other relations. An example is the re- and dbo:birthYear for birth dates, while in lation freebase:music.artist.album and its YAGO only the relation yago:birthOnDate inverse relation freebase:music.album.artist. is used. Incomplete date specifications – for in- 2. Freebase allowed users to introduce their own rela- stance, if only the year is known – are specified tions without any limits. These relations were originally in YAGO by wildcards (“#”), so that no multiple relations are needed. in each user’s namespace. So-called commons admins 4. Inverse relations: YAGO has no relations ex- were able to approve those relations so that they got plicitly specified as being inverse. In DBpedia, included into the Freebase commons schema. we can find relations specified as inverse such as OpenCyc For OpenCyc we measure 18,028 unique dbo:parent and dbo:child. relations. We can assume that most of them are dedi- 5. Reification: YAGO introduces the SPOTL(X) for- cated to statements on the schema level. mat. This format extends the triple format “SPO“ Wikidata In Wikidata a relatively small set of rela- with a specification of Time, Location and conteXt. tions is provided. Note in this context that, despite the In this way, no contextual relations are necessary fact that Wikidata is curated by a community (just like (such as dbo:distanceToLondon or dbo: Freebase), Wikidata community members cannot insert populationAsOf), which occur if the relations arbitrarily new relations as it was possible in Freebase; are closely aligned to Wikipedia template attribute instead, relations first need to be proposed and then names. get accepted by the community if and only if certain Frequency of the usage of relations. Fig. 5 shows the relative proportions of how often relations are used 68See http://mappings.dbpedia.org/index.php/ per KG, grouped into three classes. Surprisingly, DB- Main_Page, accessed on Nov 4, 2016. pedia and Freebase exhibit a high number of relations 69 For instance, The DBpedia ontology contains which are not used at all on the instance level. In case of dbo:birthName for the name of a person, while the non-mapping based property set contains dbp:name, dbp:firstname, and dbp:alternativeNames. 71See https://www.wikidata.org/wiki/Wikidata: 70For instance, dbp:alias and dbo:alias. Property_proposal, requested on Dec 31, 2016. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 25
studied DBpedia version 2015-04 the set of explicitly 100 DBpedia defined relations (mapping-based properties) and the Freebase set of implicitly defined relations (non-mapping-based dbp:alias OpenCyc properties) overlaps. An example is with dbo:alias. Wikidata 80 Freebase We can observe here a similar picture as YAGO for the set of Freebase relations: With about 785K unique predicates, Freebase exceeds the other KGs by 60 far. Note, however, that 95% of the predicates (around 743K) are used only once. This relativizes the high number. Most of the predicates are keys in the sense 40 of ids and are used for internal modeling (for instance, freebase:key.user.adrianb). OpenCyc In contrast to the 18,028 unique relations, we measure only 164 unique predicates for OpenCyc. Relative occurencies in percent 20 More predicates are presumably used in Cyc. Wikidata We measure more Wikidata predicates than Wikidata relations, since Wikidata predicates are cre- ated by modifying Wikidata relations. An example 0 1-500 >500 are the following triples, which express the statement Number of relations "Barack Obama (wdt:Q76) is a human (wdt:Q5)" by an intermediate node (wdt:Q76S123, abbreviated): Fig. 5. Frequency of the usage of the relations per KG, grouped by (i) zero occurrences, (ii) 1–500 occurrences, and (iii) more than 500 occurrences in the respective KG. wdt:Q76 wdt:P31s wdt:Q76S123. wdt:Q76S123 wdt:P31v wdt:Q5. OpenCyc, 99.2% of the defined relations are never used. We assume that those relations are used only within The relation extension “s” indicates that the RDF term Cyc, the commercial version of OpenCyc. In case of in the object position is a statement. The “v” extension Freebase, only 5% of the relations are used more than allows to refer to a value (in Wikidata terminology). 500 times and about 70% are not used at all. Analo- Besides those extensions, there is “r” to refer to a ref- gously to the discussion regarding the number of Free- erence and the “q” extension to refer to a qualifier. In base relations, we can mention again the high number general, these relation extensions are used for realizing of defined owl:inverseOf relations and the high reification via n-ary relations. For that, intermediate number of users’ relation proposals as reasons for that. nodes are used which represent statements [16]. Predicates YAGO YAGO contains more predicates than DBpe- Ranking regarding predicates. Freebase is here – dia, since infobox attributes from different language 72 like in case of the ranking regarding relations – ranked versions of Wikipedia are aggregated into one KG, first. The lowest number of unique predictes is provided while for DBpedia separate, localized KG versions are by OpenCyc, which exhibits only 165 predicates. All offered for non-English languages. KGs except OpenCyc provide more predicates then re- 5.1.5. Instances and Entities lations. Our single observations regarding the predicate Evaluation method. We distinguish between in- sets are as follows: stances Ig and entities Eg of a KG (cf. Section 2). DBpedia DBpedia is ranked third in terms of the ab- solute numbers of predicates: about 60K predicates are 1. Instances are belonging to classes. They are iden- used in DBpedia. The set of relations and the set of pred- tified by retrieving the subjects of all triples where icates varies considerably here, since also facts are ex- the predicates indicate class affiliations. tracted from Wikipedia info-boxes whose predicates are considered by us as being only implicitly defined and 72The language of each attribute is encoded in the which, hence, occur only as predicates. These are the so- URI, for instance yago:infobox/de/fläche and called non-mapping-based properties. Note that in the yago:infobox/en/areakm. 26 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
10 9 ties in the domain of media and especially song 10 8 release tracks are covered very well in Freebase: 10 7 77% of all entities are in the media domain (see 6 10 Section 5.1.3), out of which 42% are release 5 10 tracks.74 4 10 Due to the large size and the world-wide coverage 3 10 of entities in MusicBrainz, Freebase contains al- 2 10 bums and release tracks of both English and non- 10 1 Number of Instances of Number English languages. For instance, regarding the En- 0 10 glish language, the album “Thriller” from Michael Jackson and its single “Billie Jean” are there, as YAGO well as rather unknown songs from the “Thriller” DBpedia Freebase OpenCyc Wikidata album such as “The Lady in My Life”. Regard- Fig. 6. Number of instances per KG. ing non-English languages, Freebase contains for instance songs and albums from Helene Fischer 2. Entities are real-world objects. This excludes, such as “Lass’ mich in dein Leben” and “Zauber- for instance, instantiated statements for being mond;” also rather unknown songs such as “Hab’ entities. Determining the set of entities is par- den Himmel berührt” can be found. tially tricky: In DBpedia and YAGO entities 2. In case of DBpedia, the English Wikipedia is the are determined as being an instance of the source of information. In the English Wikipedia, class owl:Thing. In Freebase entities are in- many albums and singles of English artists are cov- stances of freebase:common.topic and in ered – such as the album “Thriller” and the single Wikidata instance of wdo:Item. In OpenCyc, “Billie Jean.” Rather unknown songs such as “The cych:Individual corresponds to owl:Thing, Lady in My Life” are not covered in Wikipedia. but not all entities are classified in this way. There- For many non-English artists such as the German fore, we approximately determine the set of en- singer Helene Fischer no music albums and no tities in OpenCyc by manually classifying all singles are contained in the English Wikipedia. In classes having more than 300 instances, including the corresponding language version of Wikipedia 73 at least one entity. In this way, abstract classes (and localized DBpedia version), this information such as cych:ExistingObjectType are ne- is often available (for instance, the album “Zauber- glected. mond” and the song “Lass’ mich in dein Leben”), Ranking w.r.t. the number of instances. Table 2 but not the rather unknown songs such as “Hab’ and Fig. 6 show the number of instances per KG. We den Himmel berührt.” can see that Wikidata comprises the highest number 3. For YAGO, the same situation as for DBpedia of instances (142M) in total and OpenCyc the fewest holds, with the difference that YAGO in addition (242K). imports entities also from the different language Ranking w.r.t. the number of entities. Table 2 versions of Wikipedia and imports also data from shows the ranking of KGs regarding the number of en- sources such as GeoNames. However, the above tities. Freebase contains by far the highest number of mentioned works (“Lass’ mich in dein Leben,” entities (about 49.9M). OpenCyc is at the bottom with “Zaubermond,” and “Hab’ den Himmel berührt”) only about 41K entities. of Helene Fischer are not in the YAGO, although Differences in number of entities. The reason why the song “Lass’ mich in dein Leben” exists in the KGs show quite varying numbers of entities are the the German Wikipedia since May 2014 and al- information sources of the KGs. We illustrate this with though the used YAGO version 3 is based on the 75 the music domain as example: Wikipedia dump of June 2014. Presumably, the YAGO extraction system was unable to extract any 1. Freebase had been created mainly from data im- ports such as from MusicBrainz. Therefore, enti- 74Those release tracks are expressed via freebase :music.release_track. 73For instance, cych:Individual, cych:Movie_CW and 75See http://www.mpi-inf.mpg.de/de/ cych:City. departments/databases-and-information- M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 27
10 4 8 7 3 10 6 5 10 2 4 10 1 3 2 10 0 1
to number of entities of number to 0
Average number of entities of number Average Ratio of number of instances of number of Ratio
YAGO YAGO DBpedia OpenCyc Wikidata DBpedia Freebase OpenCyc Wikidata Freebase
Fig. 7. Average number of entities per class per KG. Fig. 8. Ratio of the number of instances to the number of entities for each KG. types for those entities, so that those entities were discarded. the most frequently occurring instantiations), since en- 4. Wikidata is supported by the community and con- tities are not consistently instantiated in OpenCyc (see tains music albums of English and non-English beginning of Section 5.1.5). artists, even if they do not exist in Wikipedia. An 5.1.6. Subjects and Objects example is the song “The Lady in My Life.” Note, Evaluation method. The number of unique subjects however, that Wikidata does not provide all artist’s and unique objects can be a meaningful KG charac- works such as from Helene Fischer. teristic regarding the link structure within the KG and 5. OpenCyc contains only very few entities in the in comparison to other KGs. Especially interesting are music domain. The reason is that OpenCyc has its differences between the number of unique subjects and focus mainly on common-sense knowledge and the number of unique objects. not so much on facts about entities. We measure the number of unique subjects by count- Average number of entities per class. Fig. 7 shows ing the unique resources (i.e., URIs and blank nodes) on the average number of entities per class, which can be the subject position of N-Triples: Sg := {s | (s, p, o) ∈ written as |Eg|/|Cg|. Obvious is the difference between g}. Furthermore, we measure the number of unique DBpedia and YAGO (despite the similar number of en- objects by counting the unique resources on the ob- tities): The reason for that is that the number of classes ject position of N-Triples, excluding literals: Og := in the DBpedia ontology is small (as created manually) {o | (s, p, o) ∈ g ∧ o ∈ U ∪ B}. Complementary, the lit and in YAGO large (as created automatically). number of literals is given as: Og := {o | (s, p, o) ∈ Comparing number of instances with number of g ∧ o ∈ L}. entities. Comparing the ratio of the number of instances Ranking of KGs regarding number of unique to the number of entities for each KG, Wikidata ex- subjects. The number of unique subjects per KG is pre- poses the highest difference. As reason for that we can sented in Fig. 9. YAGO contains the highest number of state that each statement in Wikidata is modeled as an different subjects, while OpenCyc contains the fewest. instance of wdo:Statement, leading to 74M addi- Ranking of KGs regarding number of unique ob- tional instances. In other KGs such as DBpedia, state- jects. The number of unique objects is also presented in ments are modeled without any dedicated statement Fig. 9. Freebase shows the highest score in this regard, assignment. OpenCyc exposes also a high ratio, since OpenCyc again the lowest. it contains mainly common sense knowledge and not Ranking of KGs regarding the ratio of number as many entities as the other KGs. Furthermore, for our of unique subjects to number of unique objects. The analysis we do not regard 100% of the entities, but only ratios of the number of unique subjects to the number of a large fraction of it (more precisely, the classes with unique objects vary considerably between the KGs (see Fig. 8). We can observe that DBpedia has 2.65 times systems/research/yago-naga/yago/archive/, re- more objects than subjects, while YAGO on the other quested on Dec 31, 2016. side has 19 times more unique subjects than objects. 28 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Table 2 Summary of key statistics.
DBpedia Freebase OpenCyc Wikidata YAGO
Number of triples |(s, p, o) ∈ g| 411 885 960 3 124 791 156 2 412 520 748 530 833 1 001 461 792
Number of classes |Cg| 736 53 092 116 822 302 280 569 751
Number of relations |Pg| 2819 70 902 18 028 1874 106 imp No. of unique predicates |Pg | 60 231 784 977 165 4839 88 736
Number of entities |Eg| 4 298 433 49 947 799 41 029 18 697 897 5 130 031
Number of instances |Ig| 20 764 283 115 880 761 242 383 142 213 806 12 291 250 Avg. number of entities per class |Eg | 5840.3 940.8 0.35 61.9 9.0 |Cg | No. of unique subjects |Sg| 31 391 413 125 144 313 261 097 142 278 154 331 806 927
No. of unique non-literals in obj. pos. |Og| 83 284 634 189 466 866 423 432 101 745 685 17 438 196 lit No. of unique literals in obj. pos. |Og | 161 398 382 1 782 723 759 1 081 818 308 144 682 682 313 508
10 1 2 vs. 3.8M links), leading to a bias of DBpedia towards a unique subjects high number of unique objects. 10 1 0 unique objects 5.1.7. Summary of Key Statistics 8 10 Based on the evaluation results presented in the last 10 6 subsections, we can highlight the following insights:
10 4 1. Triples: All KGs are very large. Freebase is the largest KG in terms of number of triples, while 2 10 OpenCyc is the smallest KG. We notice a corre- 10 0 lation between the way of building up a KG and the size of the KG: automatically created KGs are YAGO typically larger, as the burdens of integrating new DBpedia Freebase OpenCyc Wikidata knowledge become lower. Datasets which have Fig. 9. Number of unique subjects and objects per KG. Note the been imported into the KGs, such as MusicBrainz logarithmic scale on the axis of ordinates. into Freebase, have a huge impact on the number of triples and on the number of facts in the KG. Also the way of modeling data has a great impact The high number of unique subjects in YAGO is sur- on the number of triples. For instance, if n-ary prising and can be explained by the reification style relations are expressed in N-Triples format (as in used in YAGO. Facts are stored as N-Quads in order case of Wikidata), many intermediate nodes need to allow for making statements about statements (for to be modeled, leading to many additional triples instance, storing the provenance information for state- compared to plain statements. Last but not least, ments). To that end, IDs (instead of blank nodes) which the number of supported languages influences the identify the triples are used on the first position of N- number of triples. Triples. They lead to 308M unique subjects, such as 2. Classes: The number of classes is highly varying yago:id_6jg5ow_115_lm6jdp. In the RDF ex- among the KGs, ranging from 736 (DBpedia) up port of YAGO, the IDs which identify the triples are to 300K (Wikidata) and 570K (YAGO). Despite its commented out in order to facilitate the N-Triple for- high number of classes, YAGO contains in relative mat. However, the statements about statements are also terms the most classes which are actually used transformed to triples. In those cases, the IDs identi- (i.e., classes with at least one instance). This can fying the reified statements are in the subject position, be traced back to the fact that heuristics are used leading to such a high number of unique subjects. for selecting appropriate Wikipedia categories as DBpedia contains considerably more owl:sameAs classes for YAGO. Wikidata, in contrast, contains links to external resources than KGs like YAGO (29.0M many classes, but out of them only a small fraction M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 29
is actually used on instance level. Note, however, Table 3 that this is not necessarily a burden. Evaluation results for the KGs regarding the dimension Accuracy. 3. Domains: Although all considered KGs are speci- DB FB OC WD YA fied as crossdomain, domains are not equally dis- tributed in the KGs. Also the domain coverage msynRDF 1 1 1 1 1 among the KGs differs considerably. Which do- msynLit 0.99 1 1 1 0.62 mains are well represented heavily depends on msemT riple 0.99 <1 1 0.99 0.99 which datasets have been integrated into the KGs.
MusicBrainz facts had been imported into Free- Syntactic validity of RDF documents, msynRDF base, leading to a strong knowledge representation Evaluation method. For evaluating the Syntactic va- (77%) in the domain of media in Freebase. In DB- lidity of RDF documents, we dereference the entity pedia and YAGO, the domain people is the largest, “Hamburg” as resource sample in each KG. In case likely due to Wikipedia as data source. of DBpedia, YAGO, Wikidata, and OpenCyc, there 4. Relations and Predicates: Many relations are are RDF/XML serializations of the resource available, rarely used in the KGs: Only 5% of the Freebase which can be validated by the official W3C RDF valida- relations are used more than 500 times and about tor.76 Freebase only provides a Turtle serialization. We 70% are not used at all. In DBpedia, half of the evaluate the syntactic validity of this Turtle document relations of the DBpedia ontology are not used by verifying if the document can be loaded into an RDF at all and only a quarter of the relations is used model of the Apache Jena Framework.77 more than 500 times. For OpenCyc, 99.2% of the Evaluation result. All considered KGs provide syn- relations are not used. We assume that they are tactically valid RDF documents. In case of YAGO and used only within Cyc, the commercial version of Wikidata, the RDF validator declares the used language OpenCyc. codes as invalid, since the validator evaluates language 5. Instances and Entities: Freebase contains by far codes in accordance with ISO-639. The criticized lan- the highest number of entities. Wikidata exposes guage codes are, however, contained in the newer stan- relatively many instances in comparison to the dard ISO 639-3 and actually valid. entities, as each statement is instantiated leading Syntactic validity of literals, msynLit to around 74M instances which are not entities. Evaluation method. We evaluate the Syntactic va- 6. Subjects and Objects: YAGO provides the high- lidity of literals by means of the relations date of est number of unique subjects among the KGs birth, number of inhabitants, and International Stan- and also the highest ratio of the number of unique dard Book Number (ISBN), as those relations cover dif- subjects to the number of unique objects. This is ferent domains – namely, people, cities, and books – due to the fact that N-Quad representations need and as they can be found in all KGs. In general, do- to be expressed via intermedium nodes and that main knowledge is needed for selecting representative YAGO is concentrated on classes which are linked relations, so that a meaningful coverage is guaranteed. by entities and other classes, but which do not pro- Note that OpenCyc is not taken into account for vide outlinks. DBpedia exhibits more unique ob- this criterion: Although OpenCyc comprises around jects than unique subjects, since it contains many 1.1M literals in total, these literals are essentially la- owl:sameAs statements to external entities. bels and descriptions (given via rdfs:label and rdfs:comment), i.e., not aligned to specific data 5.2. Data Quality Analysis types. Hence, OpenCyc has no syntactic invalid literals and is assigned the metric value 1. We now present the results obtained by applying As long as a literal with data type is given, its syntax the DQ metrics introduced in the Sections 3.2 – 3.5 to is verified with the help of the function RDFDatatype the KGs DBpedia, Freebase, OpenCyc, Wikidata, and .isValid(String) of the Apache Jena framework. YAGO. 76 5.2.1. Accuracy See https://w3.org/RDF/Validator/, requested on Mar 2, 2016. The fulfillment degrees of the KGs regarding the 77See https://jena.apache.org/, requested Mar 2, Accuracy metrics are shown in Table 3. 2016. 30 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Thereby, standard data types such as xsd:date can lowest fulfillment degree was obtained for DBpedia. be validated easily, especially if different data types are We found the following findings for the single KGs: In provided.78 If no data type is provided or if the literal Freebase, around 699K ISBN numbers were available. value is of type xsd:String, the literal is evaluated Out of them, 38 were assessed as syntactically incorrect. by a regular expression, which is created manually (see Typical mistakes were too long numbers and wrong below, depending on the considered relation). For each prefixes.82 In case of Wikidata, 18 of around 11K ISBN of the three relations we created a sample of 1M literal numbers were syntactically invalid. However, some in- values per KG, as long as the respective KG contains valid numbers have meanwhile been corrected. This in- so many literals. dicates that the Wikidata community does not only care Evaluation results. All KGs except YAGO per- about inserting new data, but also about curating given formed very well regarding the Syntactic validity of KG data. In case of YAGO, we could only find 400 literals. triples with the relation yago:hasISBN. Seven of the Date of Birth For Wikidata, DBpedia, and Freebase, literals on the object position were syntactically incor- all verified literal values (1M per KG) were syntacti- rect. For DBpedia, we evaluated around 24K literals. cally correct.79 For YAGO, we detected around 519K 7,419 of them were assessed as syntactically incorrect. syntactic errors (given 1M literal values) due to the us- In many cases, comments next to the ISBN numbers in age of wildcards in the date values. For instance, the the info-boxes of Wikipedia led to an inaccurate extrac- birth date of yago:Socrates is specified as “470- tion of data, so that the comments are either extracted ##-##”, which does not correspond to the syntax of as additional facts about ISBN numbers83 or together xsd:date. Obviously, the syntactic invalidity of lit- with the actual ISBN numbers as coherent strings.84 erals is accepted by the YAGO publishers in order to Semantic validity of triples, m keep the number of relations low.80 semT riple Evaluation method. The semantic validity can be re- Number of inhabitants The data types of the literal liably measured by means of a reference data set which values regarding the number of inhabitants were valid (i) contains at least to some degree the same facts as in all KGs. For DBpedia, YAGO, and Wikidata, we in the KG and (ii) which is regarded as some kind of evaluated the syntactic validity of the number of inhab- authority. We decided to use the Integrated Authority itants by checking if xsd:nonNegativeInteger, File (Gemeinsame Normdatei, GND),85 which is an xsd:decimal, and xsd:integer were used as authority file, especially concerning persons and corpo- data types for the typed literals. In Freebase, no data rate bodies, and which was created manually by Ger- type is specified. Therefore, we evaluated the values by man libraries. Due to the focus on persons (especially means of a regular expression which allows only the authors), we decided to evaluate a random sample of decimals 0-9, periods, and commas. person entities w.r.t. the following relations: birth place, ISBN The ISBN is an identifier for books and maga- death place, birth date, and death date. For each of zines. The identifier can occur in various formats: with these relations, the corresponding relations in the KGs or without preceding “ISBN,” with or without delim- 81 were determined. Then, a random sample of 100 person iters, and with 10 or 13 digits. Gupta provided a regu- entities per KG was chosen. For each entity we retrieved lar expression for validating ISBN in its different forms, the facts with the mentioned relations and assessed which we used in our evaluation. All in all, most of manually whether a GND entry exists and whether the the ISBN were assessed as syntactically correct. The values of the relations match with the values in the KG. Evaluation result. We evaluated up to 400 facts per 78In DBpedia, for instance, data for the relation KG and observed only for a few facts some discrep- dbo:birthDate is stored both as xsd:gYear and xsd:date. ancies. For instance, Wikidata states as death date of 79Surprisingly, the Jena Framework assessed data values with a negative year (i.e., B.C.; e.g., “-600” for xsd:gYear) as invalid, despite the correct syntax. 82E.g., we found the 16 digit ISBN 9789780307986931 (cf. 80In order to model the dates to the extent they are known, further freebase:m.0pkny27) and the ISBN 2940045143431 with pre- relations would be necessary, such as using :wasBornOnYear fix 294 instead of 978 (cf. freebase:m.0v3xf7b). with range xsd:gYear, :wasBornOnYearMonth with range 83See dbr:Prince_Caspian. xsd:gYearMonth. 84An example is “ISBN 0755111974 (hardcover edition)” for 81See http://howtodoinjava.com/regex/java- dbr:My_Family_and_Other_Animals. regex-validate-international-standard-book- 85See http://www.dnb.de/EN/Standardisierung/ number-isbns/, requested on Mar 1, 2016. GND/gnd.html, requested on Sep 8, 2016. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 31
“Anton Erkelenz“ (wdt:Q589196) April 24, whereas Table 4 GND states April 25. For DBpedia and YAGO we en- Evaluation results for the KGs regarding the dimension Trustworthiness. countered 3 and for Wikidata 4 errors. Hence, those KGs were evaluated with 0.99. Note that OpenCyc has DB FB OC WD YA no values for the chosen relations and thus evaluates to 1. mgraph 0.5 0.5 1 0.75 0.25 During evaluation we identified the following issues: mfact 0.5 1 0 1 1 mNoV al 0 1 0 1 0 1. For finding the right entry in GND, more informa- tion besides the name of the person is needed. This information is sometimes not given, so that entity of how new data is inserted into the KG and the method disambiguation is in those cases hard to perform. of how existing data is curated. 2. Contrary to assumptions, often either no corre- Evaluation results. The KGs differ considerably sponding GND entry exists or not many facts of w.r.t. this metric. OpenCyc obtains the highest score the GND entity are given. In other words, GND is here, followed by Wikidata. In the following, we pro- incomplete w.r.t. to entities (cf. Population com- vide findings for the single KGs, which are listed by decreasing fulfillment score: pleteness) and relations (cf. Column complete- Cyc is edited (expanded and modified) exclusively by ness). a dedicated expert group. The free version, OpenCyc, 3. Values of different granularity need to be matched, is derived from Cyc and only a locally hosted version such as an exact date of birth against the indication can be modified by the data consumer. of a year only. Wikidata is also curated and expanded manually, but In conclusion, the evaluation of semantic validity is by volunteers of the Wikidata community. Wikidata hard, even if a random sample set is evaluated manually. allows importing data from external sources such as Meaningful differences among the KGs might be re- Freebase.87 However, new data is not just inserted, but vealed only when a very large sample is evaluated, e.g., is approved by the community. by using crowd-sourcing [2,3,48]. Another approach Freebase was also curated by a community of vol- for assessing the semantic validity is presented by Kon- unteers. In contrast to Wikidata, the proportion of data tokostas et al. [34] who propose a test-driven evalu- imported automatically is considerably higher and new ation where test cases are created to evaluate triples data imports were not dependent on community ap- semi-automatically: For instance, an interval specifies provals. the valid height of a person and all triples which lie DBpedia and YAGO The knowledge of both KGs is outside of this interval are evaluated manually. In this extracted from Wikipedia, but DBpedia differs from way, outliers can be easily found but possible wrong YAGO w.r.t. the community involvement: Any user values within the interval are not detected. can engage (i) in mapping the Wikipedia infobox tem- plates to the DBpedia ontology in the DBpedia map- Our findings appear to be consistent with the evalua- 88 tion results of the YAGO developer team for YAGO2, pings wiki and (ii) in the development of the DBpedia where manually assessing 4,412 statements resulted in extraction framework. an accuracy of 98.1%.86 Trustworthiness on statement level We determine the Trustworthiness on statement level 5.2.2. Trustworthiness by evaluating whether provenance information for state- The fulfillment degrees of the KGs regarding the ments is used in the KGs. The picture is mixed: Trustworthiness criteria are shown in Table 4. DBpedia uses the relation prov:wasDerived Trustworthiness on KG level, mgraph From to store the sources of the entities and their state- Evaluation method. Regarding the trustworthiness of a KG in general, we differentiate between the method 87Note that imports from Freebase require the approval of the community (see https://www.wikidata.org/wiki/ Wikidata:Primary_sources_tool). Besides that, there are 86With a weighted averaging of 95%, see http://www.mpi- bots which import automatically (see https://www.wikidata. inf.mpg.de/de/departments/databases-and- org/wiki/Wikidata:Bots/de). information-systems/research/yago-naga/yago/ 88See http://mappings.dbpedia.org/, requested on statistics/, requested on Mar 3, 2016. Mar 3, 2016. 32 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
ments. However, as the source is always the correspond- Table 5 ing Wikipedia article,89 this provenance information Evaluation results for the KGs regarding the dimension Consistency. is trivial and the fulfillment degree is, hence, of rather DB FB OC WD YA formal nature. YAGO uses its own vocabulary to indicate the mcheckRestr 0 1 0 1 0 source of information. Interestingly, YAGO stores per mconClass 0.88 1 <1 1 0.33 statement both the source (via yago:extraction mconRelat 0.99 0.45 1 0.50 0.99 Source; e.g., the Wikipedia article) and the used ex- traction technique (via yago:extractionTech- Indicating unknown and empty values, mNoV al nique; e.g., “Infobox Extractor” or “CategoryMap- This criterion highlights the subtle data model of per”). The number of statements about sources is 161M, Wikidata and Freebase in comparison to the data mod- and, hence, many times over the number of instances in els of the other KGs: Wikidata allows for storing un- the KG. The reason for that is that in YAGO the source known values and empty values (e.g., that “Elizabeth I is stored for each fact. of England” (wdt:Q7207) had no children). However, In Wikidata several relations can be used for refer- in the Wikidata RDF export such statements are only ring to sources, such as “imported from” (wdt:P143), indirectly available, since they are represented via blank “stated in” (wdt:P248), and “reference URL” (wdt: nodes and via the relation owl:someValuesFrom. P854).90 Note that “imported from” relations are used YAGO supports the representation of unknown val- for automatic imports but that statements with such a ues and empty values by providing explicit relations reference are not accepted (“data is not sourced”).91 To for such cases.95 Inexact dates are modeled by means source data, the other relations, “stated in” and “ref- of wildcards (e.g., “1940-##-##”, if only the year is erence URL”, can be used. The number of all stored known). Note, however, the invalidity of such strings references in Wikidata92 is around 971K. Based on the as date literals (see Section 5.2.1). Unknown dates are number of all statements,93 74M, this corresponds to a not supported by YAGO. coverage of around 1.3%. Note, however, that not every 5.2.3. Consistency statement in Wikidata requires a reference according to The fulfillment degrees of the KGs regarding the the Wikidata guidelines. In order to be able to state how Consistency criteria are shown in Table 5. many references are actually missing, a manual evalua- Check of schema restrictions during insertion of new tion would be necessary. However, such an evaluation statements, m would be presumably highly subjective. checkRestr The values of the metric m , indicating re- Freebase uses proprietary vocabulary for represent- checkRestr strictions during the insertion of new statements, are n-ary relations ing provenance: via , which are in Free- varying among the KGs. The web interfaces of Free- base called Compound Value Types (CVT), data from base and Wikidata verify during the insertion of new 94 higher arity can be expressed [44]. statements by the user whether the input is compatible OpenCyc differs from the other KGs in that it uses with the respective data type. For instance, data of the neither an external vocabulary nor a proprietary vocab- relation “date of birth” (wdt:P569) is expected to be ulary for storing provenance information. in a syntactically valid form. DBpedia, OpenCyc and YAGO have no checks for schema restriction during the insertion of new statements. 89E.g., http://en.wikipedia.org/wiki/Hamburg for dbr:Hamburg. Consistency of statements w.r.t. class constraints, 90 All relations are instances of "Wikidata property to indicate a mconClass source" (wdt:Q18608359). Evaluation method. For evaluating the consis- 91See https://www.wikidata.org/wiki/Property: P143, requested Mar 3, 2016. tency of class constraints we considered the relation 92This is the number of instances of wdo:Reference. owl:disjointWith, since this is the only rela- 93This is the number of instances of wdo:Statement. tion which is used by more than half of the consid- 94E.g., for a statement with the relation freebase:location .statistical_region.population, the source can be stored via freebase:measurement_unit.dated_inte 95E.g., freebase:freebase.valuenotation.has_ ger.source. no_value. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 33 ered KGs. We only focused on direct instantiations Table 6 here: if there is, for instance, the triple (dbo:Plant, Evaluation results for the KGs regarding the dimension Relevancy. owl:disjointWith, dbo:Animal), then there DB FB OC WD YA must not be a resource which is instantiated both as dbo:Plant and dbo:Animal. mRanking 0 1 0 1 0 Evaluation results. We obtained mixed results here. Only Freebase, OpenCyc, and Wikidata perform very ertyTypeTime can represent a year or an exact date). well.96 On the talk pages of Wikidata relations users can indi- Freebase and Wikidata do not specify any constraints cate the allowed values of relations via "One of" state- with owl:disjointWith. Hence, those two KGs ments.97 Since "One of" statements are only listed on have no inconsistencies w.r.t. class restrictions and we the property talk pages and since not only entity types can assign the metric value 1 to them. In case of Open- but also concrete instances are used as "One of" values, Cyc, 5 out of the 27,112 class restrictions are incon- we do not consider those statements here. sistent. DBpedia contains 24 class constraints. Three DBpedia obtains the highest measured fulfillment out of them are inconsistent. For instance, over 1,200 score w.r.t. consistency of rdfs:range statements. instances exist which are both a dbo:Agent and a An example for a range inconsistency is that the relation dbo:Place. YAGO contains 42 constraints, dedi- dbo:birthDate requires a data type xsd:date; cated mainly for WordNet classes, which are mostly in about 20% of those relations, the data type xsd: inconsistent. gYear is used, though. YAGO, Freebase, and OpenCyc contain range incon- Consistency of statements w.r.t. relation constraints, sistencies primarily since they specify designated data mconRelat types via range relations which are not consistently Evaluation method Here we considered the rela- used on the instance level. For instance, YAGO spec- tions rdfs:range and owl:FunctionalProper ifies proprietary data types such as yago:yagoURL ty, as those are used in more than every second con- and yago:yagoISBN. On the instance level, how- sidered KG. rdfs:range specifies the expected type ever, either no data type is used or the unspecific data of an instance on the object position of a triple, while type xsd:string. owl:FunctionalProperty indicates that a rela- FunctionalProperty. The restriction indicated by tion should only be used at most once per resource. We owl:FunctionalProperty is used by all KGs only took datatype properties into account for this eval- except Wikidata. On the talk pages about the rela- uation, since consistencies regarding object properties tions in Wikidata, users can specify the cardinality would require to distinguish Open World assumption restriction via setting the relation to "single"; how- and Closed World assumption. ever, this is not part of the Wikidata data model. Evaluation results. In the following, we consider The other KGs mostly comply with the usage re- the fulfillment degree for the relation constraints strictions of owl:FunctionalProperty. Note- rdfs:range and owl:FunctionalProperty worthy is that in Freebase 99.9% of the inconsis- separately. In Table 5, we show the average of the fulfill- tencies obtained here are caused by the usages of ment scores of each KG regarding rdfs:range and the relations freebase:type.object.name and owl:FunctionalProperty. Note that the num- freebase:common.notable_for.display_ bers of evaluated relation constraints varied from KG to name. KG, depending on how many relation constraints were 5.2.4. Relevancy available per KG. The fulfillment degrees of the KGs regarding the Range. Wikidata does not use any rdfs:range Relevancy criteria are shown in Table 6. restrictions. Within the Wikidata data model, there is wdo:propertyType, but this indicates not the ex- Creating a ranking of statements, mRanking act allowed data type of a relation (e.g., wdo:prop Only Wikidata supports the modeling of a ranking of statements: Each statement is ranked with “pre-
96Note that the sample size varies among the KGs (depend- ing on how many owl:disjointWith statements are available 97See https://www.wikidata.org/wiki/Category: per KG). Therefore, inconsistencies measured on a small set of Properties_with_one-of_constraints for an overview; owl:disjointWith facts become more visible. requested on Jan 29, 2017. 34 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Table 7 due to the characteristics of how information is stored Evaluation results for the KGs regarding the dimension and extracted from Wikipedia. Completeness. 1. Classes: The DBpedia ontology was created man- DB FB OC WD YA ually and covers all domains well. However, it is incom- plete in the details and therefore appears unbalanced. mcSchema 0.91 0.76 0.92 1 0.95 For instance, within the domain of plants the DBpe- mcColumn 0.40 0.43 0 0.29 0.33 dia ontology does not use the class "tree" but the class mcP op 0.93 0.94 0.48 0.99 0.89 "ginko," which is a subclass of trees. We can mention as reason for such gaps in the modeling the fact that mcP op (short) 1 1 0.82 1 0.90 the ontology is created by means of the most frequently m (long) 0.86 0.88 0.14 0.98 0.88 cP op used infobox templates in Wikipedia. 2. Relations: Relations are considerably well cov- ferred rank” (wdo:PreferredRank), “normal rank” ered in the DBpedia ontology. Some missing relations (wdo:NormalRank), or “deprecated rank” (wdo: or modeling failures are due to the Wikipedia infobox DeprecatedRank). The "preferred rank" corre- characteristics. For example, to represent the gender of sponds to the up-to-date value or the consensus of the a person the existing relation foaf:gender seems Wikidata community w.r.t. this relation. Freebase does to fit. However, it is only modeled in the ontology as not provide any ranking of statements, entities, or re- belonging to the class dbo:language and not used lations. However, the meanwhile shutdown Freebase on instance level. Note that the gender of a person is of- Search API provided a ranking for resources.98 ten not explicitly mentioned in the Wikipedia infoboxes but implicitly mentioned in the category names (for 5.2.5. Completeness instance, "American male singers"). While DBpedia The fulfillment degrees of the KGs regarding the does not exploit this knowledge, YAGO does use it and Completeness criteria are shown in Table 7. provides facts with the relation yago:hasGender.
Schema completeness, mcSchema Freebase. Freebase shows a very ambivalent schema Evaluation method. Since a gold standard for eval- completeness. On the one hand, Freebase targets rather uating the Schema completeness of the considered KGs the representation of facts on instance level than the has not been published, we built one on our own. This representation of classes and their hierarchy. On the gold standard is available online.99 It is based on the other hand, Freebase provides a vast amount of rela- data set used in Section 5.1.3, where we needed as- tions, leading to a very good coverage of the requested signments of classes to domains, and comprises of 41 relations. classes as well as 22 relations. It is oriented towards the 1. Classes: Freebase lacks a class hierarchy and sub- domains people, media, organizations, geography, and classes of classes are often in different domains (for in- biology. The classes in the gold standard were aligned stance, the classes freebase:music.artist and to corresponding WordNet synsets (using WordNet ver- sportsmen freebase:sports.pro_athlete are sion 3.1) and were grouped into main classes. logically a subclass of the class people freebase: Evaluation results. Generally, Wikidata performs person.people but not explicitly stated as such), optimal; also DBpedia, OpenCyc, and YAGO exhibit which makes it difficult to find suitable sub- and su- results which can be judged as acceptable for most use perclasses. Noteworthy, the biology domain contains cases. Freebase shows considerable room for improve- no classes. This is due to the fact that classes are rep- 100 101 ment concerning the coverage of typical cross-domain resented as entities, such as tree and ginko. The classes and relations. The results in more detail are as ginko tree is not classified as tree, but by the generic follows: class freebase:biology.oganism_classifi DBpedia. DBpedia shows a good score regarding cation. Schema completeness and its schema is mainly limited 2. Relations: Freebase exhibits all relations requested by our gold standard. This is not surprising, given the vast amount of available relations in Freebase (see Sec- 98 See https://developers.google.com/freebase/ tion 5.1.4 and Table 2). v1/search-cookbook#scoring-and-ranking, re- quested on Mar 4, 2016. 99See http://km.aifb.kit.edu/sites/knowledge- 100Freebase ID freebase:m.07j7r. graph-comparison/, requested on Jan 29, 2017. 101Freebase ID freebase:m.0htd3. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 35
OpenCyc. In total, OpenCyc exposes a quite high Table 8 Schema completeness scoring. This is due to the fact Metric values of mcCol for single class-relation-pairs. that OpenCyc has been created manually and has its Relation DB FB OC ED YA focus on generic and common-sense knowledge. Person–birthdate 0.48 0.48 0 0.70 0.77 1. Classes: The ontology of OpenCyc covers both generic and specific classes such as cych:Social Person–sex – 0.57 0 0.94 0.64 Group and cych:LandTopographicalFeature. Book–author 0.91 0.93 0 0.82 0.28 We can state that OpenCyc is complete with respect to the considered classes. Book–ISBN 0.73 0.63 – 0.18 0.01 2. Relations: OpenCyc lacks some relations of the gold standard such as the number of pages or the ISBN combinations102 were created based on our gold stan- of books. dard created for measuring the Schema completeness. Wikidata. According to our evaluation, Wikidata is It was ensured that only those relations were selected complete both with respect to classes and relations. for a given class for which a value typically exists for 1. Classes: Besides frequently used generic classes that class. For instance, we did not include the death such as “human” (wdt:Q5) also very specific classes date as potential relation for living people. exist such as “landform” (wdt:Q271669) in the sense Evaluation results. In general, no KG yields a met- of a geomorphologial unit with over 3K instances. ric score of over 0.43. As visible in Table 8, KGs often 2. Relations: In particular remarkable is that Wiki- have some specific class-relation-pairs which are well data covers all relations of the gold standard, even represented on instance level, while the rest of the pairs though it has extremely less relations than Freebase. are poorly represented. The well-represented pairs pre- Thus, the Wikidata methodology to let users propose sumably originate either from column-complete data new relations, to discuss about their outreach, and fi- sets which were imported (cf. MusicBrainz in case of nally to approve or disapprove the relations, seems to Freebase), or from user edits focusing primarily on facts be appropriate. about entities of popular classes such as people. We YAGO. Due to its concentration on modeling classes, notice the following observations with respect to the YAGO shows the best overall Schema completeness single KGs: fulfillment score among the KGs. DBpedia. DBpedia fails regarding the relation sex for 1. Classes: To create the set of classes in YAGO, instances of class Person, since it does not contain the Wikipedia categories are extracted and connected such a relation in its ontology. If we considered the non- to WordNet synsets. Since also our gold standard is mapping-based property dbp:gender instead (not already aligned to WordNet synsets, we can measure a defined in the ontology), we would gain a coverage of full completeness score for YAGO classes. only 0.25% (about 5K people). We can note, hence, that 2. Relations: The YAGO schema does not contain the extraction of data out of the Wikipedia categories many unique but rather abstract relations, which can would be a further fruitful data source for DBpedia. be understood in different senses. The abstract rela- Freebase. Freebase surprisingly shows a very high tion names make it often difficult to infer the mean- coverage (92.7%) of the authors of books, given the ba- ing. The relation yago:wasCreatedOnDate, for sic population of 1.7M books. Note, however, that there instance, can be used reasonably for both the founda- are not only books modeled under freebase:book. tion year of a company and for the publication date book but also entities of other types, such as a descrip- of a movie. DBpedia, in contrast, provides the rela- tion of the Lord of Rings (see freebase:m.07bz5). tion dbp:foundationYear. Often the meaning of Also the coverage of ISBN for books is quite high YAGO relations is only fully understood after consider- (63.4%). ing the associated classes, using domain and range of OpenCyc. OpenCyc breaks ranks, as mostly no val- the relations. Expanding the YAGO schema by further, ues for the considered relations are stored in this KG. It more fine-grained relations appears reasonable. 102 Column completeness, m The selection of class-relation-pairs was depending on the fact cColumn which class-relation-pairs were available per KG. Hence, the choice Evaluation method. For evaluating KGs w.r.t. Col- is varying from KG to KG. Also, note that less class-relation-pairs umn completeness, for each KG 25 class-relation- were used if no 25 pairs were available in the respective KG. 36 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO contains mainly taxonomic knowledge and only thinly Evaluation results. All KGs except OpenCyc show spread instance facts. good evaluation results. Since also Wikidata exhibits Wikidata. Wikidata achieves a high coverage of birth good evaluation results, the population degree appar- dates (70.3%) and of gender (94.1%), despite the high ently does not depend on the age or the insertion method 103 number of 3M people. of the KG. Fig. 10 additionally depicts the population YAGO. YAGO obtains a coverage of 63.5% for gen- completeness for the single domains for each KG. In der relations, as it, in contrast to DBpedia, extracts this the following, we firstly present our findings for well- implicit information from Wikipedia. known entities, before we secondly go into the details Population completeness, mcP op of rather unknown entities. Evaluation method. In order to evaluate the Popu- Well-known entities: Here, all considered KGs lation completeness, we need a gold standard consist- achieve good results. DBpedia, Freebase, and Wikidata ing of a basic entity population for each considered are complete w.r.t. the well-known entities in our gold 104 KG. This gold standard, which is available online, standard. YAGO lacks some well-known entities, al- was created on the basis of our gold standard used though some of them are represented in Wikipedia. One for evaluating the Schema completeness and the Col- reason for this fact is that those Wikipedia entities do umn completeness. For its creation, we selected five not get imported into YAGO for which a WordNet class classes from each of the five domains and determined exists. For instance, there is no “Great White Shark” two well-known entities (called "short head") and two entity, only the WordNet class yago:wordnet_ rather unknown entities (called "long tail") for each of great_white_shark_101484850. those classes. The exact entity selection criteria are as Not-well-known entities: follows. First of all, not very surpris- ing is the fact that all KGs show a higher degree of com- 1. The well-known entities were chosen without tem- pleteness regarding well-known entities than regard- poral and location-based restrictions. To take the ing rather unknown entities, as the KGs are oriented most popular entities per domain, we used quan- towards general knowledge and not domain-specific titative statements. For instance, to select well- knowledge. Secondly, two things are in particular pe- known athletes, we ranked athletes by the number culiar concerning long-tail entities in the KGs: While of won olympic medals; to select the most popu- most of the KGs obtain a score of about 0.88, Wiki- lar mountains, we ranked the mountains by their data deflects upwards and OpenCyc deflects strongly heights. 2. To select the rather unknown entities, we consid- downwards. ered entities associated to both Germany and a Wikidata exhibits a very high Population complete- specific year. For instance, regarding the athletes, ness degree for long tail entities. This is a result from we selected German athletes active in the year the central storage of interwiki links between different 2010, such as Maria Höfl-Riesch. The selection Wikimedia projects (especially between the different of rather unknown entities in the domain of biol- Wikipedia language versions) in Wikidata: A Wikidata ogy is based on the IUCN Red List of Threatened entry is added to Wikidata as soon as a new entity is Species105.106 added in one of the many Wikipedia language versions. Note, however, that in this way English-language labels Selecting four entities per class and five classes per domain resulted in 100 entities to be used for evaluating for the entities are often missing. We measure that only the Population completeness. about 54.6% (10.2M) of all Wikidata resources have an English label. OpenCyc exhibits a poor population degree score of 103These 3M instances form about 18.5% of all instances in Wiki- data. See https://www.wikidata.org/wiki/Wikidata: 0.14 for long-tail entities. OpenCyc’s sister KGs Cyc Statistics, requested on Nov 7, 2016. and ResearchCyc are apparently considerably better 104See http://km.aifb.kit.edu/sites/knowledge- covered with entities [36], leading to higher Population graph-comparison/, requested on Jan 29, 2017. completeness scores. 105See http://www.iucnredlist.org, requested on Apr 2, 2016. 5.2.6. Timeliness 106Note that selecting entities by their importance or popularity is hard in general and that also other popularity measures such as the The evaluation results concerning the dimension PageRank scores may be taken into account. Timeliness are presented in Table 9. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 37
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3 People 0.2 Media Organizations 0.1 Geography Biology 0 DBpedia Freebase OpenCyc Wikidata YAGO
Fig. 10. Population completeness regarding the different domains per KG.
Table 9 Freebase had been updated continuously until its Evaluation results for the KGs regarding the dimension Timeliness. close-down and is not updated anymore. DB FB OC WD YA OpenCyc has been updated less than once per year. The last OpenCyc version dates from May 2012.109 To mF req 0.5 0 0.25 1 0.25 the best of our knowledge, Cyc and OpenCyc, respec- mV alidity 0 1 0 1 1 tively, are developed further, but no exact date of the
mChange 0 1 0 0 0 next version is known. Wikidata provides the highest fulfillment degree for this criterion. Modifications in Wikidata are via browser Timeliness frequency of the KG, mF req and via HTTP URI dereferencing immediately visible. Evaluation results. The KGs are very diverse re- Hence, Wikidata falls in the category of continuous garding the frequency in which the KGs are updated, updates. Besides that, an RDF export is provided on ranging from a score of 0 for Freebase (not updated any a roughly monthly basis (either via the RDF export more) to 1 for Wikidata (updates immediately visible webpage110 or via own processing using the Wikidata and retrievable). Note that the Timeliness frequency of toolkit111). the KG can be a crucial point and a criterion for exclu- YAGO has been updated less than once per year. sion in the process of choosing the right KG for a given YAGO3 was published in 2015, YAGO2 in 2011, and setting [17]. In the following, we outline some charac- the interim version YAGO2s in 2013. A date of the next teristics of the KGs with respect to their up-to-dateness: release has not been published. DBpedia is created about once to twice a year and is not modified in the meantime. From September Specification of the validity period of statements, 2013 until November 2016, six DBpedia versions have mV alidity been published.107 Besides the static DBpedia, DBpe- Evaluation results. Although representing the va- dia live108 has been continuously updated by tracking lidity period of statements is obviously reasonable for changes in Wikipedia in real-time. However, it does not many relations (for instance, the president’s term of provide the full range of relations as DBpedia.
109See http://sw.opencyc.org/, requested on Nov 8, 107These versions are DBpedia 3.8, DBpedia 3.9, DBpedia 2014, 2016. DBpedia 2015-04, DBpedia 2015-10, and DBpedia 2016-04. Always 110See http://tools.wmflabs.org/wikidata- the latest DBpedia version is published online for dereferencing. exports/rdf/exports/, requested on Nov 23, 2016. 108See http://live.dbpedia.org/, requested on Mar 4, 111See https://github.com/Wikidata/Wikidata- 2016. Toolkit, requested on Nov 8, 2016. 38 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Table 10 fer from KG to KG: DBpedia, for instance, uses Evaluation results for the KGs regarding the dimension Ease of rdfs:comment and dcelements:description, understanding. while Freebase provides freebase:common.topic DB FB OC WD YA .description.112 Evaluation result. For all KGs the rule applies that mDescr 0.70 0.97 1 <1 1 in case there is no label available, usually there is mLang 1 1 0 1 1 also no description available. The current metric could muSer 1 1 0 1 1 therefore (without significant restrictions) be applied to rdfs:label occurrences only. muURI 1 0.5 1 0 1 YAGO, Wikidata, and OpenCyc contain a label for almost every entity. In Wikidata, the entities without office), specifying the validity period of statements is any label are of experimental nature and are most likely in several KGs either not possible at all or only rudi- not used.113 mentary performed. Surprisingly, DBpedia shows a relatively low cov- DBpedia and OpenCyc do not realize any specifi- erage w.r.t. labels and descriptions (only 70.4%). Our cation possibility. In YAGO, Freebase, and Wikidata manual investigations suggest that relations with higher the temporal validity period of statements can be spec- arity are modeled by means of intermediate nodes ified. In YAGO, this modeling possibility is made which have no labels.114 available via the relations yago:occursSince, yago:occursUntil, and yago:occursOnDate. Labels in multiple languages, mLang Wikidata provides the relations “start time” (wdt:P580) Evaluation method. Here we measure whether the and “end time” (wdt:P582). In Freebase, Compound KGs contain labels (rdfs:label) in other languages Value Types (CVTs) are used to represent relations with than English. This is done by means of the language higher arity [44]. As part of this representation, validity annotations of literals such as “@de” for literals in periods of statements can be specified. An example is German. “Vancouver’s population in 1997.” Evaluation results. DBpedia provides labels in 13 languages. Further languages are provided in the lo- Specification of the modification date of statements, calized DBpedia versions. YAGO integrates statements mChange of the different language versions of Wikipedia into Evaluation results. The modification date of state- one KG. Therefore, it provides labels in 326 different ments can only be specified in Freebase but not in the languages. Freebase and Wikidata also provide a lot of other KGs. Together with the criteria on Timeliness, languages (244 and 395 languages, respectively). Con- this reflects that the considered KGs are mostly not trary to the other KGs, OpenCyc only provides labels sufficiently equipped with possibilities for modeling in English. temporal aspects within and about the KG. Coverage of languages. We also measured the cov- In Freebase the date of the last review of a fact can be erage of selected languages in the KGs, i.e., the extent represented via the relation freebase:freebase. to which entities have an rdfs:label with a specific valuenotation.is_reviewed. In the DBpedia language annotation.115 Our evaluation shows that DB- ontology the relation dcterms:modified is used pedia, YAGO, and Freebase achieve a high coverage to state the date of the last revision of the DBpedia with more than 90% regarding the English language. In ontology. When dereferencing a resource in Wikidata, contrast to those KGs, Wikidata shows a relative low the latest modification date of the resource is returned via schema:dateModified. This, however, does 112Human-readable resource descriptions may also be represented not hold for statements. Thus Wikidata is evaluated by other relations [15]. However, we focused on those relations which with 0, too. are commonly used in the considered KGs. 113For instance, wdt:Q5127809 represents a game fo the Nin- 5.2.7. Ease of Understanding tendo Entertainment System, but there is no further information for Description of resources, mDescr an identification of the entity available. 114 Evaluation method. We measured the extent to E.g., dbr:Nayim links via dbo:CareerStation to 10 entities of his carrier stations. which entities are described. Regarding the labels, 115Note that literals such as rdfs:label do not necessarily have we considered rdfs:label for all KGs. Regard- language annotations. In those cases, we assume that no language ing the descriptions, the corresponding relations dif- information is available. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 39 coverage regarding the English language of only 54.6%, Table 11 but a coverage of over 30% for further languages such Evaluation results for the KGs regarding the dimension Interoperability. as German and French. Wikidata is, hence, not only the most diverse KG in terms of languages, but has also the DB FB OC WD YA highest coverage regarding non-English languages. mReif 0.5 0.5 0.5 0 0.5 Understandable RDF serialization, muSer miSerial 1 0 0.5 1 1 The provisioning of understandable RDF serializa- m 0.61 0.11 0.41 0.68 0.13 tions in the context of URI dereferencing leads to a bet- extV oc ter understandability for human data consumers. DB- mpropV oc 0.15 0 0.51 >0 0 pedia, YAGO, and Wikidata provide N-Triples and N3/Turtle serializations. Freebase, in contrast, only are stored via n-ary relations.117 YAGO stores facts as provides a Turtle serialization. OpenCyc only uses N-Quads in order to be able to store meta information RDF/XML, which is regarded as not easily understand- of facts like provenance information. When the quads able by humans. are loaded in a triple store, the IDs referring to the single statements are ignored and quads are converted Self-describing URIs, muURI We can observe two different paradigms of URI us- into triples. In this way, most of the statements are still age: On the one hand, DBpedia, OpenCyc, and YAGO usable without the necessity to deal with reification. rely on descriptive URIs and therefore achieve the full Blank nodes are non-dereferencable, anonymous re- fulfillment degree. In DBpedia and YAGO, the URIs sources. They are used by the Wikidata and OpenCyc of the entities are determined by the corresponding En- data model. glish Wikipedia article. The mapping to the English Provisioning of several serialization formats, miSerial Wikipedia is thus trivial. In case of OpenCyc, two RDF DBpedia, YAGO, and Wikidata fulfill the criterion of exports are provided: one using opaque and one us- Provisioning several RDF serialization formats to the ing self-describing URIs. The self-describing URIs are full extent, as they provide data in RDF/XML and sev- thereby derived from the rdfs:label values of the eral other serialization formats during the URI derefer- resources. encing. In addition, DBpedia and YAGO provide fur- On the other hand, Wikidata and Freebase (the latter ther RDF serialization formats (e.g., JSON-LD, Micro- in part) rely on opaque URIs: Wikidata uses Q-IDs data, and CSV) via their SPARQL endpoints. Freebase for resources ("items" in Wikidata terminology) and is the only KG providing RDF only in Turtle format. P-IDs for relations. Freebase uses self-describing URIs only partially, namely, opaque M-IDs for entities and Using external vocabulary, mextV oc self-describing URIs for classes and relations.116 Evaluation method. This criterion indicates the ex- tent to which external vocabulary is used. For that, for 5.2.8. Interoperability each KG we divide the occurrence number of triples The evaluation results of the dimension Interoper- with external relations by the number of all relations in ability are presented in Table 11. this KG.
Avoiding blank nodes and RDF reification, mReif Evaluation results. DBpedia uses 37 unique exter- Reification allows to represent further information nal relations from 8 different vocabularies, while the about single statements. In conclusion, we can state that other KGs mainly restrict themselves to the external DBpedia, Freebase, OpenCyc, and YAGO use some vocabularies RDF, RDFS, and OWL. form of reification. However, none of the considered Wikidata reveals a high external vocabulary ratio, KGs uses the RDF standard for reification. Wikidata too. We can mention two obvious reasons for that fact: makes extensive use of reification: every relation is 1. Information in Wikidata is provided in a huge variety stored in the form of an n-ary relation. In case of DB- of languages, leading to 85M rdfs:label and 140M pedia and Freebase, in contrast, facts are predominantly schema:description literals. 2. Wikidata makes stored as N-Tripels and only relations of higher arity extensive use of reification. Out of the 140M triples used for instantiations via rdf:type, about 74M (i.e.,
116E.g., freebase:music.album for the class "music al- bums" and freebase:people.person.date_of_birth 117See Section 5.1.1 for more details w.r.t. the influence of reifica- for the relation "day of birth". tion on the number of triples. 40 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO about the half) are taken for instantiations of statements, Table 12 i.e., for reification. Evaluation results for the KGs regarding the dimension Accessibility. DB FB OC WD YA Interoperability of proprietary vocabulary, mpropV oc Evaluation method. This criterion determines the ex- mDeref 1 1 0.44 0.41 1 tent to which URIs of proprietary vocabulary are linked m <1 0.73 <1 <1 1 to external vocabulary via equivalence relations. For Avai each KG, we measure which classes and relations mSP ARQL 1 1 0 1 0 118 are linked via owl:sameAs, owl:equivalent mExport 1 1 1 1 1 Class wdt:P1709 owl:equiv (in Wikidata: ), and mNegot 0.5 1 0 1 0 alentProperty (in Wikidata wdt:P1628) to ex- mHT MLRDF 1 1 1 1 0 ternal vocabulary. Note that other relations such as m rdf:subPropertyOf could be taken into account; Meta 1 0 0 0 1 however, in this work we only consider equivalency relations. lent external classes. This may be due to the high num- Evaluation results. In general, we obtained low ful- ber of classes in Wikidata in general. Regarding the fillment scores regarding this criterion. OpenCyc shows relations, Wikidata provides links in particular to FOAF the highest value. We achieved the following single and schema.org and achieves here a linking coverage findings: of 2.1%. Although this is low, frequently used relations Regarding its classes, DBpedia reaches a relative are linked.121 high interlinking degree of about 48.4%. Classes are YAGO contains around 553K owl:equivalent thereby linked to FOAF, Wikidata, schema.org and Class links to classes within the DBpedia namespace DUL.119 Regarding its relations, DBpedia links to Wiki- dby:. However, as YAGO classes (and their hierarchy) data and schema.org.120 Only 6.3% of the DBpedia relations are linked to external vocabulary. were imported also into DBpedia (using the namespace http://dbpedia.org/class/yago/ Freebase only provides owl:sameAs links in the ), we do form of a separate RDF file, but these links are only on not count those owl:equivalentClass links in instance level. Thus, the KG is evaluated with 0. YAGO as external links for YAGO. In OpenCyc, about half of all classes exhibit at least 5.2.9. Accessibility one external linking via owl:sameAs. Internal links The evaluation results of the dimension Accessibility to resources of sw.cyc.com, the commercial ver- are presented in Table 12. sion of OpenCyc, were ignored in our evaluation. The considered classes are mainly linked to FOAF, UM- Dereferencing possibility of resources, mDeref BEL, DBpedia, and linkedmdb.org, the relations mainly Evaluation method. We measured the dereferenc- to FOAF, DBpedia, Dublin Core Terms, and linked- ing possibilities of resources by trying to dereference mdb.org. The relative high linking degree of OpenCyc URIs containing the fully-qualified domain name of can be attributed to dedicated approaches of linking the KG. For that, we randomly selected 15K URIs in OpenCyc to other KGs (see, e.g., Medelyan et al. [38]). the subject, predicate, and object position of triples in Regarding the classes, Wikidata provides links each KG. We submitted HTTP requests with the HTTP mainly to DBpedia. Considering all Wikidata classes, accept header field set to application/rdf+xml only 0.1% of all Wikidata classes are linked to equiva- in order to perform content negotiation. Evaluation results. In case of DBpedia, OpenCyc, 118OpenCyc uses owl:sameAs both on schema and instance and YAGO, all URIs were dereferenced successfully level. This is appropriate as the OWL primer states "The built-in and returned appropriate RDF data, so that they fulfilled OWL property owl:sameAs links an individual to an individual" this criterion completely. For DBpedia, 45K URIs were as well as "The owl:sameAs statements are often used in defining mappings between ontologies", see https://www.w3.org/TR/ analyzed, for OpenCyc only around 30K due to the 2004/REC-owl-ref-20040210/#sameAs-def (requested small number of unique predicates. We observed almost on Feb 4, 2017). 119See http://www.ontologydesignpatterns.org/ ont/dul/DUL.owl, requested on Jan 11, 2017. 121Frequently used relations with stated equivalence to external 120E.g., dbo:birthDate is linked to wdt:P569 and relations are, e.g., wdt:P31, linked to rdf:type, and wdt:P279, schema:birthDate. linked to rdfs:subClassOf. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 41 the same picture for YAGO, namely no notable errors provided by a Virtuoso server,124 the Wikidata SPARQL during dereferencing. endpoint via Blazegraph.125 Freebase and OpenCyc do For Wikidata, which contains also not that many not provide an official SPARQL endpoint. However, an unique predicates, we analyzed around 35K URIs. Note endpoint for the MQL query language for the Freebase that predicates which are derived from relations using a KG was available. suffix (e.g., the suffix "s" as in wdt:P1024s is used Especially regarding the Wikidata SPARQL endpoint for predicates referring to a statement) could not be we observed access restrictions: The maximum execu- dereferenced at all. Furthermore, the blank nodes used tion time per query is set to 30 seconds, but there is no for reification cannot be dereferenced. limitation regarding the returning number of rows. How- Regarding Freebase, mainly all URIs on subject ever, the front-end of the SPARQL endpoint crashed in and object position of triples could be dereferenced. case of large result sets with more than 1.5M rows. Al- Some resources were not resolvable even after multi- though public SPARQL endpoints need to be prepared ple attempts (HTTP server error 503; e.g., freebase: for inefficient queries, the time limit of Wikidata may m.0156q). Surprisingly, server errors also appeared impede the execution of reasonable queries. while browsing the website freebase.com, so that data Provisioning of an RDF export, mExport was partially not available. Regarding the predicate po- All considered KGs provide RDF exports as down- sition, many URIs are not dereferencable due to server loadable files. The format of the data differs from KG errors (HTTP 503) or due to unknown URIs (HTTP to KG. Mostly, data is provided in N-Triples and Turtle 404). Note that if a large number of Freebase requests format. are performed, an API key from Google is necessary. In our experiments, the access was blocked after a few Support of content negotiation, mNegot thousand requests. Hence, we can point out that without We measure the support of content negotiation re- an API key the Freebase KG is only usable to a limited garding the serialization formats RDF/XML, N3/Turtle, extent. and N-Triples. OpenCyc does not provide any content negotiation; only RDF/XML is supported as content Availability of the KG, mAvai type. Therefore, OpenCyc does not fulfill the criterion Evaluation method. We measured the availability of supporting content negotiation. of the officially hosted KGs with the monitoring service The endpoints for DBpedia, Wikidata, and YAGO Pingdom.122 For each KG, an uptime test was set up, correctly returned the appropriate RDF serialization which checked the availability of the resource "Ham- format and the corresponding HTML representation burg" as representative resource for successful URI re- of the tested resources. Freebase does currently not solving (i.e., returning the status code HTTP 200) ev- provide any content negotiation and only the content ery minute over the time range of 60 days (Dec 18, type text/plain is returned. 2015–Feb 15, 2016). Noteworthy is also that regarding the N-Triples seri- Evaluation result. While the other KGs showed al- alization YAGO and DBpedia require the accept header most no outages and were again online after some min- text/plain and not applicationn-triples. utes on average, YAGO outages took place frequently This is due to the usage of Virtuoso as endpoint. For DB- and lasted on average 3.5 hours.123 In the given time pedia, the forwarding to http://dbpedia.org/ range, four outages took longer than one day. Based on data/[resource].ntriples does not work; in- these insights, we recommend to use a local version of stead, the HTML representation is returned. Therefore, YAGO for time-critical queries. the KG is evaluated with 0.5.
Availability of a public SPARQL endpoint, mSP ARQL Linking HTML sites to RDF serializations, mHT MLRDF The SPARQL endpoints of DBpedia and YAGO are All KGs except OpenCyc interlink the HTML represen- tations of resources with the corresponding RDF repre- sentations by means of
Table 13 Table 14 Evaluation results for the KGs regarding the dimension License. Evaluation results for the KGs regarding the dimension Interlinking.
DB FB OC WD YA DB FB OC WD YA
mmacLicense 1 0 0 1 0 mInst 0.25 0 0.38 0 (.09) 0.31 mURIs 0.93 0.91 0.89 0.96 0.96 type="{content type}" href="{URL}"/> in the HTML header. 5.2.11. Interlinking The evaluation results of the dimension Interlinking Provisioning of metadata about the KG, mmeta are shown in Table 14. For this criterion we analyzed if KG metadata is available, such as in the form of a VoID file.126 DBpedia Linking via owl:sameAs, mInst integrates the VoID vocabulary directly in its KG127 and Evaluation method. Given all owl:sameAs triples provides information such as the SPARQL endpoint in each KG, we queried all those subjects thereof which 133 URL and the number of all triples. OpenCyc reveals are instances but neither classes nor relations and the current KG version number via owl:version where the resource in the object position of the triple is Info. For YAGO, Freebase, and Wikidata no meta an external source, i.e., not belonging to the namespace information could be found. of the KG. Evaluation result. OpenCyc and YAGO achieve the 5.2.10. License best results w.r.t. this metric, but DBpedia has by far The evaluation results of the dimension License are the most instances with at least one owl:sameAs link. shown in Table 13. We can therefore confirm the statement by Bizer et al. [12] that DBpedia has established itself as a hub in the Provisioning machine-readable licensing information, Linked Data cloud. mmacLicense In DBpedia, there are about 5.2M instances with at DBpedia and Wikidata provide licensing informa- least one owl:sameAs link. Links to localized DBpe- tion about their KG data in machine-readable form. For dia versions (e.g., de.dbpedia.org) were counted DBpedia, this is done in the ontology via the predi- as internal links and, hence, not considered here. In 128 cate cc:license linking to CC-BY-SA and GNU total, one-fourth of all instances have at least one 129 Free Documentation License (GNU FDL). Wikidata owl:sameAs link. embeds licensing information during the dereferenc- In Wikidata, neither owl:sameAs links are pro- ing of resources in the RDF document by linking with vided nor a corresponding proprietary relation is avail- cc:license to the license CC0.130 YAGO and Free- able. Instead, Wikidata uses for each linked data set base do not provide machine-readable licensing infor- a proprietary relation (called "identifier") to indicate mation. However, their data is published under the li- equivalence. For example, the M-ID of a Freebase in- cense CC-BY.131 OpenCyc embeds licensing informa- stance is stored via the relation “Freebase identifier” tion into the RDF document during dereferencing, but (wdt:P646) as literal value (e.g., "/m/01x3gpk"). not in machine-readable form.132 So far, links to 426 different data sources are maintained in this way. 126See https://www.w3.org/TR/void/, requested on Apr Although the equivalence statements in Wikidata can 7, 2016. be used to generate corresponding owl:sameAs state- 127See http://dbpedia.org/void/page/Dataset, re- ments and although the stored identifiers are provided quested on Mar 5, 2016. in the Browser interface as hyperlinks, there are no gen- 128 See http://creativecomons.org/licenses/by- uine owl:sameAs links available. Hence, Wikidata is sa/3.0/, requested on Feb 4, 2017. 129See http://www.gnu.org/copyleft/fdl.html, re- evaluated with 0. If we view each equivalence relation quested on Feb 4, 2017. as owl:sameAs relation, we would obtain around 130See http://creativecomons.org/publicdomain/ 12.2M instances with owl:sameAs statements. This zero/1.0/, requested on Feb 4, 2017. corresponds to 8.6% of all instances. If we consider 131See http://createivecommons.org/licenses/ by/3.0, requested on Feb 4, 2017. 132License information is provided as plain text among further 133The interlinking on schema level is already covered by the information with the relation rdfs:comment. criterion Interoperability of proprietary vocabulary. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 43 only entities instead of instances (since there are many Web resources. Here we were able to resolve around instances due to reification), we obtain a coverage of 95.5% without errors. 65%. Note, however, that, although the linked resources Noticeable is that DBpedia and OpenCyc contain provide relevant content, the resources are not always many owl:sameAs links to URIs whose domains do RDF documents, but instead HTML web pages. There- not exist anymore.136 One solution for such invalid links fore, we cannot easily subsume all "identifiers" (equiv- might be to remove them if they have been invalid for a alence statements) under owl:sameAs. certain time span. YAGO has around 3.6M instances with at least one 5.2.12. Summary of Results owl:sameAs link. However, most of them are links We now summarize the results of the evaluations to DBpedia based on common Wikipedia articles. If presented in this section. those links are excluded, YAGO contains mostly links to GeoNames and would be evaluated with just 0.01. 1. Syntactic validity of RDF documents: All KGs In case of OpenCyc, links to Cyc,134 the commercial provide syntactically valid RDF documents. version of OpenCyc, were considered as being internal. 2. Syntactic validity of Literals: In general, the KGs Still, OpenCyc has the highest fulfillment degree with achieve good scores regarding the Syntactic valid- around 40K instances with at least one owl:sameAs ity of literals. Although OpenCyc comprises over link. As mentioned earlier, the relative high linking 1M literals in total, these literals are mainly labels degree of OpenCyc can be attributed to dedicated ap- and descriptions which are not formatted in a spe- proaches of linking OpenCyc to other KGs.135 cial format. For YAGO, we detected about 519K syntactic errors (given 1M literal values) due to the Validity of external URIs, m URIs usage of wildcards in the date values. Obviously, Regarding the dimension Accessibility, we already the syntactic invalidity of literals is accepted by analyzed the dereferencing possibility of resources in the publishers in order to keep the number of rela- the KG namespace. Now we analyze the links to exter- tions low. In case of Wikidata, some invalid literals nal URIs. such as the ISBN have been corrected in newer Evaluation method. External links include owl: versions of Wikidata. This indicates that knowl- sameAs links as well as links to non-RDF-based Web edge in Wikidata is curated continuously. For DB- resources (e.g., via foaf:homepage). We measure pedia, comments next to the values to be extracted errors such as timouts, client errors (HTTP response (such as ISBN) in the infoboxes of Wikipedia led 4xx), and server errors (HTTP response 5xx). to inaccurately extracted values. Evaluation result. The external links are in most of 3. Semantic validity of triples: All considered KGs the cases valid for all KGs. All KGs obtain a metric scored well regarding this metric. This shows that value between 0.89 and 0.96. KGs can be used in general without concerns re- DBpedia stores provenance information via the re- garding the correctness. Note, however, that eval- prov:wasDerivedFrom lation . Since almost all uating the semantic validity of facts is very chal- links refer to Wikipedia, 99% of the resources are avail- lenging, since a reliable ground truth is needed. able. 4. Trustworthiness on KG level: Based on the way of Freebase achieves high metric values here, since how data is imported and curated, OpenCyc and it contains owl:sameAs links mainly to Wikipedia. Wikidata can be trusted the most. Also Wikipedia URIs are mostly resolvable. 5. Trustworthiness on statement level: Here, espe- OpenCyc contains mainly external links to non-RDF- cially good values are achieved for Freebase, Wiki- based Web resources to wikipedia.org and w3.org. data, and YAGO. YAGO stores per statement both YAGO also achieves high metric values, since it pro- the source and the extraction technique, which is vides owl:sameAs links only to DBpedia and Geo- unique among the KGs. Wikidata also supports to Names, whose URIs do not change. store the source of information, but only around For Wikidata the relation "reference URL" (wdt: 1.3% of the statements have provenance informa- P854), which states provenance information among tion attached. Note, however, that not every state- other relations, belongs to the links linking to external
136E.g., http://rdfabout.com, http://www4.wiwiss. 134I.e., sw.cyc.com fu-berlin.de/factbook/, and http://wikicompany. 135See Interoperability of proprietary vocabulary in sec. 5.2.8. org (requested on Jan 11, 2017). 44 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
ment in Wikidata requires a reference and that it each class are on average frequently used by all of is hard to evaluate which statements lack such a those class instances. We can name data imports reference. as one reason for it. 6. Using unknown and empty values: Wikidata and 13. Population completeness: Not very surprising is Freebase support the indication of unknown and the fact that all KGs show a higher degree of com- empty values. pleteness regarding well-known entities than re- 7. Check of schema restrictions during insertion of garding rather unknown entities. Especially Wiki- new statements: Since Freebase and Wikidata are data shows an excellent performance for both well- editable by community members, simple consis- known and rather unknown entities. tency checks are made during the insertion of new 14. Timeliness frequency of the KG: Only Wikidata facts in the user interface. provides the highest fulfillment degree for this 8. Consistency of statements w.r.t. class constraints: criterion, as it is continuously updated and as the Freebase and Wikidata do not specify any class changes are immediately visible and queryable by constraints via owl:disjointWith, while the users. other KGs do. 15. Specification of the validity period of statements: 9. Consistency of statements w.r.t. relation con- In YAGO, Freebase, and Wikidata the temporal straints: The inconsistencies of all KGs regarding validity period of statements (e.g., term of office) the range indications of relations are mainly due to can be specified. inconsistently used data types (e.g., xsd:gYear 16. Specification of the modification date of state- is used instead of xsd:Date). ments: Only Freebase keeps the modification dates Regarding the constraint of functional proper- of statements. Wikidata provides the modification ties, the relation owl:FunctionalProperty date of the queried resource during URI derefer- is used by all KGs except Wikidata; in most cases encing. the KGs comply with the usage restrictions of this 17. Description of resources: YAGO, Wikidata, and relation. OpenCyc contain a label for almost every entity. 10. Creating a ranking of statements: Only Wikidata Surprisingly, DBpedia shows a relatively low cov- supports a ranking of statements. This is in partic- erage w.r.t. labels and descriptions (only 70.4%). ular worthwhile in case of statements which are Manual investigations suggest that the interme- only temporally limited valid. diate node mapping template is the main reason 11. Schema completeness: Wikidata shows the highest for that. By means of this template, intermediate degree of schema completeness. Also for DBpe- nodes are introduced and instantiated, but no la- dia, OpenCyc, and YAGO we obtain results which bels are provided for them.137 are presumably acceptable in most cross-domain 18. Labels in multiple languages: YAGO, Freebase, use cases. While DBpedia classes were sometimes and Wikidata support hundreds of languages re- missing in our evaluation, the DBpedia relations garding their stored labels. Only OpenCyc con- were covered considerably well. OpenCyc lacks tains labels merely in English. While DBpedia, some relations of the gold standard, but the classes YAGO, and Freebase show a high coverage re- of the gold standard were existing in OpenCyc. garding the English language, Wikidata does not While the YAGO classes are peculiar in the sense have such a high coverage regarding English, but that they are connected to WordNet synsets, it is instead covers other languages to a considerable remarkable that YAGO relations are often kept extent. It is, hence, not only the most diverse KG very abstract so that they can be applied in differ- in terms of languages, but also the KG which con- ent senses. Freebase shows considerable room for tains the most labels for languages other than En- improvement concerning the coverage of typical glish. cross-domain classes and relations. Note that Free- 19. Understandable RDF serialization: DBpedia, base classes are belonging to different domains. Wikidata, and YAGO provide several understand- Hence, it is difficult to find related classes if they are not in the same domain. 137An example is dbr:Volkswagen_Passat_(B1), 12. Column completeness: DBpedia and Freebase which has dbo:engine statements to the intermediate nodes show the best column completeness values, i.e., in Volkswagen_Passat_(B1)__1, etc., representing different those KGs the predicates used by the instances of engine variations. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 45
able RDF serialization formats. Freebase only base we measured a quite considerable amount provides the understandable format RDF/Turtle. of dereferencing failures due to server errors and OpenCyc relies only on RDF/XML, which is con- unknown URIs. Note also that Freebase required sidered as being not easily understandable for hu- an API key for a large amount of requests. mans. 26. Availability of the KG: While all other KGs 20. Self-describing URIs: We can find mixed paradigms showed almost no outages, YAGO shows a note- regarding the URI generation: DBpedia, YAGO, worthy instability regarding its online availability. and OpenCyc rely on descriptive URIs, while We measured around 100 outages for YAGO in Wikidata and Freebase (in part; classes and rela- a time interval of 8 weeks, taking on average 3.5 tions are identified with self-describing URIs) use hours. generic IDs, i.e., opaque URIs. 27. Provisioning of public SPARQL endpoint: DBpe- 21. Avoiding blank nodes and RDF reification: DB- dia, Wikidata, and YAGO provide a SPARQL end- pedia, Wikidata, YAGO, and Freebase are the point, while Freebase and OpenCyc do not. Note- KGs which use reification, i.e., which formulate worthy is that the Wikidata SPARQL endpoint has statements about statements. There are different a maximum execution time per query of 30 sec- ways of implementing reification [27]. DBpedia, onds. This might be a bottleneck for some queries. Wikidata, and Freebase use n-ary relations, while 28. Provisioning of an RDF export: RDF exports are YAGO uses N-Quads, creating so-called named available for all KGs and are provided mostly in graphs. N-Triples and Turtle format. 22. Provisioning of several serialization formats: 29. Support of content negotiation: DBpedia, Wiki- Many KGs provide RDF in several serialization data, and YAGO correctly return RDF data based formats. Freebase is the only KG providing data on content negotiation. Both OpenCyc and Free- in the serialization format RDF/Turtle only. base do not support any content negotiation. While 23. Using external vocabulary: DBpedia and Wiki- OpenCyc only provides data in RDF/XML, Free- data show high degrees of external vocabulary base only returns data with text/plain as con- usage. In DBpedia the RDF, RDFS, and OWL tent type. vocabularies are used. Wikidata has a high ex- 30. Linking HTML sites to RDF serializations: All ternal vocabulary ratio, since there exist many KGs except OpenCyc interlink the HTML rep- language labels and descriptions (modeled via resentations of resources with the corresponding rdfs:label and schema:description). RDF representations. Also, due to instantiations of statements with 31. Provisioning of KG metadata: Only DBpedia and wdo:Statement for reification purposes, the OpenCyc integrate metadata about the KG in external relation rdf:type is used a lot. some form. DBpedia has the VoID vocabulary in- 24. Interoperability of proprietary vocabulary: We tegrated, while OpenCyc reveals the current KG obtained low fulfillment scores regarding this cri- version as machine-readable metadata. terion. OpenCyc shows the highest value. We 32. Provisioning machine-readable licensing informa- can mention as reason for that the fact that tion: Only DBpedia and Wikidata provide licens- half of all OpenCyc classes exhibit at least one ing information about their KG data in machine- owl:sameAs link. readable form. While DBpedia has equivalence statements to ex- 33. Interlinking via owl:sameAs: OpenCyc and ternal classes for almost every second class, only YAGO achieve the best results w.r.t. this met- 6.3% of all relations have equivalence relations to ric, but DBpedia has by far the most instances relations outside the DBpedia namespace. with at least one owl:sameAs link. Based on Wikidata shows a very low interlinking degree the resource interlinkage, DBpedia is justifiably of classes to external classes and of relations to called Linked Data hub. Wikidata does not provide external relations. owl:sameAs links but stores identifiers as liter- 25. Dereferencing possibility of resources: Resources als that could be used to generate owl:sameAs in DBpedia, OpenCyc, and YAGO can be derefer- links. enced without considerable issues. Wikidata uses 34. Validity of external URIs: The links to exter- predicates derived from relations that are not deref- nal Web resources are for all KGs valid in erencable at all, as well as blank nodes. For Free- most cases. DBpedia and OpenCyc contain many 46 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Step 1: Requirements Analysis Description of the use case: The publisher BBC - Identifying the preselection criteria P wants to enrich news articles with fact sheets providing - Assigning a weight wi to each DQ criterion ci ∈ C relevant information about musicians mentioned in the articles. In order to obtain more details about the mu- Step 2: Preselection based on the Preselection Criteria sicians, the user can leave the news section and access - Manually selecting the KGs GP that fulfill the preselection criteria P the musicians section where detailed information is pro- vided, including a short description, a picture, the birth Step 3: Quantitative Assessment of the KGs date, and the complete discography for each musician. - Calculating the DQ metric mi(g) for each DQ criterion ci ∈ C For being able to integrate the musicians information - Calculating the fulfillment degree h(g) for each KG g ∈ GP - Determining the KG g with the highest fulfillment degree h(g) into the articles and to enable such a linking, editors shall tag the article based on a controlled vocabulary. Step 4: Qualitative Assessment of the Result The KG Recommendation Framework can be applied - Assessing the selected KG g w.r.t. qualitative aspects as follows:
- Comparing the selected KG g with other KGs in G P 1. Requirements analysis: Fig. 11. Proposed process for using our KG recommendation frame- – Preselection criteria: According to the sce- work. nario description [33], the KG in question should (i) be actively curated and (ii) con- owl:sameAs links to RDF documents on do- tain an appropriate amount of media enti- mains which do not exist anymore; those links ties. Given these two criteria, a satisfactory could be deleted. and up-to-date coverage of both old and new musicians is expected. – Weighting of DQ criteria: Based on the pre- 6. KG Recommendation Framework selection criteria, an example weighting of the DQ metrics for our use case is given in We now propose a framework for selecting the Table 15. Note that this is only one exam- most suitable KG (or a set of suitable KGs) for a ple configuration and the assignment of the given concrete setting, based on a given set of KGs weights is subjective to some degree. Given G = {g1, ..., gn}. To use this framework, the user needs the preselection criteria, the criterion Timeli- to go through the steps depicted in Fig. 11. ness frequency of the KG and the criteria of In Step 1, the preselection criteria and the weights the DQ dimension Completeness are empha- for the criteria are specified. The preselection criteria sized. Furthermore, the criteria Dereferenc- can be both quality criteria or general criteria and need ing possibility of resources and Availability to be selected dependent on the use case. The Timeli- of the KG are important, as the KG shall be ness frequency of the KG is an example for a quality available online, ready to be queried.138 criterion. The license under which a KG is provided 2. Preselection: Freebase and OpenCyc are not con- (e.g., CC0 license) is an example for a general criterion. sidered any further, since Freebase is not being up- After weighting the criteria, in Step 2 those KGs are dated anymore and since OpenCyc contains only neglected which do not fulfill the preselection criteria. around 4K entities in the media domain. In Step 3, the fulfillment degrees of the remaining KGs 3. Quantitative Assessment: The overall fulfillment are calculated and the KG with the highest fulfillment score for each KG is calculated based on the for- degree is selected. Finally, in Step 4 the result can be as- mula presented in Section 3.1. The result of the sessed w.r.t. qualitative aspects (besides the quantitative quantitative KG evaluation is presented in Ta- assessments using the DQ metrics) and, if necessary, an ble 15. By weighting the criteria according to alternative KG can be selected for being applied for the the constraints, Wikidata achieves the best rank, given scenario. closely followed by DBpedia. Based on the quan- Use case application. In the following, we show titative assessment, Wikidata is recommended by how to use the KG recommendation framework in a the framework. particular scenario. The use case is based on the usage of DBpedia and MusicBrainz for the project BBC Music 138We assume that in this use case rather the dereferencing of as described in [33]. HTTP URIs than the execution of SPARQL queries is desired. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 47
Table 15 Framework with an example weighting which would be reasonable for a user setting as given in [33].
Dimension Metric DBpedia Freebase OpenCyc Wikidata YAGO Example of User
Weighting wi
Accuracy msynRDF 1 1 1 1 1 1
msynLit 0.994 1 1 1 0.624 1
msemT riple 0.990 0.995 1 0.993 0.993 1
Trustworthiness mgraph 0.5 0.5 1 0.75 0.25 0
mfact 0.5 1 0 1 1 1
mNoV al 0 1 0 1 0 0
Consistency mcheckRestr 0 1 0 1 0 0
mconClass 0.875 1 0.999 1 0.333 0
mconRelat 0.992 0.451 1 0.500 0.992 0
Relevancy mRanking 0 1 0 1 0 1
Completeness mcSchema 0.905 0.762 0.921 1 0.952 1
mcCol 0.402 0.425 0 0.285 0.332 2
mcP op 0.93 0.94 0.48 0.99 0.89 3
Timeliness mF req 0.5 0 0.25 1 0.25 3
mV alidity 0 1 0 1 1 0
mChange 0 1 0 0 0 0
Ease of understanding mDescr 0.704 0.972 1 0.9999 1 1
mLang 1 1 0 1 1 0
muSer 1 1 0 1 1 0
muURI 1 0.5 1 0 1 1
Interoperability mReif 0.5 0.5 0.5 0 0.5 0
miSerial 1 0 0.5 1 1 1
mextV oc 0.61 0.108 0.415 0.682 0.134 1
mpropV oc 0.150 0 0.513 0.001 0 1
Accessibility mDeref 1 0.437 1 0.414 1 2
mAvai 0.9961 0.9998 1 0.9999 0.7306 2
mSP ARQL 1 0 0 1 1 1
mExport 1 1 1 1 1 0
mNegot 0.5 0 0 1 1 0
mHT MLRDF 1 1 0 1 1 0
mMeta 1 0 1 0 0 0
Licensing mmacLicense 1 0 0 1 0 0
Interlinking mInst 0.251 0 0.382 0 0.310 3
mURIs 0.929 0.908 0.894 0.957 0.956 1 Unweighted Average 0.683 0.603 0.496 0.752 0.625 Weighted Average 0.701 0.493 0.556 0.714 0.648 48 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
4. Qualitative Assessment: The high population com- crete criteria which can be applied to any KG in the pleteness in general and the high coverage of enti- Linked Open Data cloud. Table 16 shows which of ties in the media domain in particular give Wiki- the metrics introduced in this article have already been data advantage over the other KGs. Furthermore, used to some extent in existing literature. In summary, Wikidata does not require that there is a Wikipedia related work mainly proposed generic guidelines for article for each entity. Thus, missing Wikidata en- publishing Linked Data [26], introduced DQ criteria tities can be added by the editors directly and are with corresponding metrics (e.g., [20,30]) and criteria then available immediately. without metrics (e.g., [40,29]). 27 of the 34 criteria in- The use case requires to retrieve also detailed infor- troduced in this article have been introduced or sup- mation about the musicians from the KG, such as a ported in one way or another in earlier works. The re- short descripion and a discography. DBpedia tends maining seven criteria, namely Trustworthiness on KG to store more of that data, especially w.r.t. discogra- level, mgraph, Indicating unknown and empty values, phy. A specialized database like MusicBrainz pro- mNoV al, Check of schema restrictions during insertion vides even more data about musicians than DBpe- of new statements, mcheckRestr, Creating a ranking dia, as it is not limited to the Wikipedia infoboxes. of statements, mRanking, Timeliness frequency of the While DBpedia does not provide any links to Mu- KG, mF req, Specification of the validity period of state- sicBrainz, Wikidata stores around 120K equiva- ments, mV alidity, and Availability of the KG, mAvai, lence links to MusicBrainz that can be used to pull have not been proposed so far, to the best of our knowl- more data. In conclusion, Wikidata, especially in edge. In the following, we present more details of single the combination with MusicBrainz, seems to be existing approaches for Linked Data quality criteria. an appropriate choice for the use case. In this case, Pipino et al. [40] introduce the criteria Schema com- the qualitative assessment confirms the result of pleteness, Column completeness and Population com- the quantitative assessment. pleteness in the context of databases. We introduce those metrics for KGs and apply them, to the best of The use case shows that our KG recommendation our knowledge, the first time on the KGs DBpedia, framework enables users to find the most suitable KG Freebase, OpenCyc, Wikidata, and YAGO. and is especially useful in giving an overview of the OntoQA [45] introduces criteria and corresponding most relevant criteria when choosing a KG. However, metrics that can be used for the analysis of ontologies. applying our framework to the use case also showed Besides simple statistical figures such as the average of that, besides the quantitative assessment, there is still instances per class, Tartir et al. introduce also criteria a need for a deep understanding of commonalities and and metrics similar to our DQ criteria Description of difference of the KGs in order to make an informed resources, mDescr, and Column completeness, mcCol. choice. Based on a large-scale crawl of RDF data, Hogan et al. [29] analyze quality issues of published RDF data. Later, Hogan et al. [30] introduce further criteria and 7. Related Work metrics based on Linked Data guidelines for data pub- lishers [26]. Whereas Hogan et al. crawl and analyze 7.1. Linked Data Quality Criteria many KGs we analyze a selected set of KGs in more detail. Zaveri et al. [49] provide a conceptual framework for Heath et al. [26] provide guidelines for Linked Data quality assessment of linked data based on quality cri- but do not introduce criteria or metrics for the assess- teria and metrics which are grouped into quality dimen- ment of Linked Data quality. Still, the guidelines can be sions and categories and which are based on the frame- easily translated into relevant criteria and metrics. For work of Wang et al. [47]. Our framework is also based instance, "Do you refer to additional access methods" on Wang’s dimensions and extended by the dimensions leads to the criteria Provisioning of public SPARQL Consistency [11], Licensing and Interlinking [49]. Fur- endpoint, mSP ARQL, and Provisioning of an RDF ex- thermore, we reintroduce the dimensions Trustworthi- port, mExport. Also, "Do you map proprietary vocabu- ness and Interoperability as a collective term for multi- lary terms to other vocabularies?" leads to the criterion ple dimensions. Interoperability of proprietary vocabulary, mpropV oc. Many published DQ criteria and metrics are rather Metrics that are based on the guidelines of Heath et al. abstract. We, in contrast, selected and developed con- can also be found in other frameworks [30,20]. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 49
Table 16 Overview of related work regarding data quality criteria for KGs.
DQ Metric [40] [45] [29] [26] [20] [22] [30] [48] [2] [34]
msynRDF XX msynLit XXXX msemT riple XXXX mfact XX mconClass XXX mconRelat XXXXXX
mcSchema XX mcCol XXXX mcP op XX mChange XX
mDescr XXXX mLang X muSer X muURI X mReif XXX miSerial X mextV oc XX mpropV oc X
mDeref XXXX mSP ARQL X mExport XX mNegot XXX mHT MLRDF X mMeta XXX mmacLicense XXX mInst XXX mURIs XX
Flemming [20] introduces a framework for the qual- SWIQA[22] is a quality assessment framework intro- ity assessment of Linked Data quality. This framework duced by Fürber et al. that introduces criteria and met- measures the Linked Data quality based on a sample of rics for the dimensions Accuracy, Completeness, Timeli- a few RDF documents. Based on a systematic literature ness, and Uniqueness. In this framework, the dimension review, criteria and metrics are introduced. Flemming Accuracy is divided into Syntactic validity and Sematic introduces the criteria Labels in multiple languages, validity as proposed by Batini et al. [6]. Furthermore, mLang, and Validity of external URIs, mURIs, the first the dimension Completeness comprises Schema com- time. The framework is evaluated on a sample of RDF pleteness, Column completeness and Population com- documents of DBpedia. In contrast to Flemming, we pleteness, following Pipino et al. [40]. In this article, evaluate the whole KG DBpedia and also four other we make the same distinction, but in addition distin- widely used KGs. guish between RDF documents, RDF triples, and RDF 50 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO literals for evaluating the Accuracy, since we consider and their subclasses. In our case, we cannot use this ap- RDF KGs. proach, since Freebase has no hierarchy. Hassanzadeh TripleCheckMate [35] is a framework for Linked et al. analyze the coverage of domains by listing the Data quality assessment using a crowdsourcing-approach most frequent classes with the highest number of in- for the manual validation of facts. Based on this ap- stances as a table. This gives only little overview of the proach, Zaveri et al. [48] and Acosta et al. [2,3] analyze covered domains, since instances can belong to multi- both syntactic and semantic accuracy as well as the ple classes in the same domain, such as dbo:Place consistency of data in DBpedia. and dbo:PopulatedPlace. For determining the Kontokostas et al. [34] present the test-driven evalu- domain coverages of KGs for this article, we there- ation framework RDFUnit for assessing Linked Data fore adapt the idea of Hassanzadeh et al. by manu- quality. This framework is inspired by the paradigm ally mapping the most frequent classes to domains and of test-driven software development. The framework deleting duplicates within the domains. That means, introduces 17 SPARQL templates of tests that can be if an instance is instantiated both as dbo:Place used for analyzing KGs w.r.t. Accuracy and Consis- and dbo:PopulatedPlace, the instance will be tency. Note that those tests can also be used for eval- counted only once in the domain geography. uating external constraints that exist due to the usage of external vocabulary. The framework is applied by Kontokostas et al. on a set of KGs including DBpedia. 8. Conclusion
7.2. Comparing KGs by Key Statistics Freely available knowledge graphs (KGs) have not been in the focus of any extensive comparative study so Duan et al. [14], Tartir [45], and Hassanzadeh [25] far. In this survey, we defined a range of aspects accord- can be mentioned as the most similar related work re- ing to which KGs can be analyzed. We analyzed and garding the evaluation of KGs using the key statistics compared DBpedia, Freebase, OpenCyc, Wikidata, and presented in Section 5.1. YAGO along these aspects and proposed a framework Duan et al. [14] analyze the structuredness of data in as well as a process to enable readers to find the most DBpedia, YAGO2, UniProt, and in several benchmark suitable KG for their settings. data sets. To that end, the authors use simple statistical key figures that are calculated based on the correspond- ing RDF dumps. In contrast to that approach, we use References SPARQL queries to obtain the figures, thus not limiting ourselves to the N-Tripel serialization of RDF dump [1] M. Acosta, E. Simperl, F. Flöck, and M. Vidal. HARE: A files. Duan et al. claim that simple statistical figures are Hybrid SPARQL Engine to Enhance Query Answers via Crowdsourcing. In Proceedings of the 8th International not sufficient to gain fruitful findings when analyzing Conference on Knowledge Capture, K-CAP 2015, pages the structuredness and differences of RDF datasets. The 11:1–11:8. ACM, 2015. authors therefore propose in addition a coherence met- [2] M. Acosta, A. Zaveri, E. Simperl, D. Kontokostas, S. Auer, and ric. Accordingly, we analyze not only simple statisti- J. Lehmann. Crowdsourcing linked data quality assessment. In cal key figures but further analyze the KGs w.r.t. data The Semantic Web–ISWC 2013, pages 260–276. Springer, 2013. [3] M. Acosta, A. Zaveri, E. Simperl, D. Kontokostas, F. Flöck, quality, using 34 DQ metrics. and J. Lehmann. Detecting Linked Data Quality Issues via Tartir et al. [45] introduce with the system OntoQA Crowdsourcing: A DBpedia Study. Semantic Web, 2016. metrics that can be used for analyzing ontologies. More [4] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and precisely, it can be measured to which degree the Z. Ives. DBpedia: A Nucleus for a Web of Open Data. In schema level information is actually used on instance Proceedings of the 6th International Semantic Web Conference and 2nd Asian Semantic Web Conference, ISWC 2007/ASWC level. An example of such a metric is the class richness, 2007, pages 722–735. Springer, 2007. defined as the number of classes with instances divided [5] S. Auer, J. Lehmann, A.-C. Ngonga Ngomo, and A. Zaveri. by the number of classes without instances. SWETO, Introduction to Linked Data and Its Lifecycle on the Web. In TAP, and GlycO are used as showcase ontologies. Reasoning Web. Semantic Technologies for Intelligent Data Tartir et al. [45] and Hassanzadeh et al. [25] analyze Access, volume 8067 of Lecture Notes in Computer Science, pages 1–90. Springer Berlin Heidelberg, 2013. how domains are covered by KGs on both schema and [6] C. Batini, C. Cappiello, C. Francalanci, and A. Maurino. instance level. For that, Tartir et al. introduce the mea- Methodologies for Data Quality Assessment and Improvement. sure importance as the number of instances per class ACM Comput. Surv., 41(3):16:1–16:52, July 2009. M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO 51
[7] S. Bechhofer, F. van Harmelen, J. Hendler, I. Horrocks, D. L. (ECIS2011), volume 15, page 19, 2011. McGuinness, and P. F. Patel-Schneider. OWL Web Ontology [23] R. Guns. Tracing the origins of the Semantic Web. Journal of Language Reference. https: the American Society for Information Science and Technology, //www.w3.org/TR/2004/REC-owl-ref-20040210, 64(10):2173–2181, 2013. 2004. [Online; accessed 06-Apr-2016]. [24] H. Halpin, P. J. Hayes, J. P. McCusker, D. L. McGuinness, and [8] T. Berners-Lee. Linked Data. http: H. S. Thompson. The Semantic Web – ISWC 2010: 9th //www.w3.org/DesignIssues/LinkedData.html, International Semantic Web Conference, ISWC 2010, Shanghai, 2006. [Online; accessed 28-Feb-2016]. China, chapter When owl:sameAs Isn’t the Same: An Analysis [9] T. Berners-Lee. Linked Data Is Merely More Data. of Identity in Linked Data, pages 305–320. Springer Berlin http://www.w3.org/DesignIssues/LinkedData.html, 2006. Heidelberg, Berlin, Heidelberg, 2010. [Online; accessed 28-02-2016]. [25] O. Hassanzadeh, M. J. Ward, M. Rodriguez-Muro, and [10] T. Berners-Lee, J. Hendler, and O. Lassila. The Semantic Web. K. Srinivas. Understanding a Large Corpus of Web Tables Scientific American, 284(5):29–37, 5 2001. Through Matching with Knowledge Bases – An Empirical [11] C. Bizer. Quality-Driven Information Filtering in the Context Study. In Proceedings of the 10th International Workshop on of Web-Based Information Systems. VDM Publishing, 2007. Ontology Matching collocated with the 14th International [12] C. Bizer, J. Lehmann, G. Kobilarov, S. Auer, C. Becker, Semantic Web Conference, ISWC 2015, 2015. R. Cyganiak, and S. Hellmann. DBpedia–A crystallization [26] T. Heath and C. Bizer. Linked data: Evolving the web into a point for the Web of Data. Web Semantics: science, services global data space. Synthesis lectures on the semantic web: and agents on the world wide web, 7(3):154–165, 2009. theory and technology, 1(1):1–136, 2011. [13] X. Dong, E. Gabrilovich, G. Heitz, W. Horn, N. Lao, [27] D. Hernández, A. Hogan, and M. Krötzsch. Reifying RDF: K. Murphy, T. Strohmann, S. Sun, and W. Zhang. Knowledge What Works Well With Wikidata? In Proceedings of the 11th Vault: A Web-Scale Approach to Probabilistic Knowledge International Workshop on Scalable Semantic Web Knowledge Fusion. In Proceedings of the 20th ACM SIGKDD Base Systems co-located with 14th International Semantic Web International Conference on Knowledge Discovery and Data Conference, pages 32–47, 2015. Mining, KDD ’14, pages 601–610, New York, NY, USA, 2014. [28] J. Hoffart, F. M. Suchanek, K. Berberich, and G. Weikum. ACM. YAGO2: A Spatially and Temporally Enhanced Knowledge [14] S. Duan, A. Kementsietsidis, K. Srinivas, and O. Udrea. Base from Wikipedia. Artificial Intelligence, 194:28–61, 2013. Apples and Oranges: A Comparison of RDF Benchmarks and [29] A. Hogan, A. Harth, A. Passant, S. Decker, and A. Polleres. Real RDF Datasets. In Proceedings of the ACM SIGMOD Weaving the Pedantic Web. Proceedings of the WWW2010 International Conference on Management of Data, SIGMOD Workshop on Linked Data on the Web, 628, 2010. 2011, pages 145–156, 2011. [30] A. Hogan, J. Umbrich, A. Harth, R. Cyganiak, A. Polleres, and [15] B. Ell, D. Vrandeciˇ c,´ and E. Simperl. Proceedings of the 10th S. Decker. An empirical survey of linked data conformance. International Semantic Web Conference (ISWC 2011), chapter Web Semantics: Science, Services and Agents on the World Labels in the Web of Data, pages 162–176. Springer Berlin Wide Web, 14:14–44, 2012. Heidelberg, Berlin, Heidelberg, 2011. [31] P. Jain, P. Hitzler, K. Janowicz, and C. Venkatramani. There’s [16] F. Erxleben, M. Günther, M. Krötzsch, J. Mendez, and No Money in Linked Data. http://corescholar. D. Vrandecic. Introducing Wikidata to the Linked Data Web. libraries.wright.edu/cse/240, 2013. accessed In Proceedings of the 13th International Semantic Web July 20, 2015. Conference, ISWC 2014, pages 50–65. Springer, 2014. [32] J. M. Juran, F. M. Gryna, and R. S. Bingham, editors. Quality [17] M. Färber, F. Bartscherer, C. Menne, and A. Rettinger. Linked Control Handbook. McGraw-Hill, 1974. Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and [33] G. Kobilarov, T. Scott, Y. Raimond, S. Oliver, C. Sizemore, YAGO. Semantic Web Journal, 2017, to be published. M. Smethurst, C. Bizer, and R. Lee. Media Meets Semantic [18] M. Färber, C. Menne, and A. Rettinger. A Linked Data Web – How the BBC Uses DBpedia and Linked Data to Make Wrapper for CrunchBase. Semantic Web Journal, 2017, to be Connections. In Proceedings of the 6th European Semantic published. Web Conference on The Semantic Web: Research and [19] C. Fellbaum. WordNet – An Electronic Lexical Database. MIT Applications, ESWC 2009 Heraklion, pages 723–737, Berlin, Press, 1998. Heidelberg, 2009. Springer. [20] A. Flemming. Qualitätsmerkmale von Linked [34] D. Kontokostas, P. Westphal, S. Auer, S. Hellmann, Data-veröffentlichenden Datenquellen (Quality characteristics J. Lehmann, R. Cornelissen, and A. Zaveri. Test-driven of linked data publishing datasources). Diploma Thesis, evaluation of linked data quality. In Proceedings of the 23rd Humboldt University of Berlin, international conference on World Wide Web, pages 747–758. http://www.dbis.informatik.hu- ACM, 2014. berlin.de/fileadmin/research/papers/ [35] D. Kontokostas, A. Zaveri, S. Auer, and J. Lehmann. diploma_seminar_thesis/Diplomarbeit_ TripleCheckMate: A Tool for Crowdsourcing the Quality Annika_Flemming.pdf, 2011. Assessment of Linked Data. In Knowledge Engineering and [21] G. Freedman and E. G. Reynolds. Enriching Basal Reader the Semantic Web – 4th International Conference, KESW 2013, Lessons with Semantic Webbing. Reading Teacher, St. Petersburg, Russia, October 7-9, 2013. Proceedings, pages 33(6):677–684, 1980. 265–272. Springer, 2013. [22] C. Fürber and M. Hepp. SWIQA – A Semantic Web [36] C. Matuszek, J. Cabral, M. J. Witbrock, and J. DeOliveira. An Information Quality Assessment Framework. In Proceedings of Introduction to the Syntax and Content of Cyc. In AAAI Spring the 19th European Conference on Information Systems Symposium: Formalizing and Compiling Background 52 M. Färber et al. / Linked Data Quality of DBpedia, Freebase, OpenCyc, Wikidata, and YAGO
Knowledge and Its Applications to Knowledge Representation [43] F. M. Suchanek, G. Kasneci, and G. Weikum. YAGO: A Large and Question Answering, pages 44–49. AAAI - Association for Ontology from Wikipedia and WordNet. Web Semantics: the Advancement of Artificial Intelligence, 2006. Science, Services and Agents on the World Wide Web, [37] M. Mecella, M. Scannapieco, A. Virgillito, R. Baldoni, 6(3):203–217, 2008. T. Catarci, and C. Batini. Managing data quality in cooperative [44] T. P. Tanon, D. Vrandecic, S. Schaffert, T. Steiner, and information systems. In On the Move to Meaningful Internet L. Pintscher. From Freebase to Wikidata: The Great Migration. Systems 2002: CoopIS, DOA, and ODBASE, pages 486–502. In Proceedings of the 25th International Conference on World Springer, 2002. Wide Web, WWW 2016, pages 1419–1428, 2016. [38] O. Medelyan and C. Legg. Integrating Cyc and Wikipedia: [45] S. Tartir, I. B. Arpinar, M. Moore, A. P. Sheth, and Folksonomy meets rigorously defined common-sense. In B. Aleman-meza. OntoQA: Metric-Based Ontology Quality Wikipedia and Artificial Intelligence: An Evolving Synergy, Analysis. In IEEE Workshop on Knowledge Acquisition from Papers from the 2008 AAAI Workshop, page 65, 2008. Distributed, Autonomous, Semantically Heterogeneous Data [39] F. Naumann. Quality-Driven Query Answering for Integrated and Knowledge Sources, 2005. Information Systems, volume 2261. Springer Science & [46] R. Y. Wang, M. P. Reddy, and H. B. Kon. Toward quality data: Business Media, 2002. An attribute-based approach. Decision Support Systems, [40] L. L. Pipino, Y. W. Lee, and R. Y. Wang. Data Quality 13(3):349–372, 1995. Assessment. Communications of the ACM, 45(4):211–218, [47] R. Y. Wang and D. M. Strong. Beyond Accuracy: What Data 2002. Quality Means to Data Consumers. Journal of management [41] E. Sandhaus. Semantic Technology at the New York Times: information systems, 12(4):5–33, 1996. Lessons Learned and Future Directions. In Proceedings of the [48] A. Zaveri, D. Kontokostas, M. A. Sherif, L. Bühmann, 9th International Semantic Web Conference on The Semantic M. Morsey, S. Auer, and J. Lehmann. User-driven quality Web - Volume Part II, ISWC’10, pages 355–355, Berlin, evaluation of dbpedia. In Proceedings of the 9th International Heidelberg, 2010. Springer. Conference on Semantic Systems, pages 97–104. ACM, 2013. [42] A. Singhal. Introducing the Knowledge Graph: things, not [49] A. Zaveri, A. Rula, A. Maurino, R. Pietrobon, J. Lehmann, and strings. https://googleblog.blogspot.de/2012/ S. Auer. Quality Assessment for Linked Data: A Survey. 05/introducing-knowledge-graph-things- Semantic Web, 7(1):63–93, 2015. not.html, retrieved on Aug 29, 2016, 2012.