OntoConf Ontology visualization and instance matches confirmation
Marian Szabo OntoConf Ontology visualization and instance matches confirmation
by
Karol Marian Szabo
in partial fulfillment of the requirements for the degree of
Master of Science in Computer Science
at Delft University of Technology, to be defended publicly on April 20, 2017
Student number: 4255771 Thesis committee: Chair: Prof. dr. ir. G.J.P.M. Houben, TU Delft University supervisor: Dr. ir. A.J.H. Hidders, TU Delft Company supervisor: Drs. Gert-Jan van Lochem, HintTech B.V., Delft Committee Member: Dr. A.E. Zaidman, TU Delft
Web Information Systems(WIS) HintTech B. V. Faculty EEMCS, Delft University of Technology Delft, the Delftechpark 37i Netherlands Delft, the Netherlands wis.ewi.tudelft.nl www.hinttech.com
GertJan Jan van Lochem Hidders
Family Acknowledgements Dana Friends Stuparu
Ties Rijcken
academic guidance, encouragements encouragements, coffee, drinks, $$$
Abstract
Joining multiple ontologies with the purpose of connecting existing knowledge can be done with the help of ontology matching tools. Part of this process involves establishing links between instances of classes from the paired ontologies. This process, referred to as instance matching, is done by algorithms that automatically generate the links between them. Since the suggested links might not be correct, people who are familiar with the involved ontologies can improve the outcome of this process by assessing the generated matches.
The aim of this thesis is to investigate how to design a system which facilitates ontology understanding and involves users in the instance matching evaluation process. The tool is meant to be used by domain experts, with limited knowledge in Semantic Web technologies. To achieve this, a graph visualization of the ontology was developed and then evaluated during a two-phase iterative process. Later on, an instances-matched confirmation module was created and coupled with the visualization system. The user evaluation showed that domain experts are able to perform ontology exploration tasks similar to existing ontology visualization platforms and underlines people’s behaviour when they are exposed to large amount of data. The evaluation results also support the idea that users manage to improve the quality of the instance-matching by successfully using the match confirmation module.
Contents
1. Introduction …………………………………………………………………………..... 1 1.1. Research questions ……………….…………………………………………... 2 1.2. Scope ……………….……………………………………………………….... 4 1.3. Outline ……………….……………………………………………………….. 4 2. Basics of on ontologies and Semantic Web …………………………………………... 5 3. Related work ……………………………………………………………………...…… 8 4. System design and implementation ……………………………………………...…… 13 4.1. Elements which form the ontology visualization …………………………….. 13 4.1.1. OWL classes or their instances ……………………………………... 13 4.1.2. Connecting concepts using SubclassOf relation ……………………. 14 4.2. Visualization layout ………………….……………………………………….. 15 4.3. Technical choices …………………….……………………………………….. 16 4.4. Implementation ……………………….………………………………………. 21 4.4.1. First phase ……………………….………………………………….. 22 4.4.1.1. Overview ……………………….……………………….... 22 4.4.1.2. Zoom and pan ……………….……………………………. 23 4.4.1.3. Search function ……………….…………………………... 26 4.4.1.4. Filtering ………………………….……………………….. 26 4.4.1.5. Details on demand ………………………….…………….. 26 4.4.1.6. Relate ………………………….………………………….. 27 4.4.2. Second phase ………………………….……………………………. 28 4.4.2.1. List of individuals module ………………………………... 29 4.4.2.2. Detailed information module ……………………………... 30 4.4.2.3. Vertical navigation toolbar ……………………………….. 31 4.4.2.4. Search module ……………………………………………. 31 4.4.2.5. Ontology smells detection module ……………………….. 32 4.4.2.5.1. Misclassified instances …………………………. 32 4.4.2.5.2. Incorrect similarity links between individuals ….. 33 5. Evaluation ……………………………………………...………………………………. 34 5.1. Targeted users ………………………..……………………………………….. 34 5.2. Data …………………………………..……………………………………….. 34 5.3. Methodology …………………………………..……………………………… 36 5.4. Evaluation of the ontology visualization tool ……………………………….... 37 5.5. Evaluation of the ontology smells module ………………………………….... 39 6. Results ……………………...…………………………………………………...……… 41 6.1. First phase …………………………………………………………………….. 41 6.2. Second phase …………………………………………………………………. 42 6.2.1.Introductory section …………………………………………………. 42 6.2.2. Ontology visualization …………………………………………….... 45 6.2.3. Ontology smells module …………………………………………..... 49 7. Conclusions and future work ……………………...………………………………….. 62 8. References …………………………………………...…………………………………. 65 9. Appendix ……………………………………………...………………………………... 68
1
1. Introduction
Semantic Web technologies have started playing an important role in the technologies stack which form the information systems of healthcare, IT, legal and publishing companies. Among them is Newz, which was established in 2012 by twelve leading news publishers from The Netherlands. They aim to create a common shared repository for storing the high quality content produced by their professionals while writing newspapers, magazines, books, websites, blogs. Using this platform, the publishers are able to gather the content and sell it to other parties, generally companies. Newz does more than simply store content in a database, it enriches it semantically through an automated process which, based on existing ontologies (DBpedia, GeoNames, Freebase and the Newz ontology itself), creates RDF triples that are assigned to each piece of content. The semantic enrichment is an ongoing process which makes the ontology grow bigger and bigger. Currently, there are more than 150 millions triples in the knowledge base, so having an overview of the current state of the ontology is challenging, especially for non-specialists. An important thing that has to be mentioned is that using multiple ontologies together is not specific only to Newz, but rather a desired behaviour in the direction of linked data. To support this, multiple researchers investigated options for matching ontologies and came up with tools for performing this task. In general, the ontology matching process takes place in two stages: schema mapping and individual matching. The first phase happens at schema level when ontology experts together with domain experts define the correspondences between classes from each ontology. After this step is performed, it is desirable to run entity matching algorithms to establish correspondences between the individuals (instances of classes) which refer to the same real-life concepts. Fortunately, these procedures are supported by automatic or semi-automatic tools which ease the work. However, these tools are not completely accurate, so correcting the errors manually is required. A common thing of the individuals matching algorithms is that they provide for each identified match a confidence coefficient (usually between 0 and 1 or 1-100%) which can be a good indicator that human intervention is needed. Because these kinds of errors are related to the knowledge base, they must be evaluated and corrected by domain experts and not by ontology engineers who design and perform the matching process. Domain experts can play an important role in improving the quality of ontologies by correcting other types of errors such as duplicates, missing entities and wrong entity types. These kinds of errors show up when semantic enrichment is done automatically, and this is not something strictly related to news processing.
2
Since these issues often occur in systems which use ontologies, many graphic tools have been developed to facilitate ontology management. Some of them are only focused on ontology experts but there are also tools designed for domain experts with basic or no knowledge in Semantic Web technologies and which will be referred as “non-expert” users, from now on. Another way of browsing ontologies is by using the SPARQL query language to extract the desired data. This approach is more suitable for ontology experts, while the graphical interfaces can be easier to use by non-experts. A common thing of most ontology visualization tools for non-experts is that they try to be exhaustive in displaying the information related to the structure of the ontology even if the focus of their target group of users is rather the knowledge base itself, than the structure-related information. Considering this, we believe that the users’ needs could be better fulfilled by building up a visualization which has the knowledge base as starting point, with less emphasis on the way it is structured.
1.1. Research questions
Research question 1: How to design a visualization tool for ontologies that provides an effective support for the non-expert users in quickly understanding what are the main areas covered by the ontology and which offers knowledge base exploration support?
The ontology visualisation tool should show a graphical representation of the ontology on different levels of detail that are characterized by the amount of information presented to the users. At a higher level, the tool should display the most representative entities for the ontology and the links between them whereas, at a more detailed level, granular information should be shown. An interesting challenge in this context is to determine what is representative for the ontology and to obtain this data in a short amount of time. Another thing which is challenging to design is the functionality for displaying the detailed information about the concepts in the knowledge base because it should be generic enough to be able to show any kind of data, while making the information easy to read and understand by non-experts. This is more than only a requirement for the ontology understanding goal, because the quality of the knowledge base can improve if non-expert users report the errors they discover while browsing it. Errors reporting should be seen as a complementary way to improve the quality of the knowledge base to the automatic algorithms which are not fully accurate.
Research question 2: How to design a visualization tool for ontologies which can help non-expert users detect errors in the knowledge base?
The investigation is focused on errors caused by matching multiple ontologies together. Even if the automatic matching processes achieve high accuracy, incorrect matches or incorrectly
3
classified entities errors can still occur. An important number of these errors can be detected using automatic tools, which often require human intervention to decide which of them are false positives/negatives.
Initially, the scope of this research question was extended to errors related to the structure of the ontology, but after assessing the Newz project ontology and realizing that the main issues were related to the knowledge base, it was narrowed down to the current scope, which is described in the following section.
4
1.2. Scope
Identifying the scope of this research means clarifying two elements: who are the targeted users and under which circumstances the non-expert users can improve the quality of the knowledge base by reporting the errors they discover. There are two types of users on the focus of this research: ontology experts and domain experts. The difference between these two types of users is that ontology experts and domain experts have different levels of understanding the techniques used to describe ontologies. The first type of users can always understand the structure of the ontology (since they are in charge of designing it) and they might have a limited understanding of the data itself because it might require in-depth knowledge in the field described by the ontology. On the other hand, domain experts can achieve a basic understanding of the structure of the ontology (which is good enough for the kind of interaction they need to have with it) and they know in detail the information from the knowledge base. As it has been defined in the second research question, the goal of the visualization tool is to improve the quality of the knowledge base, therefore the group of users on which this research is focused is restricted to domain experts.
The goal of this research is to develop tools for improving the knowledge base quality in the context of ontology matching. This means that domain specialists use their expertise to validate matches between instances of classes coming from multiple ontologies. This step is required when two ontologies are matched together and also during the ontology enrichment process when the newly created instances have to be connected with existing ones in the knowledge base.
1.3. Outline
Chapter 2 presents the basics of ontologies and Semantic web which are useful for a proper understanding of this report. Next, Chapter 3 surveys the main ontology visualization techniques and explains the connection between ontology visualization and errors detection. Chapter 4 presents, step by step, the design choices, gives details about the system implementation and about the technologies used to create the system. Chapter 5 is focused on the users evaluation and the results are discussed in Chapter 6. The last chapter draws the conclusions of this research and provides some ideas for further developing the system.
5
2. Basics of ontologies and Semantic Web
This chapter presents basic concepts about ontologies and Semantic Web: what ontologies and knowledge-bases are and how are they related to each other. The readers of this research have to be familiar with these notions to properly understand the report. The theoretical information is combined with practical examples to show how real life concepts are modeled using an ontology. Some statistical information about the similar concepts from distinct ontologies are given to show that even if two ontologies describe the same domain, they can model the data in different ways, therefore the ontology schema and the knowledge base are different. According to Gruber [26], in the computer science field, “an ontology defines a set of representational primitives with which to model a domain of knowledge or discourse”. This means that an ontology provides a collection of “ingredients” which can be used to represent the data from a specific domain of knowledge. Any ontology is defined by a schema which contains the class definitions, data type definitions and the relations between classes. A class is an abstract notion which corresponds to a type of object from real life, like animal, bird, building, museum which is characterized by attributes (name, weight, height). It can be observed that the attributes have different meanings and therefore specific pieces of information are expected for each of them. For example, the name attribute is a string of characters while the weight or height are numbers. These kind of classifications are called data types. An attribute can also hold a reference to another class in which case it expresses a relationship between two classes. In the DBPedia ontology the museum class is a subclass of building, which means that all museums are also buildings. Now that the basic notions of an ontology schema were mentioned, it is time to introduce the notion of knowledge base which represents the actual data that comply to the restrictions defined by the ontology schema. “Rijksmuseum”, for example, is an instance of type museum, the museum class defined in the ontology schema, located in Amsterdam, The Netherlands. “Amsterdam” is of type city, which is, again, a class defined by the ontology schema. It is easy to observe that any class from the ontology schema can have multiple instances in the knowledge base. In DBPedia, the class museum has 4195 instances whereas LinkedGeoData contains 46320 museums in the knowledge base. The same cardinality relation between class definitions and their instances holds for every entity from the schema definition. The number of classes is limited and it usually grows when new types of entities are required or when a refinement of the current classes is needed, for a better precision. DBpedia ontology contains 735 class definitions, LinkedGeoData 1200 but SUMO ontology contains 4525 classes. Overall, the number of instances is significantly higher than the number of classes: in DBPedia there are 735 classes and 4324525 instances and in LinkedGeoData there are 1200 classes and 34.501.590 instances, which means that the knowledge base is much larger than the
6
ontology schema. An important thing that should be mentioned is that the classes are linked to each other through axioms and expressions. OWL 2, which stands for Web Ontology Language, is used to define multiple types of axioms and expressions (about which you can read in the remaining of this chapter) that can bring implicit reasoning capabilities and can guarantee the data consistency. There are three species of OWL: OWL Lite, OWL DL and OWL Full which offers an increasing expressive power. For the purpose of this thesis, OWL DL will be considered. In OWL, expressions are complex structures formed by restrictions related to the attributes which describe the individuals (instances of classes). All the individuals which match the restrictions form a group of instances of the corresponding class expression. An example of such an expression is given by Horridge et al. [1]: Person AND hasChild SOME (Person AND (hasChild ONLY Man) AND (hasChild SOME Person)) which defines a group grandparents who have only grandsons. It can be said that expressions are used to extract a group of individuals which match a set of criteria.
The OWL specification also defines axioms as relations which are known to be true in the domain described by ontologies. Depending on the types of structures they refer to, there are three types of axioms: class axioms, data and object property axioms. The first type is designed to define relations on class level: for example, student is a subclass of person. In this example, student and person are two distinct classes and the class axiom which refers to them is the “subclass” relation. OWL specification uses rdfs:subClassOf relation to express this type of connection between classes. An important thing which can be mentioned is that in OWL specification rdfs:subClassOf relation is transitive, which means that if class A is a subclass of class B and class B is a subclass of C, then class A is a subclass of class C. In DBPedia, for example, University is a subclass of Educational institution, which is a subclass of Organization, therefore a University is a subclass of Organization. Although this kind of relations are defined at schema level, to put order in the structure of the ontology, their effects can be seen automatically in the knowledge base. Taking into consideration the University - Educational Institution - Organization example, an instance of University is automatically both an instance of Educational Institution and Organization, due to the transitivity of the subclass of relation which connects those three types of classes. Therefore, since TU Delft is an instance of University, then TU Delft is also an instance of Educational Institution and an instance of Organization (the other way around is not true because rdfs:subClassOf is not defined as a symmetric property).
7
In total, the OWL specification defines 17 classes and 23 types of properties which can be used to build a visual representation of any OWL based ontology. Out of the 17 classes, only owl:Thing and owl:Nothing predefined as class identifiers which means that every OWL class is a subclass of owl:Thing and the other one is a subclass of any OWL class. Defining a new class can be done by creating an instance of type owl:Class. Therefore, retrieving the collection of classes in an ontology is done by asking for all individuals with the type owl:Class.
Although it might look complex at a first glance, researchers found ways to build visualizations for ontologies which make the life of ontology engineers and domain experts easier. An overview of these visualizations tools is given in the next chapter.
8
3. Related work
Ontology visualization is a domain which has been investigated in the past couple of years by multiple researchers who tried to design easy to understand tools for creating, managing and browsing ontologies. In general, they have as starting point the structure of the ontology which involves showing the relationships between classes, classes inheritance and the attributes which define classes. Different scientists used different means to represent these concepts and the relations between them intuitively for the users. In general, the investigations were initiated to find out if a specific type of visualization (e.g. graph, indented tree) can perform well on specific tasks which were mostly about understanding the structure of the ontology.
The usability and effectiveness of most tools were proven by user testing, and valuable insights about the users’ abilities to cope with the complexity of ontologies were obtained. The purpose of this section is to give an overview of the visualization techniques employed in designing tools that are addressed to non-expert users. Another goal is to present the evaluation methods of the developed tools. A selection of papers accepted at the International Semantic Web Conference starting in 2005 and from Lecture Notes in Computer Science was performed based on the following criteria: - the main topic of the paper is ontology visualization; - non-experts are among the target groups of users.
FlexViz [2] is a web-based ontology-visualization tool based on Adobe Flex technology which is able to display the information using multiple layout types (tree, spring and grid-based layouts), search entities by label, filter by nodes on relationships between nodes types. Each concept from the ontology is a node in the visualization and the links between concepts are represented as arcs. The user experience in the application is enriched by zoom&pan and drag&drop functionalities which allow users to organize the displayed information according to their needs. By implementing this features, the researchers try to solve the visualization support Semantic Web-related challenge, which is among other five, enumerated by Benjamins et al. in [3]. To prove the flexibility of this tool, they built three extensions on top of FlexViz: BioPortal integration (for storing biomedical related ontologies), FOAF support (for representing Friend Of A Friend relations), and Bio-Mixer (for exploration of biomedical ontologies). The user testing evaluation is missing, but the researchers state that FlexViz is able to reach a large audience, reduce task complexity, track users behavior and enhance the social data analysis.
9
Sung-min et al. [4] investigate the possibility of visualizing a summary DBPedia knowledge base using Hadoop for data processing and HTML 5 for creating the user interface. Having as starting point a single .nt file [5], the system is able to compute the number of entities of each DBPedia class type and the most used properties of each class in a timely manner (42 and 52 seconds for datasets of 1.9GB and 8.8GB). Although the data processing part can can be obtained faster by querying the DBPedia SPARQL endpoint, this approach does not require storing the dataset into a triple store which saves a considerable amount of time and resources, which is also the main strength of the research. The data visualization consists of tables used for listing the results of the computation which makes it generic enough to represent any type of data. The user experience is limited, but it is important to mention that this system can be use to create a visualization (or at least extract the required data) of any ontology only at the cost of a Hadoop deployment, which is significantly lower than deploying large amount of data into a triple store.
Knoocks [6] is an ontology visualization tool designed for ontology engineers and also for domain experts (users who are specialized in the domain described by the ontology). It offers functionalities for visualizing the ontology schema and also for digging into the knowledge base. To accomplish these two features, the user interface is divided into three parts: one used for displaying the classes and subclasses defined in the ontology as a list of names, another one which shows more details (including the relationships between that class and other classes) about the class on focus in the first section and a main area used for zooming in different parts of the ontology. The visualization tool has easy to understand features like bookmarks or coloring the classes based on the number of instances. Knoocks was developed in 4 iterations driven by the human centered design principle: each development phase is followed by users testing. The output of the users testing phase represent the input of a new iteration. The user evaluation had 2 goals: improving the tool after each iteration and comparing the tool with other existing ones. To accomplish these goals, users were asked to perform a some tasks: identify specific instances and their data types, one set focused on dependencies between classes and instances. Users were observed while performing the tasks and they were also encouraged to think aloud. A usability questionnaire, as defined by Prümper [7], was applied to assess the usability of the application. Knoocks was compared with other tools: CropCircles and Jambalaya in the first iteration, TGVizTab and TGVizTab in the third iteration, Jambalaya and OntoViz in the fourth one. The researchers used an ontology that defines the structure of the bachelor in computer science curriculum which contains 86 classes, 122 instances, 2 object properties and 8 datatype properties. Overall, the users feedback was positive and showed that Knoocks brings improvements to the already existing tools. Although the users were able to understand the ontology with which they interacted with, it would be interesting to see if the global overview on the ontology was indeed easily comprehended by people. One reason for which this is a pertinent question is that displaying all the classes as a list and only show the relationships between a specific class and the others might
10
affect users perceptions of the ontology as a whole. Also it would be interesting to see how the tool is perceived by users when they have to deal with larger ontologies (like DBPedia, Sumo, LinkedGeoData, etc).
KC-Viz [8] tests the hypothesis that an effective ontology visualization tool can be built by implementing a key concept extraction algorithm (KCE). KCE is used to extract the most important entities from the cognitive perspective by computing metrics related to the “making sense” dimension of the information existing in the ontology: density (information-rich in a formal knowledge representation sense), coverage (coverage of the ontology with respect to its is-a hierarchy), popularity (concepts that are likely to be most familiar to users). KC-Viz was developed as NeOn Toolkit plugin and it is based on node-link diagram paradigm. This tool was designed to scale up to for large ontologies which require displaying a high number of entities on the screen and also to scale down so it allows detailed visualization on specific concepts. The initial view is built by using the concept extraction algorithm which gets as input an ontology and returns a list of nodes (the number of elements can be set by the end users) and the relationships between them. They are displayed as a connected graph in which each node represents an OWL class and the edges between them are rdfs:subClassOf (direct/indirect) relations (illustrated by direct or dotted lines). The visualization also contains the number of direct/indirect subclasses, class type, URI, direct superclasses (actual/shown), direct subclasses (actual/shown), individuals (direct/all). Diving into more details can be achieved by clicking on the nodes which are rendered on the screen. An important feature of this visualization tool is that the graph visualization which represents initial view of the ontology allows users to set the maximum number of concepts (nodes in the graph) which have to be displayed on the screen. This approach makes the tool usable on very large ontologies due to the techniques used for zooming into and hiding of specific parts of an ontology. KC-Viz [6] was empirically evaluated by 21 users who were initially trained through a tutorial about the tools involved in the evaluation. After that, users received 4 ontology exploration tasks that had to be performed on 3 distinct tools (one of them was KC-Viz) for visualizing ontologies. The results showed that KC-Viz outperforms the other tools.
Nadia Catenazzi and Lorenzo Sommaruga present EasyOnto in [9], an ontology visualization tool which is designed for domain experts to explore ontologies. There are two ways of visualizing the structure of the ontology. The first one combines the hierarchical and role relations between the OWL classes into a single graph visualization in which each node represents an owl:Class and each edge is a relation between classes.The second option is also based on the graph visualization but the hierarchical relations between classes are represented as nested sets (classes are encapsulated in rectangles which represent their parent class), while the role relations are represented using arcs. There is no detailed user evaluation presented but the
11
researchers mentioned that the system was successfully used by domain experts from the water management field (CTI project nr. 9402.1 PFES-ES). In the conclusion of the paper it is stated that graph based visualization can be successfully used for ontologies with a limited number of classes, but it becomes unreadable on large sets.
Pavel Lomov and Maxim Shishaev [10] approach the ontology visualization topic from a different angle: the users’ ability to understand OWL ontologies by using cognitive frames. A cognitive frame is defined as a visualization structure which is able to create a clear image of a concept in the users mind. They identify two mandatory attributes that a cognitive frame should have: completeness and compactness which are in contradiction because a complete visualization of a concept (which can be an OWL class or an instance of a class) might include many connections to other concepts which also need to be represented in the visualization. When a cognitive frame contains too many objects, the compactness attribute has to suffer because the human mind is able to process 7-9 objects by using the short-term memory, according to Miller in [11]. To cope with this situation the researchers choose to reduce the complexity of the ontology which has to be presented by constructing an “upper level ontology” which is a subset of the original one. The output of this research is a generic algorithm (which can be applied to any type of OWL ontology) for constructing the upper level ontology which contains the right amount of information that a normal user can grasp. Even if there is no tool built for validating the algorithm proposed by the researchers, the results look promising because the research is strongly based on the users abilities to understand representations of abstract concepts and it proposes a model of reducing the complexity of large ontology to simplified models which easier to understand and manipulate.
Discussion
Creating user interfaces aimed to help domain experts browsing and understanding ontologies is challenging because ontologies are complex and the knowledge bases are large. To deal with these situations, scientists created interfaces which present ontologies on multiple levels correlated with the amount of presented details. The first level of detail usually presents an overview of the ontology and it is the starting point for the entire visualization. Scientists have different approaches for deciding which pieces of information should be presented in each layer. Kriglstein et al. in [6] and Catenazzi et al. in [9] are taking this decision based on the structure of the ontology while others Lomov et al. in [10], Motta et al. in [8] try to determine the most important concepts of the ontology. Kriglstein et al. and Catenazzi et al. use the hierarchical relations (i.e. rdfs:subClassOf) to determine the root classes of the ontologies which are the visualization starting point. Lomov et al. in [10] and Motta et al. in [8] try to use the structure of the ontology (the relations between classes) to find out which are the core concepts of the ontology. Therefore, it can be said that [10] and [8] do not use the raw data for creating the
12
visualization, but they build another logic layer on top of the ontology which is a step further compared to the other solutions.
The next topic, which follows the discussion about the techniques used to choose the right amount of data which have to be put on display to the users, analyses the ontology visualization techniques. The most simplistic one is the indented list view, implemented by Bo Fu et al. in [27], offering the advantage of simplicity and familiarity while lacking efficiency for large ontologies and it can be used to represent the entities only based on a single relation between them. Falconer et al. in [2] and Kriglstein et al. in [6] propose a graph-based visualization of ontologies in which entities forming ontologies are nodes (vertices) and the relations between them are represented as links. Compared to indented lists, this visualization type can represent multiple types of relations by using distinct colors for the links between entities, Catenazzi et al. in [9], Kriglstein et al. in [6] or it can visualize certain properties of the relations (the number of relations between classes Motta et al. in [9]) by varying the thickness of the links. On the other hand, the graph visualization does not use the screen space in an efficient way because it is difficult to fit large ontologies in a single screen while making the drawing easy to read. This problem is better handled by other visualization types like Euler diagrams. They were implemented by Parsia et al. [12] in CropCircles, an ontology visualization system that displays the hierarchical structure of ontologies by using nested circles, in which each class is represented by a circle and the hierarchical relations between classes are represented by encapsulating one circle into another. Another example of using the screen space efficiently comes from Storey et al. in [13] who designed a system which uses Treemaps to visualize ontologies.
Furthermore, to assess to which extent the proposed solutions fit the users’ needs, researchers performed user evaluations in which they tested the ability of understanding ontologies at high and deep levels. For this, Kriglstein et al. in [6] and Motta et al. in [8] organized interviews with non-experts in which they were asked to perform tasks related to ontology exploration and understanding, while using their tools. The results of the empirical evaluations are both qualitative and quantitative and the newly designed tools are compared with existing alternatives to check if any improvements are obtained.
The previously mentioned tools were adhering to usability principles and they are aimed at both ontology and domain experts (because part of their functionality is to allow users to create and manage ontologies). The idea of supporting domain experts with limited knowledge of Semantic Web technology is addressed by creating user friendly graphical interfaces which reveals the right amount of information about the ontology that is understandable by people. This approach proved to be successful when tested on small ontologies but it was insufficiently tested on ontologies with complex structure or with large knowledge bases, therefore we consider this an open problem.
13
4. System design and implementation
This chapter will focus on presenting the decisions which were taken in the process of creating the visualization tool. The process starts with deciding what pieces of information from the ontology should be used in the visualization system (and its layout) and, following that, performing an evaluation of the technologies used to implement the tool. It continues with an enumeration of the principles on which the visualization tool is based. The following section explains how the theoretical principles are implemented in this practical case and also the two phases approach for developing the user interface.
4.1. Elements which form the ontology visualization
The “Related work” chapter presented some of the approaches which dealt with displaying ontologies in understandable ways by domain and ontology experts. Some of them are oriented towards displaying the structure of the ontology while others are more focused on the knowledge base. Moreover, the scientists used various visualization types in their work. The next part of this paper presents an analysis of the alternatives for each of these points and explains the decisions that were taken.
4.1.1. OWL classes or their instances As it has already been mentioned an ontology contains classes and instances of classes, therefore these are the two ingredients which can be used to build the overview. A common characteristic noticed in many ontologies is the high amount of classes and instances of classes they contain, which is a real challenge in building the user interface for the ontology visualization. On the one hand the difficulty resides in the human inability to grasp high amounts of information (e.g. there are ~12.5 millions instances in DBPedia, and a lot more in LinkedGeoData), but on the other hand, there are also technological difficulties in rendering all this data (e.g. related to layout, memory consumption). One way of dealing with such a high amount of instances would be to only show the most representative ones from the ontology. In this scenario, the users would recognize the type of data they are looking at and they could go deeper in the ontology. Even if this scenario seems to solve the big data problems, another issue occurs: how do we know what is representative for an ontology with respect to each person’s interests? Considering that finding a solution for such a problem deviates from the scope of this thesis, it has been decided to keep this scenario in mind, but to investigate other possibilities.
14
An alternative to this approach is using the defined classes in the ontology as starting points for browsing the knowledge base. Since the amount of classes is considerably smaller than the number of instances, the problem of displaying millions of classes is attenuated. One particularity worth mentioning is that the only classes which are taken into consideration are the ones which have instances or which have children, with respect to the rdfs:subClassOf relation. Although this is a minor detail, on some ontologies like DBPedia, it reduces the amount of classes which have to be displayed by 35% (there are 791 defined classes and 279 do not have any instances). On LinkedGeoData ontology, this choice does not significantly improve the situation because there are 1200 defined classes and only 67 have no instances associated. Considering these challenges, the following decision was taken: use OWL classes which have instances (directly or indirectly) to represent ontologies.
4.1.2. Connecting concepts using SubclassOf relation A clear overview of the ontology would not be complete without displaying the relations between classes. One question that emerges is about which type of relation defined by OWL is possible to be displayed and be also accessible to people. The first part of the problem has a rather simple solution because, since it was decided to use OWL classes for creating the visualization for overview, only classes axioms can be used. OWL specification defines three axioms: rdfs:subClassOf, owl:equivalentClass and owl:disjointWith with the following meaning: - rdfs:subClassOf is meant to establish a hierarchical classification between classes, for example an automobile is a subclass of vehicle; - owl:equivalentClass is meant to mark classes which are equivalent to each other, which means that they refer to the same concepts from real life; - owl:disjointWith is used to mark classes which cannot have any instances in common. It can be observed that owl:equivalentClass and owl:disjointWith express notions which are implicit to the human mind. For example, it is trivial to say that all Men are different from Women (an analogy for owl:disjointWith), so representing such statement might not improve the users understanding of ontologies. The first relation, rdfs:subClassOf, on the other hand, is meant to establish a hierarchy between types of entities (classes in OWL) and helps people to have a more structured view of the data which is presented. Previous studies on controlled natural languages for OWL 1.1 showed that non-expert users are able to understand the sub-class relation. All these arguments recommends the rdfs:subClassOf relation as the right choice for showing the relations between classes in the context of data visualization. To sum up, the design choices taken so far are to use the owl:Class and rdfs:subClassOf structures to create the user interface for visualizing an ontology.
15
4.2. Visualization layout
The “Related work” chapter makes a summary of the existing techniques applied for visualizing ontologies: node-link diagrams (Knoocks and KC-Viz), graph visualization (EasyOnto, FlexViz), cognitive frames, UML (in plugins for Protégé) and treemaps. Keeping in mind the number of steps which the users have to perform for viewing the entire set of nodes and the number of classes which have to be displayed in the user interface, we chose to use an oriented graph visualization in which each node is an owl:Class and each edge is a rdfs:subClassOf relation. Assuming that the goal of the visualization is to display a large amount of nodes (more than 500) connected by edges, choosing the right layout is of high importance. As Yifan Hu [14] states, a good graph layout is characterized by a small amount of crossing edges and no overlapping between nodes. To achieve this goal we took into consideration five types of layout algorithms: - force-based layouts; - orthogonal layout; - layered graph drawing; - arc diagrams; - circular layout. Circular and arc diagrams layouts were excluded right from the beginning because they imply positioning the vertices around a circle or in a straight horizontal line, which would be impractical for more than 500 nodes (the visualization would become very large and unreadable). The orthogonal layout was removed from the list of options for the same reason (the visualization would become too large). Layered graph drawing is not suitable for building the ontology visualization because the second step in this algorithm implies removing the cycles in the graph, which would translate in not showing all the rdfs:subClassOf relations (if any cycle exists). Compared to the already discussed layout algorithms, the force-based layouts do not have constraints related to the graph topology and the visualization size is as compact as possible (in accordance to the configuration parameters). This type of algorithm is based on a physical model in which each node is represented by a magnetic ring and each edge is represented by a string which connects two nodes. All the rings have the same magnetic charge, so they repel each other but they are kept in proximity to each other by the strings which connect them. The length of the strings modifies according to the forces which act on both sides of them. Force-based algorithms run in rounds. In each round the coordinates of all nodes are settled according to the forces that are acting upon them. As soon as the coordinates are stable, the algorithm ends. A common element which draws the attention of many researchers is related to performance , defined by the number of crossing edges versus the computation effort for determining the optimal layout
16
configuration. In the current research, the most important concern, performance-wise, is computing the layout fast enough to ensure a good user experience. Tim Dwyer [15] proposes a force-based layout algorithm which is able to finish the computation for 57, 400 and respectively 1138 nodes in 1.43, 1.39 respectively 8.4 seconds which is fast enough for the goal of this research. To conclude, the following design decisions were taken: use of a force-based layout to draw a graph which gives an overview of the ontology in which facilitates the transition from classes to instances.
4.3. Technical choices
The first fact which is discussed is whether the software is a desktop or web-based application. Although Protégé, one of the most well-known ontology management applications, is desktop based, we chose to develop a web application because it can be accessed instantly (without installing any additional software) by the users of the application through a computer browser. Another advantage of building a web application is that it is platform-independent (final goal is to also make it be browser independent).
The next choice is the drawing technology which makes creating the graph visualization possible. Lienert et al. [16] gives a comprehensive overview of drawing technologies which are available in browsers: Adobe Flash, Microsoft Silverlight, JavaFX, SVG, Canvas, WebGL. Adobe Flash is a multimedia framework which is designed to create multimedia applications. It has its own rendering engine, cross platform compatible, and it is capable of displaying vector graphics. The rendering engine is included in Flash Player which comes as an extension/plugin that can be installed in the browsers. Microsoft Silverlight, developed by Microsoft is similar to Adobe Flash and uses a subset of Microsoft .NET framework which provides a render engine. Similar to Adobe Flash, it if delivered as a plugin which has to be installed in the browser. Another similar framework which can be used for building visualizations is JavaFX and it is developed by Oracle. Its rendering engine is based on Java Runtime Environment which is also something additional which has to be installed in the browser. The previous mentioned technologies have their own authoring tool for building applications and they communicate with the rest of the html page through ActionScript or JavaScript. Scalable vector graphics (SVG) is a XML markup language, recommended by W3C, for drawing vector graphics in two dimensional space. All major browsers offer native support for SVG technology (according to http://caniuse.com/#search=svg) which means that no additional
17
software is required for using it. SVG objects can be embedded in html pages and they can be handled by JavaScript. Canvas is an HTML element which is manipulated by JavaScript to draw images pixel by pixel. As SVG technology, it is a native technology to all major browsers (according to http://caniuse.com/#search=canvas). WebGL (Web Graphics Library) is a JavaScript API used to render 2D and 3D images. It is faster than SVG, and it can be used to create complex animations which are not possible using the regular JavaScript API to manipulate Canvas or SVG. No browser plugin is required to use this technology, but the adoption rate is rather small (not yet fully supported by Firefox or Opera, no support for Internet Explorer 10 or Safari 8 and below, according to http://blog.pluralsight.com/webgl-basics).
Overall, Adobe Flash, Microsoft Silverlight and JavaFX have the disadvantage of not being native technologies to the browsers. Because the goal of the application is to make it accessible to people without asking to install additional plugins, they are excluded from the list of possible solutions. Therefore there are three technologies which remain in the competition: SVG, Canvas and WebGL. To analyse which option brings the most benefits for creating a system which can be widely used by people, we evaluated SVG, Canvas and WebGL technologies based on the following criteria: - browser support (because the application targets a wide range of users); - the ease of implementing panning, clicking and zooming functionalities (because the amount of time given for implementation is limited ); - performance (because a large number of elements must be rendered while keeping a good user experience).
Browser support represents the ability of a browser to handle a specific technology or piece or functionality. In the web development world, this notion is strongly connected with the browser version and, sometimes, to the operating system which is used. It is very often the case that one of the technology to be partially supported by a browser, which means that not all of the functionalities described by the standard which defines the technology work on the browser. Also, it might happen that a piece of functionality works only in specific situations in which we say that it is partially supported. For the purpose of this research, only desktop browsers are considered because drawing large graphs on mobile devices (with smaller screens, less memory and slower processors) is a another topic. The most worldwide used browsers, on desktops, between August 2014 - August 2015, which cover 96% of the share market, are Google Chrome, Internet Explorer, Firefox, Safari, according to statcounter.com. The browsers support assessment for the SVG, Canvas and WebGL technologies is based on these 4 solutions:
18
Browser Version SVG Canvas WebGL
Chrome 44 Supported Supported Supported
45 Supported Supported Supported
IE 10 Supported Supported Not supported
11 Supported Supported Supported
Safari 7.1 Supported Supported Partial support
8 Supported Supported Supported
Firefox 39 Supported Supported Partial support
40 Supported Supported Partial support Table 1 - Browser support for SVG, Canvas and WebGL technologies
So far, it can be seen that SVG and Canvas technologies are fully supported by the selected browsers, while WebGL can rise compatibility issues.
The second aspect which was considered for making the choice between the three technologies how straightforward it is to implement mandatory functionalities for the visualization tool: panning, zooming and clicking elements.
Zoom Pan Click events
SVG API methods: API method: svg elements scale(in float scaleFactor) translate(in float become part of the scaleNonUniform(in floatscaleFactorX, x, in float y) HTML dom in float scaleFactorY) model which means that clicking an element is an embedded functionality of the browser
Canvas API method: API method: supports on-click ctx.scale(x, y); void handlers, but on ctx.translate(x, y); the canvas level, without being able to identify the
19
object which is clicked
WebGL zoom objects by manually defining the translate objects possible through transformation equation (zoom factor * by defining the WebGL objects object coordinates) desired position of model each object Table 2 - Technical means to implement the zoom, pan and click events handlers using the SVG, canvas and WebGL technologies
As it can be observed, all three options offer support (to a certain extent) for the requested operations. Canvas and SVG offer specific methods for zooming in and out, while in WebGL the situation is more complex because the programmer itself has to write the transformation equations. The same situation holds for translating objects which is easier to perform with Canvas and SVG and more laborious in WebGL. An important difference between SVG and WebGL, on one hand, and Canvas on the other hand is that in the first ones, the click events triggered by the browser offers information about the object on which the click was performed, whereas on Canvas, the browser offers details about the coordinates of the click event. This is not surprising because Canvas object only handles rasterized images, it does not keep in memory any objects (like circles, rectangles, etc) which are drawn using the specific API. One questions which pops up when talking about Canvas not keeping in memory the model base on which the image was built, is why is it possible on WebGL, since it uses the Canvas object to display the image? The answer is intuitively: WebGL is a software layer on top of Canvas which contains the required functionalities for delivering information about the objects on which users click (among many others). This short analysis is enough to exclude the Canvas technology (by itself) from the list of possible technologies for creating the desired user interface. The last criterion of the selection process is the performance of the rendering process given a large amount of elements (nodes and vertices which connect them). Although both SVG and WebGL are able to handle 3D graphical representations, the proposed user interface only uses the 2D graphics which means that WebGL will not be used at its full capacity. To be more concise, the only elements which have to be drawn are circles (for the nodes of the graph), lines (for drawing the vertices), arrows (for showing the vertices orientation) and text (for labeling the nodes). Therefore, the performance comparison between SVG and WebGL should be assessed by measuring which technology is faster at rendering these types of elements. This topics has been addressed by Hyung et al. in [17] and Moelker et al. in [18]. They investigated the performance of WebGL on browsers for desktop and mobile devices. Li X et al. in [19] evaluates the performance of SVG technology compared to canvas. The commonly accepted idea in the web developers community is that SVG is slower than WebGL but, there is no study
20
which compares these two technologies to show under which circumstances this hypothesis holds.
It can be seen that each technology has advantages and disadvantages. The most important features of SVG are that elements become part of DOM and the straightforward API for drawing them. On the other hand, WebGL is faster than SVG and, after some preliminary steps, drawing the objects does (at 2D level) requires a basic understanding of how vectors and matrices work. From the browsers compatibility perspective, it can be observed that SVG has a slight advantage, but, on long term, it will disappear because more and more browsers (and computers) will become WebGL compatible. One advantage of SVG technology is that using it requires minimal understanding of drawing because there are primitive functions for all basic geometrical figures. On the other hand, using WebGL involves understanding the mathematics involved in 2D graphics and the technology specific steps for starting drawing in WebGL. Because of this, SVG technology was chosen.
One way of drawing the graph visualization is using plain JavaScript functions provided by the SVG api, but there is an alternative to this: using a library on top of JavaScript. The later options makes drawing less complex, from the programming point of view, by embedding multiple functionalities into a single function. Existing libraries like L1, L2, L3, L4 (see the table below) were designed for SVG graphics. L4 offers features for drawing charts and for placing the elements in the visualization according to layout algorithms (like the ones exemplified in section 4.2). L5 implements a broad variety of layout algorithms, including the force directed layout, which was chosen at section 4.2, but it does not offer features for manipulating SVG elements. The other libraries do not provide an implementation of the force directed layout, therefore the easiest and final choice for the drawing library is D3.js.
Library name URL
L1 Snap.svg http://snapsvg.io/docs/
L2 SVG.JS http://svgjs.com/
L3 Raphaël http://raphaeljs.com/
L4 D3.js http://www.d3js.org
L5 Mike Bostock Layouts https://github.com/mbostock/d3/wiki/Layouts
Table 3 - List of SVG libraries which are candidates for the implementation of the graph visualization
21
The application developed for the purpose of this research contains only html, css and javascript files, therefore it can be deployed on any http server. For the purpose of this research (and without reducing the generality of the application), it was deployed on an Apache HTTP Server. As it was already mentioned, the data source is represented by an ontology which has to be exposed by a triple store. Among the various available solutions Virtuoso OpenLink server, version 7.1 is the selected option. This server was configured to accept cross-origin HTTP requests because the html application initiates the queries the data from the Apache server to the Virtuoso server, using ajax requests.
4.4. Implementation
The main goal of an ontology visualization system is allowing users to make a clear image about the real life concepts which are modeled in the knowledge base. According to Shneiderman in [16], the general information discovery procedure is performed in a few steps: overview first, followed by zoom and filter and, for more specific cases, seeking for details. Having this in mind, Shneiderman defines 7 tasks which should be possible to perform by visualization systems: - overview; - zoom; - filter; - details-on-demand; - relate (show relations between concepts); - history; - extract.
The implementation process of the ontology visualization tool took part in two incremental phases which were separated by users testing (which is detailed in the evaluation section). Out of the seven tasks defined by Shneiderman, five of them were implemented and improved: overview, zoom, filter, details-on-demand and relate.
22
4.4.1. First phase The following image offers a snapshot of the first version of the visualization tool:
Figure 1 - Overview of the visualization system
The next paragraphs explains how this visualization tool was designed to support the five tasks.
4.4.1.1. Overview The overview is given by a graph visualization in which each node represents a type of entity (owl:Class) and each edge, shows the inheritance relationship between two entities. Nodes are represented as circles and edges as lines with arrows assigned to them for showing the direction of inheritance relationship. Each circle has a textual label associated to it, which is the rdfs:label property describing the class or, when it is not available, the label is extracted from the class URI by taking the string of characters which come after the last “/” from the URI. The introduction chapter mentioned the situation in which a knowledge base is formed by joining multiple knowledge bases which has as a consequence a higher number of entities types in the visualization but it can also translate into having similar entities types displayed in the user interface. For example, if one would create a knowledge base by joining DBPedia with LinkedGeoData, the final knowledge base would contain similar entities types and also similar individuals; to be more specific: it would contain both
23
which is “city” concept from LinkedGeoData and
4.4.1.2. Zoom and pan The second piece of functionality (which also brings a contribution to the overview) is zooming, which, according to Shneiderman , is used by people view only the portion of the data which is of interest for them. In this project, two zoom levels were defined: the first one contains only the nodes which have children (with respect to the rdfs:subClassOf relation) and the second zoom level contains the remaining ones. In both zoom levels, the nodes are connected to each other through directed links. For example, DBPedia the Event type has five direct subtypes: MusicalFestival, MilitayConflict, Election, FilmFestival and SportsEvents. The SportsEvents type has its own sub types: Olympics, Race and TennisTournament. Under this circumstances, the Event and SportsEvent are part of the first zooming level with an arrow pointing from SportsEvent to Event:
24
Figure 2 - Illustration of the class-subclass relation on the first zoom level
The second zoom level includes the remaining nodes:
Figure 3 - Illustration of the class-subclass relation on the second zoom level
Switching from the first level to the next one is performed using the mouse scroll button. Overall, switching from the first zoom level to the second one means jumping from 53 nodes to 209 nodes in DBPedia and from 26 to 1105 in LinkedGeoData. There are intermediary steps before switching from one zoom level to another in which the length of the links between nodes is increased (when users zoom in) or decreased (when users zoom out) which is meant to improve readability. For example, the PowerThing class from LinkedGeoData is the parent class of other 16 classes, which creates a few label collisions in the graph drawing:
25
Figure 4 - Exemplification of labels collision
One solution for overcoming this problem is to zoom in further into the graph until the distances between nodes becomes large enough to make the collision problem disappear:
Figure 5 - The label collision is solved by zooming in in the ontology
26
A side effect of being able to zoom in is that the visualization becomes too large to fit the computers screens. To solve this problem we implemented a panning function which which allows users to drag into the visible area of the screen the portion of the graphical representation they desire. The zoom and pan implementation is performed by using the SVG transform attribute which receives as parameters the transformation functions, in our case “translate” for panning and “scale” for zooming. In other words, when users pan the visualization (by moving the mouse while the click button is pressed) the mouse coordinates on the screen are taken and a transition on the SVG image is performed. The same happens when users use the mouse wheel to zoom in or out: the SVG image is scaled according to the direction of the scroll wheel movement.
4.4.1.3. Search function An additional functionality which comes to support the zooming function is the search function. Because SVG technology was chosen, which makes each node and also the label associated to it, part of the DOM, the embedded search function offered by the browsers can be used, so searching a specific entity type is as easy as searching a specific word in a normal page. Users have to activate the search box from the browser and insert a keyword. Next, the browser will highlight the labels of the nodes which match the criterion. An indication of the number of results is also presented by default in the browser. The search function covers only data which is displayed in the graph visualizer, therefore, users are able to find classes by typing their name.
4.4.1.4. Filtering The third implemented feature, filtering, is designed to display a list of instances of classes for a specific class which is chosen by clicking a node from the graph visualization. After this happens, the users are redirected to the table which contains the list of individuals of the selected type. Because the number of items which are to be displayed can vary from less than a hundred (i.e. 13 formula one teams in DBpedia ontology, English version) to a few millions (more than 2 million Persons in DBPedia ontology, English version), loading all the data at once is not effective. To solve this problem, the following loading strategy was adopted: the first chunk of individuals are loaded first and, when the users scroll to the end of the list, another chunk of individuals is loaded. This procedure can continue until there is no more information to load.
4.4.1.5. Details on demand The details-on-demand function is achieved on the instances level only under the form of a popup dialog which shows up when users click on an title from the instances list. For example, the DBPedia instance for Curacao is displayed like this:
27
Figure 6 - Detailed view on the “Curacao” instance
4.4.1.6. Relate Showing connections between the class displayed in the user interface is graphically visible in the nodes of the graph, which have an pointer attached to indicate the direction of the rdfs:subClassOf relation. To strengthen this, all the children nodes of a node over which the mouse is over are highlighted (as it can be seen in the next image).
Figure 7 - View of the class hierarchy when the mouse is over the class “Animal”, which is the parent of classes “Arachnid”, “Fish”, “Insect”, “Mammal”, “Bird”, “Reptile”.
28
4.4.2. Second phase The second implementation phase brings improvements to the system based on the feedback received during the evaluation with the users (see section 6.1. for more information). It also introduces new features focused on improving the knowledge base quality by involving users in the ontology curation process. Overall the user interface was re-organized by: - removing the following features: - using multiple colors for the nodes in the graph (based on the ontology to which they originally belong) - the tooltips associated to each class, - modifying the application layout: - the list of items is displayed on top of the graph visualization in a collapsible panel - the detailed information about an individual is displayed on top of the graph visualization in a collapsible panel - adding new modules: - vertical navigation toolbar - search function - ontology smells module The final user interface with the list of items opened and with the search and ontology smells modules looks like this:
Figure 8 - Overview of the graph visualization with the list of instances displayed
29
The following paragraphs explains how the modules which form the visualization have been changed.
4.4.2.1. List of individuals module The users testing revealed that the transition from the graph visualization to the list of individuals is confusing. To solve this problem the list of individuals module was re-configured to be shown as a box on top of the graph visualization (with the possibility of hiding it on the right vertical bar). The width of the widget is 25% of the entire window, full height, therefore the graph is still accessible and the users are able to browse it even when this widget is displayed on the screen. As it was already mentioned, the number of items in the list of individuals module can vary from around a hundred to more than 1 million. In the design from the first iteration the situation was handled by loading the records page by page, as the user scrolled through the table which proved to be difficult to use when users were looking for specific individuals. To improve that situation, the new design of the widget allows random access to the paginated results (one user can jump from page 1 to page 8, for example, without having to go through the intermediate pages). Although this choice brings advantages to users, it rises the challenge of accessing the data from the triple store in a quick manner. In this context, a page of records is defined as a chunk of limited size of individuals from the knowledge base. Each page has a number (or index) associated to it which defines the section of records from the entire dataset the page refers to and it also defined by the number of records it contains. If we assume that a page contains 10 records, displaying the page number 2 of the dataset means displaying all the records between the 11th and 20th ones (including them). It can be seen that the order of the records matters and the question that follows is which is the right order which should be considered. One option is to use the default order which is given by the triples store, yet another alternative is to order the records alphabetically by their label/name. Queries which do not specify any ordering of the records (default ordering) are fast on their execution but using them for displaying the records in the list of individuals has the disadvantage of not showing the items in a specific order, which is not intuitive for users. The other option, of getting the individuals ordered by their label or name, allows users to understand the way the data is ordered but it takes longer to process. This problem becomes more serious when the query engine has to deal with millions of records which have to be retrieved in a specific order, even in the cases when only a small fraction of them has to be returned. The reason behind this is that all the records have to be sorted first and only after this procedure is finished, the query engine can return the desired ones with respect to the pagination parameters. Another constraint is given by triple stores themselves. From performance considerations, they do not allow queries that run for too long or which have many records to process. Taking into consideration the previously mentioned aspects and also that ontologies contain classes with both low and high number of instances, we consider it is necessary to implement
30
both querying strategies. Initially, the query which retrieves ordered results is issued to retrieve the first 10000 results (or less). If running it takes more than 30 seconds (which is configurable in the application) the second type of query is used. The same query is run until all the individuals are retrieved from the store. The list of individuals is ordered by label in the browser, so even when the second type of query is in action, the results are still ordered in the user interface. The list of individual module contains a progress bar which indicates the fraction of data which is loaded from the server. Even if when the data is not loaded completely, the users are able to navigate through the existing results or to use the inline search box.
4.4.2.2. Detailed information module On the other hand, visualizing full information about an individual in a dialog box proved to be more clear to the users, even though the transition from the graph visualization to the list of individuals is still confusing. To keep the detailed individuals visualization clear while improving the way the list of individuals is shown, the following decisions were taken and implemented: - the list of individuals is displayed on top of the graph visualization (with the possibility of hiding it on the right vertical bar) - the detailed information panel is takes 75% of the screen and, together with the list of individuals, cover the screen entirely. The following image presents the detailed information module:
Figure 9 - The list of instances is displayed on the left side of the window while the details of an instance is shown in the right side
31
4.4.2.3. Vertical navigation toolbar The vertical navigation toolbar is used to activate the collapsible widgets which are available in the user interface, so it contains one icon for each collapsible widget in the page, as it can be seen in the figure below. Clicking on one icon makes the corresponding module appear on the screen while the icon itself is hidden. The toolbar is placed on the right side of the screen.
Figure 10 - Vertical navigation toolbar
4.4.2.4. Search module The evaluation process revealed that the search function on which the user interface is based has serious limitations which prevent users from seeing the results. Although users are notified with the precise number of results and they are properly highlighted, the browser is not able to focus the visible area of the page to the nodes which are the responses to a search query. This often occurs when nodes are positioned out of the visible area of the screen. Such behaviour is not unexpected if we take into account that the drawing area is larger than the screen size and the horizontal and vertical scroll bars were replaced by the panning function (as it has been explained in 4.4.1.2). In this context, no browser is able to locate the coordinates of the node on focus and to point the view to those coordinates. To solve this problem, a new search widget is proposed, which is able to focus the visualization on the results. The structure of the widget (see Figure 11) is similar to the one embedded in the browser: a textbox for keywords, two arrows buttons to navigate between results, a textual section for showing the number of results and the close button. As it was mentioned before, the search process runs over the node labels and it is not case sensitive. Going back and forward through the results list is performed by clicking on the arrow buttons from the search widget, which will move the focus of the interface to the current result item. This movement is smooth and allows users to see the direction towards where the interface is driven, so that the users’ mental model about the graph visualization is not lost. Shifting the focus from point A to to point B, where point A is the current view of the graph and point B is the view of the graph in which the node B is in the center of the screen, is performed by an SVG transition which takes 750 milliseconds.
Figure 11 - Search toolbar
32
4.4.2.5. Ontology smells detection module The “ontology smells” term is inspired from the software engineering field in which the term “code smells” [28] refers to flaws in the code which are detected by software analysis tools. The output is used by software engineers to improve the quality of the code which increases the maintainability of the software applications. Similar to that, the “ontology smells” are detected by automatic tools and, later on, the domain experts play the role of software engineers when they are improving the knowledge base by evaluating the “smells”. At this point, based on the results of the evaluation, it can be considered that users can easily browse and search the knowledge base based on the ontology which means they have the tools for finding and reading specific pieces of information. This can be a useful alternative to querying the SPARQL endpoint for domain experts users who can access the information in a more effective way. At the same time it can also be beneficial for the knowledge base because users can improve the quality of the entities by reporting any errors which they discover. Since the project is centered around non-expert users, the types of errors that people can discover are not related to the structure of the ontology, but rather to the data it contains. Among the types of faults which appear in knowledge bases we selected the following three: misclassified instances, missing instances and incorrect links between instances. To involve users in improving the quality of the knowledge base, a module was built on top of the existing visualization tool. It can be activated or deactivated by clicking the corresponding icon from the vertical toolbar described in section 4.4.2.3. The purpose of the ontology smells detection module is to get feedback from users about possible errors in the knowledge base. It does not allow people to correct errors, but rather to confirm if a fault is indeed an error or not. The information about possible errors is fed to the system in JSON format by administrators and it is considered a known fact.
General overview
To activate the ontology smells module, users have to click the corresponding button from the vertical toolbar. After that, all nodes from the graph representation having instances which might have errors are colored in orange. Another way to reflect the activation of the module in the user interface is displaying each individual in the list of individuals in red background whether it is a candidate for having an error.
4.4.2.5.1. Misclassified instances As it has been mentioned before, each instance in an OWL ontology has at least one class associated to it used to define which type of concept from the real world describes that specific instance.
33
For example, “The Netherlands” is an instance of type “Country” and having this instance classified as “Continent” is an example of a misclassified instance. On the other hand, having it classified as “Habitable place” is not an error because any “Country” is a “Habitable place”, but if the ontology contains the definition of type “Country”, having “The Netherlands” only marked as “Habitable place” is partially incorrect because there a class which better describes that entity is available, but it is not used. The previous examples illustrates two flavours of misclassified instances: when an instance has an incorrect type and when an instance is classified with a more generic class while a more specific type is defined in the ontology. Because the goal of this module is the validation of possible faults, enhancing users to manually add the right class is out of scope. Therefore the focus is restricted to discovering incorrect types associated to instances. To handle this situation, the detailed information module from the user interface, displays classes associated to an instance as removable tags which allows users to to remove a tag when the class which it represents is not correct.
4.4.2.5.2. Incorrect similarity links between individuals The purpose of this functionality is to determine whether the owl:sameAs relation between individuals is correct or not. As it was mentioned before, the owl:sameAs relation is used to mark instances of classes as being similar. For each instance which has a similarity link that can be faulty, the detailed information module displays a link, together with the similarity score, which splits the view vertically, in two parts: the left one containing the information about the selected instance while the right one displaying information about the paired instance. After reading the details of each individual, users are requested to press a button which confirms or reject the link. Avoiding giving an answer by pressing the skip button is possible.
Figure 12 - The left panel contains the list of instances of class “City” with their similarity score. The panel from the middle is the instance of class “Zadar” from DBPedia, while the panel from the right represents the paired class from LinkedGeoData (as suggested by the matching algorithm). The toolbar from the middle, which is highlighted with red contains the buttons which users have to use to express their opinion on about the similarity between the two instances of class.
34
5. Evaluation
The main purpose for conducting user evaluations is to find out if the designed system is suitable for ontology understanding and error reporting by non-experts. This process is synchronized with the application development phase: the first phase is centered around the ontology understanding and visualization and the second one is mainly focused on error validation. Having the results of the first evaluation phase applied to improve functionalities related to the ontology visualization, we decided to use the second phase to ensure that these improvements are effective and do not hinder the understanding of the ontology.
5.1. Targeted users The selection of users for the evaluation process was done by considering the following criteria: - have basic or no knowledge about ontologies and ontology engineering (users with advanced knowledge about ontologies were excluded); - are able to understand hierarchical data representation; - are domain experts in the field described by the chosen ontology.
5.2. Data Choosing the dataset for user evaluations has to take into consideration the following aspects: - the ontology must make use of rdfs:subClassOf to represent hierarchical data structures; - the knowledge base and ontology schema must be understandable by users (users must be domain experts); - the ontology has to be large (hundreds of classes and millions of instances); - the ontology must be freely available.
Among the publicly available ontologies, DBPedia, Wikidata and LinkedGeoData are the ones on the short list which meet the previously-mentioned criteria. DBPedia and Wikidata ontologies are similar, but, taking into consideration that the Newz ontology, the starting point of this research, is based on DBPedia, picking DBPedia between these two is a natural choice. LinkedGeoData ontology was built based on OpenStreetMaps, therefore it is an ontology mainly used for spatial representation of data. Even if it can be easily understood by people, DBPedia is more suitable for the scope of general understanding of an ontology because the information presented has a broader scope. Considering this, DBPedia was selected for the user evaluation phase.
35
The DBPedia ontology can be accessed via a SPARQL endpoint at dbpedia.org/sparql or it can be downloaded and imported on a local server. The latter option was chosen in the current research because the ontology which is stored on the DBPedia live servers is continuously changing, whereas a fixed dataset is more desirable for evaluation purposes. Importing the DBPedia ontology has been performed by splitting the single DBPedia dump file (which was downloaded from DBPedia website) into files of 500MB each (as they can be more easily handled by the Virtuoso triple store) and later loaded on the server. This data is sufficient for evaluating the ontology visualization tool. The second stage of the research is related to error validation from the ontology matching perspective. This means that additional data is required: instances of classes originating in the LinkedGeoData ontology together with the information about their paired instances from DBPedia. Because LinkedGeoData does not provide dumps of a specific selection of data (instances of types continent, country, city and theatre), they were downloaded using a multithreaded Java utility based on Apache Jena, designed for this purpose. After that, the downloaded triples were imported on the local server, which also contains the DBPedia ontology. Pulling the data directly from the LinkedGeoData live environment, at runtime, would have been an alternative but, as it has been mentioned before, having a fixed dataset is preferable. The missing information which is needed for evaluating the ontology smells module is the similarity score between paired instances from the two ontologies. To obtain them, the online website of LogMap [21] instance matching algorithm was used. The input data consisted in triples about theatres (from Europe), cities (from Europe), countries and continents from local DBPedia on the one hand and similar information from the LinkedGeoData ontology on the other. One of the encountered challenges is that the online version failed to retrieve any matches when the input files contained more than 5000 instances. This is the reason why the range of theaters and cities was restricted to the ones located in Europe. The results of the instance matching process were imported in the visualization system in JSON format. In total, there were 716 matches which were available for users evaluation. The chart below shows the amount of matches from each of the previous mentioned types.
36
Figure 13 - The distribution of matches by type
5.3. Methodology The evaluation of the system consists of semi-structured interviews with people who were asked to fill in a questionnaire based on a selection of tasks which they had to perform. This type of interview was preferred because it makes it possible on the one hand to explore specific topics (e.g. how people react when they have to deal with large amount of data, what are the assumptions they make when they are faced with semi-structured data), and also to gather valuable insights on how the tool can be optimized and improved. The latter advantage is of particular value considering that the respondents have different backgrounds and expertise. The users’ behaviour while using the graph visualizer tool is studied and users are encouraged to think out loud while performing the tasks. The oral feedback given by the users, together with the answers from the surveys filled in during the interviews, represent the outcome of the evaluation phase which are further analyzed in the next chapter.
The surveys used in the interviews were structured in two main sections. The first section does not require using the user interface and is common for both evaluation phases. It consists of reasoning exercises which check if the testers are able to understand the concepts of instance of a class and class hierarchies (including the graphic representation in the graph visualization). At the same time, the figures from the fourth and fifth questions are meant to prepare the users for the real interaction with the graph visualization. The six questions formulated for this purpose are given in Table 4 and the correct answer is written in bold.
Question Answers Why this question?
Dog is to mammals as a duck ● fishes The first three questions are is to ● birds similar and the difficulty level ● reptiles gradually increases. For ● amphibians answering them, users have to picture the subject (duck, Senator is to politician as ● organization university or tunnel) as a university is to ● library subtype or an instance of the ● educational class defined by the correct institution answer. Even if having two ● college ways of reaching the correct answer could be confusing, ● Actor is to person as tunnel is road we consider it close to a real ● to infrastructure world scenario in which a ● bridge concept can be an instance of ● architectural structure a class or a a class itself (for example “car” can be
37
represented as an instance of “mean of transportation” or as a subclass of “mean of transportation”, which is a decision taken by ontology engineers).
Choose the missing label in ● skyscraper This question follows the the picture below: ● bridge same principles as the ● city previous ones but it ● cave introduces the graphical representation of the relations between the two concepts (which is used in the graph visualization).
Choose the correct pair words ● astronaut, person The purpose of this question for the picture below: ● person, astronaut is to check whether users understand the importance of the arrow orientation from the picture.
How would you name the This is an open question. This question was answered relations marked by the green to see which type or relation arrows? Table 4 - List of questions used in the first part of the interviews with users
5.4. Evaluation of the ontology visualization tool The first phase of the evaluation process is focused on assessing to which extent the graph visualization tool enables users to discover and understand the ontology. For answering this question, two scenarios were considered: one which determines if people have an overview of the ontology and another one to detect if users are able to use the tool to discover more detailed information about specific entities. To evaluate the ontology overview scenario, the users were asked to browse through the visualization for a few minutes and answer the open questions from Table 5. Questions 2 and 3 are more related to the topological aspect of the ontology and they are similar to the evaluation performed by Kriglstein et al. in [6], Motta et al. in [8] and they are meant to ensure that users can understand the data organization of the ontology. Even if the first question is straightforward, in combination with the other two, it is meant to reveal if users describe the ontology by
38
mentioning the entities with the most subtypes (because they are more visible in the graph since they have many edges and they occupy a central position given by the force layout) or if they mention the entities which makes more sense to them. In other words, the answers to this question can show if the user’s perception of the ontology is influenced by the structure of the ontology and the way it is displayed or rather by the meaning of the concepts presented in the ontology.
Question Possible answers Why this question?
1. What is the data presented about? nature, science, Show if the graph You can mention in one or two sentences what is the main transport, politics, visualization is subject of the information presented (e.g.: nature, science, general readable and users transport, politics, general, etc) are able to understand the concepts contained by the ontology.
2. Which data type has the most subtypes? Person, Place etc. Show if users understand the hierarchical 3. Which data type has the least subtypes? Chemical organization of the Substance, ontology and its Medicine graphical representation. Table 5 - List of questions used to evaluate the understanding of the overview on the ontology
The second scenario under evaluation in this phase is related to whether the visualization is effective enough to allow users to find instances by their types and to find a specific instance of class. Users were asked to answer 4 questions (listed in the table below) which required performing some tasks. Question Possible answers Why this question?
4. Find five actors and write down their names Any actor from Shows if users are (one per line). the ontology. able to find entities by type.
5. Find five artists and fill in their names (one per Any actor from This question was line). the ontology. asked to see if users try to find the answer by using the “Actor” type from the ontology, or if
39
they vary it by diving into subtypes of class “Actor”.
6. Where would you place “Albert Einstein" in Person, Scientist Shows if users try to the graph and explain your choice? find the most specific type for an instance or a more generic one.
7. Can you find "Albert Einstein" in the table? YES/NO Shows if the table for displaying the instances of classes can be used effectively to find a specific entity. Table 6 - List of questions used to evaluate if the ontology visualization is suitable for finding specific instances of classes
5.5. Evaluation of the ontology smells module This phase is focused on error confirmation from the perspective of ontology matching. This means that possible errors in the knowledge base are shown to users and they have to confirm or reject them. Thas been attracting an increasing amount of interest in recent years. For example, since 2014, an Ontology Alignment Evaluation Initiative (OAEI) event is organized by multiple international parties, to evaluate the ontology alignment/matching systems. The output of this event is a paper which specifies the tests which are performed and publish the performance of each system on the specific tests. As it is defined by Castano et al. in [28], the instance matching procedure gets as input instances of classes from two distinct ontologies, compares them (using specific algorithms) and retrieves pairs of instances (one from each ontology) together with a similarity coefficient. The similarity coefficient is a number between 0 and 1 (in general) where 0 means that the two instances are totally different and 1 means that they are 100% similar. In this case, users have to say whether the Rijksmuseum (http://dbpedia.org/page/Rijksmuseum) entity form DBPedia ontology is the similar to the Rijksmuseum entry (http://linkedgeodata.org/page/triplify/way29989787) from LinkedGeoData ontology (an exhaustive list of details extracted from dbpedia and linkedgeodata about these two entities can be found in the appendix). This operation is performed using the ontology smells module, which is the focus of this evaluation stage. To do this, each user was asked to evaluate matches from the DBPedia and
40
LinkedGeoData ontologies about continents, countries, cities and theaters. Any match can be evaluated only once.
41
6. Results
This chapter is divided in two main parts in which the results of the first and second evaluation phase are presented.
6.1. First phase
The first evaluation phase of the visualization tool was performed by four users who estimated their competences in Semantic Web technologies above average (above 3, on a scale from 1 to 5), of which one of them is a senior programmer while the other ones have basic knowledge of programming. The introductory questions were successfully answered by the people, with one exception: one user (novice in programming but with average knowledge in Semantic Web technologies) classified the “tunnel” as “architectural structure” which is rather a lack of precision than a mistake. The open text question which asked users to name the relationship between two entities received the following answers: “belongs to class of”, “is a”, “analogy”, “is a specific form of”. The first two answers were given by people who were involved in projects related to Semantic Web, which explains their precision in naming the relation. The same accuracy was encountered when the participants answered the question related to the general topic of the ontology (“What is the data presented about?”). More than that, after browsing the ontology, three of the users already knew they are looking at DBPedia. At that point we decided to focus the evaluation only on the user experience because the answers to all the other questions could be biased, since the users were already familiar with the ontology. At the same time, this was encouraging because it means that people were able to identify the right ontology based on the designed visualization system. During the interview users did not find any added value in presenting the ontology from which the data comes from and they totally ignored that information. More than that, even if the name of the graph in which the triples are stored in the knowledge base were shown on the screen, the people who successfully identified the ontology which they were browsing did not pay attention to that piece of information and still enquired me if they are looking at DBPedia ontology. Therefore, showing this information by using multiple colors in the graph does not improve the results. Another feature which was ignored by users was the tooltip which presents the namespace of each class together with the number of instances. Users find it difficult to locate classes by using the search function embedded in the browser. Although a clear indication of the number of results is shown by default, pointing to the exact position in the graph was not possible, therefore the user users had to use the zoom function combined with panning for finding the desired classes. This was an important flaw in the user experience even for people familiar with the ontology used for the evaluation process. As a
42
consequence, we decided to stop the evaluation process, bring improvements to the visualization tool and test it again with users who better match the profile described in section 5.1. To summarize, the most important findings of this phase are: - presenting the namespace of classes or individuals in the user interface is meaningless to the users; - showing the number of instances of each class does not bring any added value to users; - users are able to cope with a high number of nodes displayed in the graph-based visualisation.
6.2. Second phase This sub-chapter starts with a presentation of the results related to the section of the survey that is common for both user evaluation parts and it continues with analyzing the answers to the graph visualization tool. Next, based on users’ behaviour observed during the interviews, an evaluation of each module, as they were described in chapter 5, is given and the chapter ends with the assessment on the errors confirmation module.
6.2.1.Introductory section
The purpose of the first part of the survey is to test whether users who took part in the evaluation process have the potential of understanding the hierarchical organization of data in an ontology. Since this project is focused on non-experts, the first questions asked the users to which extent they are familiar with Semantic Web technologies and with programming. Their answers revealed that most of them (8 out of 10) are novice in Semantic Web, while the majority (7 out of 10) have none or basic knowledge in programming which made the selection of people who tested the application to be in accordance with the goal of the thesis.
43
Figure 14 - Information about users expertise in Semantic Web and programming
The chart below offers the answers to the preliminary questions (from Table 7) which were answered by the users.
Q1 Dog is to mammals as a duck is to birds. Q2 Senator is to politician as university is to educational institution. Q3 Actor is to person as tunnel is to infrastructure.
Q4 Choose the missing label: Space shuttle → Mean of transportation Skyscraper → Building
Q5 Choose the missing label: Space shuttle → Mean of transportation Astronaut → Person Table 7 - List of questions used in the first part of the interviews with users
44
Figure 15 - The distribution of correct and incorrect answers grouped by questions
It can be observed that each question was incorrectly answered by one user. Although the people who filled incorrect answers declare themselves novice in programming and Semantic Web technology, we do not consider it a valid reason for their mistakes. One user wrongly filled in the answer to the first question because he did not know the meaning of term “amphibian”. For Q2, Q3 and Q4 people rather chose more generic answers over more specific ones that existed: they opted for classifying university as organization (instead of educational institution), tunnel as architectural structure (instead of infrastructure) and associated the term “bridge” to “building” instead of “skyscraper”. This does not necessarily indicate a mistake, but a lack of precision in determining the type of an instance which can later impact the perception of the ontology. An additional reason for which these mistakes do not raise concerns is that none of the testers answered incorrectly more than one question, the wrong answers are uniformly distributed among the five users.
The last item in this section of the interview is an open question which requires users to mention in their own words the name of the relation between the concepts which can be seen as an “instance of” or “subtype of” relation. Their answers, which are listed in the Table 8, cover both ways of expressing the relationship which should facilitate the ontology understanding task. One answer that does not speak for itself is “ownership”, to which the person who answered explained that the entity on the left side “belongs” to the entity on the right side of the arrow, therefore he found the word “ownership” a good match for this question.
45
Among all the questions so far, users considered this one the most difficult one, which shows that users have a good intuition about concepts hierarchies but it was difficult to find a definition for them. An important part of them got stuck in this question during the interview and, to help them, we suggested to construct a sentence that includes the two concepts and then extract the core meaning from it.
Programming competences Semantic Web competences Answer
1 0 is to
1 2 Function-Description
5 3 is a subset of
2 0 from particular to general
1 0 specific option of
1 0 inclusion
0 0 specific to general
0 0 is part of
4 0 Ownership
1 0 classification Table 8 - Answers to the last question of the survey associated with the programming and Semantic Web knowledge
Although the answers to the questions are not perfect, we consider that they are good evidences to conclude that users have the required skills to understand the hierarchical organization of the data which forms ontologies.
6.2.2. Ontology visualization
This section presents the results of the survey on the ontology visualization tool by mentioning and analyzing the respondent’s answers, followed by a discussion about behavioral aspects that were observed during the semi-structured interviews.
46
The first three question of the survey were related to the general overview of the ontology. The answers to the first question were expressed by giving a general statement (i.e: “It is about everything”) or by enumerating the concepts (6 out of 10 answers) of the ontology. It was interesting to discover that 75% of the users who described the ontology by enumerating concepts, mentioned the word “Person”. To find a reason for this particularity we analyzed the graph when it is displayed at the first level (because that is the level which was mainly used to get the overview of the ontology) and we discovered that “Person” is the type with the highest number of direct sub types, which makes the node associated to it in the graph more visible than the others. Another remark that is worth mentioning is that users described and browsed the ontology based on their background knowledge: people who are studying medicine, biology or biochemistry mentioned terms like: medicine, biomedical, biomolecules, enzymes, while the one who is studying civil engineering mentioned the term infrastructure. This is important because it shows that the way people picture the ontology is not related only to the structure of the ontology (which is more or less reflected in the graph visualization) but also by their own expertise. This strengthened the premise that the user interface can be successfully used for ontologies which are specialized on specific domains.
The next two tasks required the users to mention the type with the highest/lowest number of subtypes. 80% of the users performed this task just by looking at the graph itself, mostly on the first level, while the rest questioned their answer by checking the amount of instances of each type. Despite this, 90% of the users correctly identified the type with most subtypes (Person), while one of them identified the class with the most instances (Thing), which was not in the scope of this item. On the other hand, finding the types with the least subtypes proved to be a failure to the users since only 30% of them successfully performed the task. Users’ strategy for finding these types was to locate isolated nodes in the graph(although any leaf node would have been a good answer), but they did not take into account that more nodes are added on the second visualization level, and a leaf node in the first level might become a non-leaf node in the second visualization level. People’s reasoning for finding the requested types was indeed correct, but it was incorrectly applied. One reason for this failure is that the user interface does not provide any indication about the additional data which exists when it is not yet displayed. Another conclusion which can be derived from the observed users behavior is that the arrow which indicates the type → subtype relation is useless because users understand the relation between two types by checking which of the two types is more generic. Therefore a simple line between two nodes in the graph is enough and people will automatically deduce the direction of the relationship, based on the meaning of each node.
47
Figure 16 - The left side of the image presents the current graphical representation of the direction of parent - child class relation while the right one shows a potential simplified version of it (after removing the arrows).
The next section of the users evaluation checked if the designed interface is well built to allow finding specific instances of classes in the ontology. People who took part in the evaluation process did not encounter any issues with finding actors or artists in the knowledge base, it was an easy task which was 100% correctly performed. However, 40% of them found the second question redundant because an “actor” is an “artist”, which is a positive thing because it shows that people relate with the data they are dealing with. Among those, 2 users gave identical answers to the first two questions (by copying and pasting the answers from one question to the other one). Another person did a step forward when it comes to reasoning about the questions involving the artists and answered to this question by mentioning instances of artist, fashion designer, musician and writer, which are all subclasses of class “Artist”. The goals of these two questions were partially achieved because users proved that they were able to find instances by types, but, on the other hand, the amount of people who did some further reasoning on the data types (with respect to subclassOf relation) only reached 40%. This might have happened because the question was not well formulated, the answer was too direct: users could have localized the “Actor” type just by using the search module. One way to rephrase the question would be to ask users to retrieve actors who had roles in animes. This scenario would work because users could go to the type “Actor” where they could discover the type “VoiceActor” for which there are plenty of instances which are related to animees.
The last questions of the survey tried to find out if users are able to use the interface to find the right type for a specific individual. To place Albert Einstein in the graph, 3 of the users type the word “science” in the search box associated to the graph, while the others used the “Person” class as a starting point and analyzed its children classes. Two out of 10 users searched for physicist class, but since it is not defined in the ontology, they mentioned the class “Scientist”. This task was successfully performed by all the users but finding Albert Einstein in the list of
48
instances of scientists proved to be a challenge: only 70% of them managed to perform it. The other 30% failed because they looked for it before the entire list of instances was loaded.
Discussion Overall, the results show that the proposed graph visualization can be used to create a good overview of the ontology. Users mainly looked at the first level of visualization (which shows the non-leaf nodes) to create a mental model of the information which was presented. This means that showing ~22% of the data types (98 out of 444), together with the type related relations between them, proved to be enough for the people to have the feeling that they have a good view on the data. There are three non-exclusive explanations for focusing mainly on the first level of visualization: - according to users, the additional information that is brought in the second level of visualization does not show any new concepts. They are regarded as refinements of the concepts portrayed in the first level. In short, focusing on the latter does not bring added value related to a better understanding of the ontology. - the visualization becomes larger than the screen, therefore users cannot see entirely in a single screen, which makes it difficult to comprehend. Also, the number of concepts (444) displayed on the screen becomes too high for users to even read them; - the transition from the first level to the second level is takes around 30 seconds, required for the force-based layout to finish the computation, in which the nodes which form the graph keep moving on the screen.
Using panning to reveal the parts of the graph which are not in visible on the screen was well understood by users and proved to be useful especially when the entire graph was shown. Zooming in and out was used to increase the visibility of the nodes in the graph when labels overlapped on each other especially on nodes with many children. The same feature was used and to switch between the visualization levels, but this scenario did not meet the users’ expectations. One issue which was reported by all users is that the switch between the levels occurs unexpectedly, without any notification in advance. This problem has a higher impact when occurring while users zoom the visualization to take a closer look on a specific part of the ontology (either to avoid names overlapping or just to isolate the part of the in which they are interested) because the new nodes which are added change the equilibrium of the force layout (which is used to compute the position of each not in the graph). In consequence, until the new layout configuration is computed again, the nodes are moving on the screen. Another reported problem is that nodes do not keep their position on the screen when switching between layers or after the page is refreshed. For example, one user will lose the focus of one node (or group of nodes) and the mental model of the way the data is organized when the transition from the first to the second level of visualization is performed. This happens because
49
the force-based layout algorithm is non deterministic, therefore the layout configuration which is the output of the algorithm can be different between multiple runs with identical input.
Detailed information module Users had a positive reaction related to the detailed information module without noticing any major usability problems. They used it for reading more information about a specific instance and, in many cases, they used the links inside this module to navigate to external pages which provide more details about an attribute or even more information about individual on focus. A good remark which was often mentioned is that some of the information is meaningless for users (information about data provenance, version, etc) or not understandable at all. Two users used external websites to make sense of the geolocation data and see to which location on the globe an entity is assigned. One conclusion which can be drawn from this is that facets-based data presentation is desirable in order to maximize the users understanding of the data.
Search module The updated version of the search module fixed the problem of directing the users to nodes which are part of the results set of a search operation. Although the functionality itself worked well and people were able to locate the desired nodes in the graph, the search module created confusion among users because their first instinct was to search for individuals by name using the search panel, which is designed to search for data types. When their search was unsuccessful, users found an alternative by looking for the data type associated to the entity they wanted to find, click on it in the graph to load all the instances of that type and used the filter function from the individual panels. This behaviour suggest that the distinction between the concept of data type and instance of that type is not strongly illustrated by the user interface. At the same time, the user interface was intuitive enough so that users were able to perform their tasks. Four out of seven users who tested the user interface used the search module for trying to find individuals. The common characteristic of them is that they have limited programming and Semantic Web knowledge. On the other hand, all these users successfully found “Actors” and “Artists” in the knowledge base which suggests that their behaviour is rather related to a confusion than to a poor understanding of the “type - instance of type” concepts. Also, this kind of user interaction could be induced by the “Single Search Box”, a principle intensively used for querying information systems nowadays, which is familiar to all users.
6.2.3. Ontology smells module
The user testing showed no usability faults of the ontology smells module from the individuals matching perspective. The main challenge was related to understanding the concept of similar entities. As an answer to this question, during the semi-structured interviews, users were
50
explained that one entity is similar to another entity if they refer to the same concept from the real world, but the information about them can be presented from two different perspectives. Because this answer was rather abstract for users, an additional example was given: the two perspectives are represented by a geography and a history book which both talk about the city of Amsterdam. Therefore the concept “Amsterdam” from a geography book is similar to the concept “Amsterdam” from a history book. After these explanations were given, the task of confirming matches between entities was properly understood. As it has been mentioned before, the individuals list and the panel which contains the details of a specific instance contains the similarity coefficient (a number between 0 and 1) which express the probability that two instances are similar. Users did not consider the similarity coefficient itself useful, they only used it to identify the instances which require their feedback. In the current context, most users based the choice between “similar” and “different” on the latitude and longitude coordinates of the paired entities. Displaying the instances whose matches are under evaluation process on the top of the individuals list proved to be a good idea because it was easy for users to find them (users were encouraged to pick the instances they wanted to evaluate and they stopped going further in the list of individuals as soon as they observed the first ones which were not marked with the similarity coefficient).
51
Figure 17 - List of the instances of classes module when the ontology smells module is active
The panel which contains the buttons for confirming or discarding matches was trivial, therefore it did not rise any challenges for the users. Also, vertically splitting the panel which displays the details about specific individuals in two (one half for each instance) was successfully used by users and no complaints were received. People used the vertical scroll of each panel for reading the information which was not displayed in the visible area of the screen. One of the concerns that was considered while taking the decision of splitting the panel in half is if this type of design is suitable for users to identify similarities between the data from the left and right panel. There were no complains related to this choice because people initially read the entire information from the left panel, followed by seeking each piece of information in the second panel. In other words, users tried to build an overview of the information from the left panel first and after that they tried to find related information on the right panel. One question which occurs in this case is why did the users choose to first get an overview of the information displayed on the left panel. Considering that by design the information from the left panel is displayed first and only later on, when users request it, the right panel is shown, we can conclude that users focus on the first block of information which is displayed. On the other hand, in this particular case, the information shown on the left panel contains more general data which is easy to understand by people (cities, countries, theatres, continent from DBPedia), while on the right side, the data is more cryptic (because it comes from LinkedGeoData, therefore it is more about coordinates and geo location), so it might happen that people have a preference for the data which is more user friendly. Having a clear explanation for this issue is important because it could improve the speed and accuracy of the decision process if users focus on the data which is easier to comprehend first, and switch to the other information source later.
Matches validation During interviews users were free to choose which matches to evaluate, the only constraint was to evaluate matches of each available type. The ones about continents were quickly gone because there were only six of them. In total, people expressed their opinion about 189 matches, on average almost 19 per person. One encountered challenge was the unstructured data which made establishing equivalences between pieces of information that form the instances involved in matching process not a trivial task. To be able to asses the feedback from the users, we built a golden standard by manually evaluating the 189 matches and partitioning them in 2 categories, by the level of difficulty in establishing if two instances are a match or not, with the following meanings: - Easy: the users are given enough details to be able to take a correct decision without consulting an external information source; - Difficult: users have to consult an external information source to be able to take a correct decision.
52
The amount of information presented to the users strictly depends on the number of triples from DBpedia and LinkedGeoData (the two data sources selected for this research). The current system does not display information from any external source, therefore when users want to read more details, they have to access it either by clicking on the links from the user interface (they are in general redirected to dbpedia.org or linkedgeodata.org or just by using a search engine. Table 9 gives an overview of the pieces of information that have to be contained by each instance to be assigned in one of the 2 categories. “Attribute (0-1)” expression means that one instance from a possible match contains that specific attribute, while the other one does not. When the attribute name is shown alone, it means that both instances have that attribute. Also, other attributes which are not specified in the table are available to the users.
Instance type Easy Difficult
Continent ■ Name -
Country ■ Name ■ Name ■ Capital
City ■ Name ■ Name ■ Geolocation ■ Geolocation (0-1)
Theatre ■ Name ■ Name ■ Geolocation ■ Address/Geolocation (1-0)
Table 9 - Classification criteria which have to be met by instances of classes to fall under “Easy” or “Difficult” to evaluate categories
The results of the matches classification are presented in the following diagram:
53
Figure 18 - Matches grouped by difficulty and class type
It can noticed that all matches about continents and most about cities and countries can be easily evaluated, while the matches about theatres are more challenging. The next paragraphs will present the results of the users evaluation.
In total, users evaluated 159 items and chose to avoid expressing their opinion about 30 matches. If we ignore the skipped pairs of instances, the success rate is 97%. However, since the purpose of the evaluation is to discover to which extent non-expert users are able to confirm or discard matches, a more accurate way of reporting the success rate is by mentioning the ratio between the matches which were evaluated by users and the number of matches which were successfully evaluated, therefore the success rate is 82%. The chart below shows the distribution of the evaluated matches by the difficulty level.
54
Figure 19 - Number of skipped, correctly and incorrectly evaluated matches grouped by difficulty
Correctly evaluated matches In the current context a match is correctly evaluated if the user feedback on it is identical to results from the golden standard (therefore it does not have any relation with the correctness of the instance matching algorithm). The amount of correctly/incorrectly identified matches is depicted in the following diagram:
55
Figure 20 - Correctly evaluated matches grouped by class type
The number of easy to evaluate matches exceeds the difficult ones in all cases except the matches about theatres. This was expected because the same pattern can be found in the golden standard. The success rate for matches of types continent, country, city and theatre is: 100%, 92%, 100% and 62.5%, respectively. The high difference in performance between theatres and the other matches types can be explained by the fact that people are not familiar with these data and this kind of behaviour is expected.
Skipped matches The amount of the matches which were incorrectly reported by the users is low (only 4 in total) while the number of the skipped one reaches 30. The first conclusion which can be derived from this is that people prefer to skip a match, rather than taking the risk of mistakenly reporting it. This is a good attitude as long as the matches which are reported are indeed very difficult to evaluate. To understand when users decide to skip a match, it is required to take a closer look on them. The following diagram reports the number of skipped matches by difficulty level and grouped by type:
56
Figure 21 - Skipped matches grouped by class type
It can be observed that people did not skip any matches about cities or continents, which is understandable because the names that describe them are in a vast majority unique and familiar. One of the matches related to countries (http://dbpedia.org/page/Commemorative_coins_of_Romania) was skipped because the instance from DBPedia was incorrectly marked as a country and the remaining two matches contain instances which refer to countries which used to exist in the past, but which do not have any correspondent nowadays (for example: Malagasy Protectorate which existed until 1897, according to DBPedia). As expected, evaluating matches about theatres proved to be more difficult and the results are shown in details in the following diagram:
57
Figure 22 - Incorrectly evaluated matches of class “Theatre”
It can be observed that the number of skipped matches which are difficult to evaluate represent almost 40% of the total matches from this category. Since this situation is discouraging, further explanations are needed because it is required to understand why users chose to skip so many matches of this type. The following diagram shows the distribution of skipped difficult matches about theatres per user:
58
Figure 23 - Distribution of difficult to evaluate matches which were skipped, grouped by user
66% of these matches were skipped by three people (users #3, #4, #6). During the interviews, we observed that some of the users did not find the latitude and longitude information meaningful, while the difficult matches about theatres can be evaluated mainly based on this piece of information. When all the difficult matches are taken into consideration (not only the ones about theatres), it can be seen that users #3, #4, #6 skipped 50%, 62.5%, respectively 57% of the difficult matches which were under evaluation. At the same time, their evaluation precision was 100% (no mistakes done on difficult matches). This is another reason which supports the conclusion that people prefer to skip a match, rather than taking the risk of mistakenly reporting it. However, there are people who successfully evaluated similar matches (from the difficulty and type of information) point of view, which suggests that the skipped matches could be successfully analyzed by other users, who are not ontology experts. Therefore having multiple users expressing their opinion on the same matches can increase the success rate.
So far, the evaluation was focused on the people’s performance relative to the golden standard but it is interesting to see to which extent users improved the set of matches suggested by the LogMap [22] instance matching algorithm. The following chart presents this situation by comparing the matches suggested by LogMap algorithm with the ones suggested by users and with the golden standard. The results are ordered by the similarity coefficient computed using LogMap and it varies between 0.84 and 1. They suggest that the matches with a low similarity score generated by the algorithm are unlikely to be true, but users can successfully identify the
59
mistakes and correct them, without creating any false positives. The situation is slightly different when the algorithm generates matches with 100% confidence. In this case, around 23% are false positives and users were able to detect 75% of them. Overall, the people who took part at experiments successfully improved the instances matching process.
Figure 24 - Comparison on the correctness of the LogMap algorithm, users feedback with the golden standard grouped by the similarity coefficient given by LogMap algorithm
Misclassified instances The process of finding instances with wrong types assigned to them was not a distinct task. At the beginning of the semi-structured interviews users were asked to report any misclassified instance they observe while browsing the ontology. This proved not to be a good strategy because few people pay special attention to this aspect, hence the low number of reported errors. Users were more focused on exploring the ontology, the way data is organized and tried to understand the main parts of the ontology. In total, 5 users reported 9 errors (2, 2, 3 and respectively 1 reported error per user). In 5 cases users were correct, but in some other 4 others there were wrong. Three out of four wrongly reported errors are caused by a flaw in the user interface: only the label/name of the class is shown in the user interface, not the namespace/schema to which they belong and seeing an instance tagged as “Person” more than one time was considered an error by the user, so 3 types (highlighted in the image below) were reported as being incorrect.
60
Figure 25 - Exemplification of the ambiguity generated by assigning three distinct classes with the same name to an instance
This shows that further improvements can be made on showing the types by including in the user interface information about the namespace (users are currently able to check the definition on each type by clicking on it). In this specific case the full URI of the three types are: http://schema.org/Person, http://dbpedia.org/ontology/Person and http://xmlns.com/foaf/0.1/Person. The other mistake was caused, once again, by a limitation in the way the types are displayed. This happens only when the ontology does not contain information about the title or label of a specific type and the URI of the type is not user friendly. For example, the type “Q215627” which is shown in the previous picture, represents the “http://www.wikidata.org/entity/Q215627” type, which is described as “abstract term defining a being, e.g. a human, that has certain capacities or attributes constituting personhood”, but the DBPedia dump which was imported in the triple store does not contain more detailed information about that type.
Because the users’ feedback on this topic is limited, there are no conclusions which can be drawn. One difference which has to be pointed out is that showing full information on the class type proved to be necessary in the context of reporting misclassified instances, but it is not mandatory when for instance matching or just for exploring the ontology.
Other reported errors The user interface proved to help users with discovering other types of errors, which is something that was intended. One of the persons who was tested the user interface observed inconsistencies related to the owl:subClassOf relation. According to the “MusicGenre”, class definition, the user interface presents it as a subclass of “Genre”, therefore any instance of class “MusicGenre” should have the “Genre” type associated to it, which is not the case with the instances of “MusicGenre” class. The same user observed that the instances of “Engine” are marked as “Device” (“Engine” is a subclass of “Device”) and, by comparing the two situations, he reported that problem. This is an important discovery because it is the first time when an user discovers a generic problem which affects an entire set of instances. It also shows a deeper understanding of the class hierarchies and of the notion of instance of a class.
61
To summarize, the most important findings of this phase are: - the general overview of an ontology is influenced by two key elements: - the way the structure of the ontology is presented (in this case the classes with the most subclasses were always mentioned at question 1, first part of the survey); - the domain of expertise of each user; - the arrows which indicate the direction of relationship between nodes does not bring any added value when users are familiar with the data which is presented ignored the arrows from the links between the nodes, which is a commonly used way to express the direction of the relationship between entities; - no separation between the structure of the ontology and the actual data should exist, because users expected to see instances of classes represented as leaf-nodes in the graph; - showing the similarity coefficient between entities does not bring any added values to users; - giving the option of skipping a match is a good approach because users use that when the information is unclear, which increases the accuracy of the feedback.
6.3. Threats to validity As stated before, the visualization tool developed in this thesis targets the domain experts in specific fields of knowledge. Because accessing this type of users might be difficult, the following strategy was adopted: a general domain was chosen because all people can be considered domain experts. The underlying assumption of this choice is that the same people would successfully perform the same tasks given an ontology specialized in their domain of expertise. Since no evidence to support this was given, this generalization can be considered a threat to validity. Another threat to validity is related to the ontology smells module because a part of the data used in the evaluation is geographically bound, which might increase the success rate in performing the tasks. Last but not least, one threat to validity comes from the low number of users which tested the application. Although the interviews with them revealed interesting insights, more user testing is required to make strong statements about how such application should be designed, therefore a quantitative study on this topic should be performed.
62
7. Conclusions and future work
This chapter analyzes to which extent the research questions were answered in the current work and suggest ideas for future research and improvements. The current work proposed a graph visualization for displaying ontologies and knowledge base browsing. Also, it tested the people’s abilities to decide if similarity relations between instances of classes are correct or not. In addition to that, users were empowered by the graphical interface to report any misclassified instances of class which they observe while discovering the knowledge base. All these things were delivered through a HTML application which is deployable on any http server. Also, the system can connect to any triple store which accepts CORS requests for data querying. These actions were performed for answering to the two research questions which are enumerated and discussed in the following paragraphs.
First research question: How to design a visualization tool for ontologies that provides an effective support for the non-expert users in quickly understanding what are the main areas covered by the ontology and which offers knowledge base exploration support?
The main concern considered when the ontology visualization tool was built is if non-experts can cope with a visualization which displays a large amount of data. The results presented in section 6.2 suggest that users are able to to perform bottom-up (finding the most appropriate class for a specific instance) and top-bottom (finding a specific instance from the knowledge base) exploration of the ontology. They were also able to summarize the main topics existing in the knowledge base and the exploration behaviour was based on their background knowledge (which represented the starting point for further exploration of the data). A key choice of the visualization system is that it is based on a basic relation: rdfs:subClassOf which proved to be easy to understood by people. This choice, together with the introductory questions from the survey (in which the relation was first represented textually and later graphically), allowed users to create a good overview of the ontology. One idea that was used in other researches is to set the size of the visual components which represent classes according to the amount of instances. This concept was not present in the current work because we consider it could influence the way users’ general perception on the ontology. The assumption proved to be correct, because not having multiple dominant entities from the visual point of view (with the exception of class “Person”) showed that the general image of an ontology is created based on the background knowledge, instead on the attributes
63
related to the structure of the ontology (relations between classes or instances, number of instances, etc). Although showing the direction of rdfs:subClassOf relation proved to be useless for users (see section 6.2.2), we consider it a feature which should be present in visualization systems in which the direction of a relation is important because it can become useful when people are looking at concepts which are new to them, therefore making an analogy to realize the direction could be impossible. One answer to this research question is derived from the people’s confusion while trying to find either instances of classes or the classes themselves: even if there is a visual separation between classes and instances of classes, only one common search component should be implemented. The users testing showed that in a visualization system which progressively displays the data, users expect to see an indication of the amount of information which still has to be loaded and shown in the user interface. This is another “take away message”.
Second research question: How to design a visualization tool for ontologies which can help non-expert users detect errors in the knowledge base?
This research question has been tackled from the ontology matching perspective which requires feedback from people to decide if potential matches between instances of classes are correct or not. The main challenge in this scenario resides in displaying the information about each instance clear enough so that users can establish equivalence relationships. The users evaluation proved that people are able to successfully perform this type of task even when the information about instances is unclear or not well structured. Their behavior shows that they first try to understand one entity (usually the one which is presented first) followed by trying to find matching pieces of information in the second one, therefore the recommendation for designing such an interface is to visually emphasize the pieces of data which uniquely identifies instances of classes. Another idea which has been investigated for giving an answer to this research question consists in asking people to report any misclassified instances. Although the response rate to this question was not too high, one conclusion can be drawn: equivalent classes(owl:equivalentClass) should be clustered together in a single entry in the section which displays the information about an instance.
A strong point point of this research is that the designed system is generic enough to be applied to any OWL ontology with a high number of classes. Compared to the systems which were presented in the “Related work” chapter, the current work has been tested using an ontology which contains more than 500 classes by users who (mostly) do not have any studies or work in the computer science domain.
64
Chapter 6 presents some faults which were reported by users during the evaluation phase which we consider the first option for future work because fixing them would improve the user experience. One question which emerged during this research and would be worth of further investigation is whether providing a visualization for the entire ontology brings any improvements to the matches confirmation process. Section 6.2 states that the designed system is suitable both for ontology understanding and for fixing errors in the instances matching context, but no relation between these two processes was established. This is important because if having a good overview of the ontology does not improve the quality of the users feedback in the instance mapping process, then the graph visualization represents just one sophisticated way of navigating to the matches which require feedback, while more simple ones can be adopted.
Another idea for future work which is more about engineering than science, is to find out whether the current system is generic enough to be used as part of the ontology alignment workflow. As OAEI2015 reports in [23], a series of instance matching algorithms like LogMap, AML [24], ServOMBI [25] require users feedback on the similarity relations which they generate. The first question is if the ontology smells module can be fed with information from any of those algorithms. The other issues which has to be solved is how to handle the feedback from the users. The current system uses the OAEI format as input for the suggested matches (so theoretically one part of the problem is solved) but the output is stored in the JSON format (generating the reports from the “Results” chapter was done by manipulating the JSON data coming from the users). In addition to finding the right data format of reporting the results, contextual information about the decision process is also important. The underlying assumption for this statement is that users might perform worse at the beginning of the matching confirmation task but they could improve later on.
65
8. References
1. Horridge M, Drummond N, Goodwin J, Rector A, Wang HH. The Manchester OWL Syntax. Proc of the 2006 OWL Experiences and Directions Workshop. 2006;216. Available: http://ceur-ws.org/Vol-216/submission_9.pdf
2. Falconer SM, Chris C, Margaret-Anne S. A Visualization Service for the Semantic Web. Lecture Notes in Computer Science. 2010. pp. 554–564.
3. Benjamins, R., Contreras, J., Corcho, O., Gomez-Perez, Six Challenges for the Semantic Web. In: KR 2002 Workshop on Formal Ontology, Knowledge Representation and Intelligent Systems for the Web (2002)
4. Kim S-M, Sung-min K, Seong-hun P, Young-guk H. Scalable Visualization of DBpedia Ontology Using Hadoop. Lecture Notes in Computer Science. 2013. pp. 301–306.
5. N-Triples, . Available: https://www.w3.org/2001/sw/RDFCore/ntriples/ 6. Kriglstein S, Simone K, Günter W. Human Centered Design in Practice: A Case Study with the Ontology Visualization Tool Knoocks. Communications in Computer and Information Science. 2013. pp. 123–141.
7. Prümper P, Test it: ISONORM 9241/10. In: Bullinger H-J and Ziegler J (eds), Proceedings of HCI International, Munich. 1999. Lawrence Erlbaum Associates, Mahwah, NJ, USA
8. Motta E, Enrico M, Paul M, Silvio P, Mathieu D, Gomez-Perez JM, et al. A Novel Approach to Visualizing and Navigating Ontologies. Lecture Notes in Computer Science. 2011. pp. 470–486.
9. Catenazzi N, Nadia C, Lorenzo S. Experiences of Knowledge Visualization in Semantic Web Applications. Advances in Intelligent and Soft Computing. 2011. pp. 49–59.
10. Lomov P, Pavel L, Maxim S. Technology of Ontology Visualization Based on Cognitive Frames for Graphical User Interface. Communications in Computer and Information Science. 2013. pp. 54–68.
11. Miller GA. The magical number seven, plus or minus two: some limits on our capacity for processing information. 1956. Psychol Rev. 1994;101: 343–352.
12. Parsia B, Wang T, Golbeck J. Visualizing web ontologies with cropcircles. End User Semantic Web Interaction WS @ ISWC2005. 2005;
13. Storey M-A, Musen M, Silva J, Best C, Ernst N, Fergerson R, et al. Jambalaya: Interactive
66
visualization to enhance ontology authoring and knowledge acquisition in Protégé. in Protégé Workshop on Interactive Tools for Knowledge Capture. 2001;K-CAP-2001.
14. Hu Y. Current and Future Challenges in the Visualization of Large Networks. AT&T Labs–Research, unpublished. 2013; Available: http://web2-clone.research.att.com/export/sites/att_labs/techdocs/TD_100926.pdf
15. Dwyer T, Tim D. Scalable, Versatile and Simple Constrained Graph Layout. Comput Graph Forum. 2009;28: 991–998.
16. Lienert C, Christophe L, Bernhard J, Olaf S, Lorenz H. Current Trends in Vector-Based Internet Mapping: A Technical Review. Lecture Notes in Geoinformation and Cartography. 2012. pp. 23–36.
17. Hyung WK, Yang-Won L. Single and Multiple Thread Programming for Geo-visualization by Using WebGL with Web Workers. Proceedings of the World Congress on Engineering and Computer Science 2013. 2013;1.
18. Moelker RR, Wijbrandi WE. HTML5 data visualization capabilities of mobile devices. 9th SC@RUG 2011 - 2012. 2012; 23–28.
19. Li X, Xianfeng L, Zhiqiang B. Performance Characterization of Web Applications with HTML5 Enhancements. 2014 IEEE 12th International Conference on Dependable, Autonomic and Secure Computing. 2014. doi:10.1109/dasc.2014.52 20. Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations. Proceedings 1996 IEEE Symposium on Visual Languages. doi:10.1109/vl.1996.545307 21. LogMap Web Access, Available: http://csu6325.cs.ox.ac.uk/ 22. Jiménez-Ruiz E, Ernesto J-R, Grau BC. LogMap: Logic-Based and Scalable Ontology Matching. Lecture Notes in Computer Science. 2011. pp. 273–288.
23. Cheatham M, Dragisic Z, Euzenat J, Faria D, Ferrara A, Flouris G, et al. Results of the ontology alignment evaluation initiative 2015. In 10th ISWC workshop on ontology matching (OM). 2015;
24. Santos E, Faria D, Pesquita C, Couto FM. Ontology Alignment Repair through Modularization and Confidence-Based Heuristics. PLoS One. 2015;10: e0144807.
25. Kheder N, Diallo G. ServOMBI at OAEI 2015. Proceedings of the ISWC Workshop. 2015; 200–207.
26. T. R. Gruber. A translation approach to portable ontologies. Knowledge Acquisition, 5(2):199-220, 1993
27. Fu, Bo, Natalya F Noy, and Margaret-Anne Storey. "Indented tree or graph? A usability
67
study of ontology visualization techniques in the context of class mapping evaluation." International Semantic Web Conference 21 Oct. 2013: 117-134. 28. Castano S, Ferrara A, Montanelli S, Varese, G, Ontology and Instance Matching in Lecture Notes in Computer Science, 2011, pp. 167-195
Appendix
URI - DBPedia URI - LinkedGeoData Score Class type http://dbpedia.org/resource/City_of_London http://linkedgeodata.org/triplify/node107775 0.85 City http://dbpedia.org/resource/Thalia_Hall http://linkedgeodata.org/triplify/node1119733684 0.85 Theatre http://dbpedia.org/resource/Thalia_Hall http://linkedgeodata.org/triplify/node1411382042 0.85 Theatre http://dbpedia.org/resource/Riverside_Municipal_Auditorium http://linkedgeodata.org/triplify/node1024921328 0.85 Theatre http://dbpedia.org/resource/Nieuwe_Republiek http://linkedgeodata.org/triplify/node432425099 0.86 Country http://dbpedia.org/resource/United_States http://linkedgeodata.org/triplify/node424317935 0.86 Country http://dbpedia.org/resource/Paris_Theatre http://linkedgeodata.org/triplify/node1489712125 0.86 Theatre http://dbpedia.org/resource/Municipal_Theatre_of_Santiago http://linkedgeodata.org/triplify/node2557803400 0.86 Theatre http://dbpedia.org/resource/Second_Stage_Theatre http://linkedgeodata.org/triplify/node1500073886 0.86 Theatre http://dbpedia.org/resource/Allen_Theater_(Allentown,_Pennsylvania) http://linkedgeodata.org/triplify/node1233404053 0.86 Theatre http://dbpedia.org/resource/Playhouse_Theatre_(New_York_City) http://linkedgeodata.org/triplify/node2016432854 0.86 Theatre http://dbpedia.org/resource/Playhouse_Theatre_(New_York_City) http://linkedgeodata.org/triplify/node1598907894 0.86 Theatre http://dbpedia.org/resource/Theater_Basel http://linkedgeodata.org/triplify/node1268172860 0.86 Theatre http://dbpedia.org/resource/Lisner_Auditorium http://linkedgeodata.org/triplify/node2470040343 0.86 Theatre http://dbpedia.org/resource/Lisner_Auditorium http://linkedgeodata.org/triplify/node2351212848 0.86 Theatre http://dbpedia.org/resource/Lisner_Auditorium http://linkedgeodata.org/triplify/node1209852915 0.86 Theatre http://dbpedia.org/resource/Lisner_Auditorium http://linkedgeodata.org/triplify/node2651653092 0.86 Theatre http://dbpedia.org/resource/Lisner_Auditorium http://linkedgeodata.org/triplify/node2144825177 0.86 Theatre http://dbpedia.org/resource/Lisner_Auditorium http://linkedgeodata.org/triplify/node1377304471 0.86 Theatre http://dbpedia.org/resource/State_Theatre_(New_Brunswick,_New_Jersey) http://linkedgeodata.org/triplify/node1546964592 0.86 Theatre http://dbpedia.org/resource/Vista_Theatre_(Los_Angeles,_California) http://linkedgeodata.org/triplify/node2292413358 0.86 Theatre http://dbpedia.org/resource/Woods_Theatre http://linkedgeodata.org/triplify/node1259832113 0.86 Theatre http://dbpedia.org/resource/Forum_Theatre http://linkedgeodata.org/triplify/node2619979271 0.86 Theatre http://dbpedia.org/resource/Usher_Hall http://linkedgeodata.org/triplify/node1001484787 0.86 Theatre http://dbpedia.org/resource/DramaTech http://linkedgeodata.org/triplify/node1750883902 0.86 Theatre http://dbpedia.org/resource/Fox_Theater_(Bakersfield,_California) http://linkedgeodata.org/triplify/node1467255293 0.86 Theatre http://dbpedia.org/resource/City_of_Preston,_Lancashire http://linkedgeodata.org/triplify/node21436318 0.87 City http://dbpedia.org/resource/Turkmen_Puppet_Theatre http://linkedgeodata.org/triplify/node1139753572 0.87 Theatre http://dbpedia.org/resource/Theatre_Row_(New_York_City) http://linkedgeodata.org/triplify/node1246402205 0.87 Theatre http://dbpedia.org/resource/Theatre_Row_(New_York_City) http://linkedgeodata.org/triplify/node1140902644 0.87 Theatre http://dbpedia.org/resource/Polk_Theatre_(Lakeland,_Florida) http://linkedgeodata.org/triplify/node2413219995 0.87 Theatre http://dbpedia.org/resource/Morecambe_Winter_Gardens http://linkedgeodata.org/triplify/node255387590 0.87 Theatre http://dbpedia.org/resource/Miller_Theatre http://linkedgeodata.org/triplify/node1027880742 0.87 Theatre http://dbpedia.org/resource/Lensic_Theater http://linkedgeodata.org/triplify/node2377507679 0.87 Theatre http://dbpedia.org/resource/The_Park_Theatre http://linkedgeodata.org/triplify/node2198357382 0.87 Theatre http://dbpedia.org/resource/Millennium_Theatre_(Prague) http://linkedgeodata.org/triplify/node314676506 0.87 Theatre http://dbpedia.org/resource/Marina_Theatre http://linkedgeodata.org/triplify/node1311897051 0.87 Theatre http://dbpedia.org/resource/Americas http://linkedgeodata.org/triplify/node36966069 0.88 Continent http://dbpedia.org/resource/City_of_Bradford http://linkedgeodata.org/triplify/node26701367 0.88 City http://dbpedia.org/resource/Warner_Grand_Theatre http://linkedgeodata.org/triplify/node1140517824 0.88 Theatre http://dbpedia.org/resource/Warner_Grand_Theatre http://linkedgeodata.org/triplify/node1403244532 0.88 Theatre http://dbpedia.org/resource/Warner_Grand_Theatre http://linkedgeodata.org/triplify/node141475500 0.88 Theatre http://dbpedia.org/resource/McSwain_Theatre http://linkedgeodata.org/triplify/node1598982544 0.88 Theatre http://dbpedia.org/resource/McSwain_Theatre http://linkedgeodata.org/triplify/node1683587406 0.88 Theatre http://dbpedia.org/resource/Cascade_Theatre http://linkedgeodata.org/triplify/node1377239775 0.88 Theatre http://dbpedia.org/resource/Teatro_8 http://linkedgeodata.org/triplify/node2841321263 0.89 Theatre http://dbpedia.org/resource/Teatro_8 http://linkedgeodata.org/triplify/node1516384862 0.89 Theatre http://dbpedia.org/resource/Granada_Theatre_(Sherbrooke) http://linkedgeodata.org/triplify/node291563858 0.89 Theatre http://dbpedia.org/resource/Under_the_Gun_Theater http://linkedgeodata.org/triplify/node1649055310 0.89 Theatre http://dbpedia.org/resource/Under_the_Gun_Theater http://linkedgeodata.org/triplify/node108127128 0.89 Theatre http://dbpedia.org/resource/Miller_Outdoor_Theatre http://linkedgeodata.org/triplify/node3492970737 0.89 Theatre http://dbpedia.org/resource/Darwen_Library_Theatre http://linkedgeodata.org/triplify/node2622856469 0.89 Theatre http://dbpedia.org/resource/City_of_Chelmsford http://linkedgeodata.org/triplify/node21025675 0.9 City http://dbpedia.org/resource/City_of_Canterbury http://linkedgeodata.org/triplify/node17916174 0.9 City http://dbpedia.org/resource/Bar,_Ukraine http://linkedgeodata.org/triplify/node68528585 0.9 City http://dbpedia.org/resource/The_Paris_Theater_(Manhattan) http://linkedgeodata.org/triplify/node1268184551 0.9 Theatre http://dbpedia.org/resource/State_Theatre_(Minneapolis) http://linkedgeodata.org/triplify/node1283783642 0.9 Theatre http://dbpedia.org/resource/Manila_Grand_Opera_House http://linkedgeodata.org/triplify/node1415945880 0.9 Theatre http://dbpedia.org/resource/Lope_de_Vega_Theatre_(Madrid) http://linkedgeodata.org/triplify/node948769767 0.9 Theatre http://dbpedia.org/resource/Lope_de_Vega_Theatre_(Madrid) http://linkedgeodata.org/triplify/node2442964672 0.9 Theatre http://dbpedia.org/resource/Alley_Theatre http://linkedgeodata.org/triplify/node1528795258 0.91 Theatre http://dbpedia.org/resource/Long_Wharf_Theatre http://linkedgeodata.org/triplify/node1638965614 0.91 Theatre http://dbpedia.org/resource/Allen_Theater_(Allentown,_Pennsylvania) http://linkedgeodata.org/triplify/node1669486946 0.91 Theatre http://dbpedia.org/resource/McCarter_Theatre http://linkedgeodata.org/triplify/node3671630165 0.91 Theatre http://dbpedia.org/resource/Princess_Theatre_(Edmonton) http://linkedgeodata.org/triplify/node400017941 0.91 Theatre http://dbpedia.org/resource/La_Monnaie http://linkedgeodata.org/triplify/node1720357061 0.91 Theatre http://dbpedia.org/resource/Brno_City_Theatre http://linkedgeodata.org/triplify/node2318883803 0.91 Theatre http://dbpedia.org/resource/Strand_Theater_(Allentown,_Pennsylvania) http://linkedgeodata.org/triplify/node2474372156 0.91 Theatre http://dbpedia.org/resource/National_Theatre_(Washington,_D.C.) http://linkedgeodata.org/triplify/node1687584252 0.91 Theatre http://dbpedia.org/resource/Queens_Theatre_in_the_Park http://linkedgeodata.org/triplify/node319117504 0.92 Theatre http://dbpedia.org/resource/Garden_Theater http://linkedgeodata.org/triplify/node2488873131 0.94 Theatre http://dbpedia.org/resource/Orpheum_Theatre_(Omaha) http://linkedgeodata.org/triplify/node2728572820 0.95 Theatre http://dbpedia.org/resource/Teatro_8 http://linkedgeodata.org/triplify/node2101186992 0.95 Theatre http://dbpedia.org/resource/Teatro_8 http://linkedgeodata.org/triplify/node2194713687 0.95 Theatre http://dbpedia.org/resource/Teatro_8 http://linkedgeodata.org/triplify/node2383250710 0.95 Theatre http://dbpedia.org/resource/Teatro_8 http://linkedgeodata.org/triplify/node265581228 0.95 Theatre http://dbpedia.org/resource/Teatro_8 http://linkedgeodata.org/triplify/node3198400474 0.95 Theatre http://dbpedia.org/resource/Olympia_Theatre_(New_York_City) http://linkedgeodata.org/triplify/node1296487138 0.95 Theatre http://dbpedia.org/resource/Capitol_Theatre,_Perth http://linkedgeodata.org/triplify/node327614614 0.95 Theatre http://dbpedia.org/resource/Polk_Theatre_(Lakeland,_Florida) http://linkedgeodata.org/triplify/node394950089 0.97 Theatre http://dbpedia.org/resource/Grand_Theatre,_Leeds http://linkedgeodata.org/triplify/node26301254 0.98 Theatre http://dbpedia.org/resource/Europe http://linkedgeodata.org/triplify/node25871341 1 Continent http://dbpedia.org/resource/North_America http://linkedgeodata.org/triplify/node36966063 1 Continent http://dbpedia.org/resource/Oceania http://linkedgeodata.org/triplify/node249399679 1 Continent http://dbpedia.org/resource/Africa http://linkedgeodata.org/triplify/node36966057 1 Continent http://dbpedia.org/resource/Palestinian_National_Authority http://linkedgeodata.org/triplify/node2110225115 1 Country http://dbpedia.org/resource/Weimar_Republic http://linkedgeodata.org/triplify/node1683325355 1 Country http://dbpedia.org/resource/Fifth_Republic_of_South_Korea http://linkedgeodata.org/triplify/node249399362 1 Country http://dbpedia.org/resource/Principality_of_Serbia http://linkedgeodata.org/triplify/node369137525 1 Country http://dbpedia.org/resource/Saint_Kitts_and_Nevis http://linkedgeodata.org/triplify/node424310758 1 Country http://dbpedia.org/resource/County_of_Luxemburg http://linkedgeodata.org/triplify/node424298019 1 Country http://dbpedia.org/resource/Tonga http://linkedgeodata.org/triplify/node424311848 1 Country http://dbpedia.org/resource/Qatar http://linkedgeodata.org/triplify/node432425062 1 Country http://dbpedia.org/resource/Malawi http://linkedgeodata.org/triplify/node424311645 1 Country http://dbpedia.org/resource/Angola http://linkedgeodata.org/triplify/node424310875 1 Country http://dbpedia.org/resource/Laos http://linkedgeodata.org/triplify/node424316307 1 Country http://dbpedia.org/resource/Supreme_Council_for_National_Reconstruction http://linkedgeodata.org/triplify/node249399362 1 Country http://dbpedia.org/resource/Russian_Empire http://linkedgeodata.org/triplify/node424314830 1 Country http://dbpedia.org/resource/Allied-occupied_Germany http://linkedgeodata.org/triplify/node1683325355 1 Country http://dbpedia.org/resource/Guatemala http://linkedgeodata.org/triplify/node1644732479 1 Country http://dbpedia.org/resource/Socialist_Republic_of_Croatia http://linkedgeodata.org/triplify/node424311353 1 Country http://dbpedia.org/resource/Azerbaijan http://linkedgeodata.org/triplify/node424313675 1 Country http://dbpedia.org/resource/First_Republic_of_South_Korea http://linkedgeodata.org/triplify/node249399362 1 Country http://dbpedia.org/resource/Monarchy_of_the_North http://linkedgeodata.org/triplify/node2377028247 1 Country http://dbpedia.org/resource/Governorate_General_of_Brazil http://linkedgeodata.org/triplify/node424313734 1 Country http://dbpedia.org/resource/Niger http://linkedgeodata.org/triplify/node424311675 1 Country http://dbpedia.org/resource/Togo http://linkedgeodata.org/triplify/node424311827 1 Country http://dbpedia.org/resource/Egypt http://linkedgeodata.org/triplify/node424316018 1 Country http://dbpedia.org/resource/China http://linkedgeodata.org/triplify/node424313582 1 Country http://dbpedia.org/resource/Kingdom_of_Granada_(Crown_of_Castile) http://linkedgeodata.org/triplify/node424316074 1 Country http://dbpedia.org/resource/Sahrawi_Arab_Democratic_Republic http://linkedgeodata.org/triplify/node2621901768 1 Country http://dbpedia.org/resource/Colombia http://linkedgeodata.org/triplify/node424314631 1 Country http://dbpedia.org/resource/Kingdom_of_Fiji http://linkedgeodata.org/triplify/node424313823 1 Country http://dbpedia.org/resource/Empire_of_Vietnam http://linkedgeodata.org/triplify/node424298150 1 Country http://dbpedia.org/resource/Benin http://linkedgeodata.org/triplify/node424297384 1 Country http://dbpedia.org/resource/Nicaragua http://linkedgeodata.org/triplify/node424316671 1 Country http://dbpedia.org/resource/Kingdom_of_Egypt http://linkedgeodata.org/triplify/node424316018 1 Country http://dbpedia.org/resource/Sierra_Leone http://linkedgeodata.org/triplify/node424314123 1 Country http://dbpedia.org/resource/Grenada http://linkedgeodata.org/triplify/node424316074 1 Country http://dbpedia.org/resource/Odoacer__Regnum_Italicum__1 http://linkedgeodata.org/triplify/node26347360 1 Country http://dbpedia.org/resource/Oman http://linkedgeodata.org/triplify/node424315291 1 Country http://dbpedia.org/resource/Tajikistan http://linkedgeodata.org/triplify/node424314276 1 Country http://dbpedia.org/resource/Mauritius http://linkedgeodata.org/triplify/node1708980402 1 Country http://dbpedia.org/resource/Lagos_Colony http://linkedgeodata.org/triplify/node424316307 1 Country http://dbpedia.org/resource/France http://linkedgeodata.org/triplify/node1363947712 1 Country http://dbpedia.org/resource/Malagasy_Republic http://linkedgeodata.org/triplify/node432425028 1 Country http://dbpedia.org/resource/Belarus http://linkedgeodata.org/triplify/node249399300 1 Country http://dbpedia.org/resource/Pitcairn_Islands http://linkedgeodata.org/triplify/node1388794217 1 Country http://dbpedia.org/resource/First_Mexican_Republic http://linkedgeodata.org/triplify/node432425036 1 Country http://dbpedia.org/resource/Marshall_Islands http://linkedgeodata.org/triplify/node424311955 1 Country http://dbpedia.org/resource/French_Third_Republic http://linkedgeodata.org/triplify/node1363947712 1 Country http://dbpedia.org/resource/The_Gambia http://linkedgeodata.org/triplify/node424297579 1 Country http://dbpedia.org/resource/Portuguese_Mozambique http://linkedgeodata.org/triplify/node424311658 1 Country http://dbpedia.org/resource/Burma http://linkedgeodata.org/triplify/node424316478 1 Country http://dbpedia.org/resource/Chile http://linkedgeodata.org/triplify/node424311104 1 Country http://dbpedia.org/resource/Slovakia http://linkedgeodata.org/triplify/node432425079 1 Country http://dbpedia.org/resource/Lithuania http://linkedgeodata.org/triplify/node424297773 1 Country http://dbpedia.org/resource/Second_French_Empire http://linkedgeodata.org/triplify/node1363947712 1 Country http://dbpedia.org/resource/Bermuda http://linkedgeodata.org/triplify/node433483519 1 Country http://dbpedia.org/resource/Nepal http://linkedgeodata.org/triplify/node249399402 1 Country http://dbpedia.org/resource/Malavi http://linkedgeodata.org/triplify/node424311645 1 Country http://dbpedia.org/resource/Kyrgyzstan http://linkedgeodata.org/triplify/node1700777271 1 Country http://dbpedia.org/resource/Comoros http://linkedgeodata.org/triplify/node249399360 1 Country http://dbpedia.org/resource/Sint_Maarten http://linkedgeodata.org/triplify/node271048940 1 Country http://dbpedia.org/resource/United_Kingdom_of_Great_Britain_and_Ireland http://linkedgeodata.org/triplify/node838090640 1 Country http://dbpedia.org/resource/Interim_Government_of_Iran http://linkedgeodata.org/triplify/node424298311 1 Country http://dbpedia.org/resource/British_Indian_Ocean_Territory http://linkedgeodata.org/triplify/node424313891 1 Country http://dbpedia.org/resource/Liberia http://linkedgeodata.org/triplify/node424311573 1 Country http://dbpedia.org/resource/Afghanistan http://linkedgeodata.org/triplify/node26847706 1 Country http://dbpedia.org/resource/Equatorial_Guinea http://linkedgeodata.org/triplify/node424313853 1 Country http://dbpedia.org/resource/East_Timor http://linkedgeodata.org/triplify/node432424972 1 Country http://dbpedia.org/resource/United_Nations_Transitional_Authority_in_Cambodia http://linkedgeodata.org/triplify/node432424952 1 Country http://dbpedia.org/resource/Socialist_Republic_of_Bosnia_and_Herzegovina http://linkedgeodata.org/triplify/node1983711862 1 Country http://dbpedia.org/resource/Colonial_Mauritania http://linkedgeodata.org/triplify/node424316493 1 Country http://dbpedia.org/resource/Captaincy_General_of_Cuba http://linkedgeodata.org/triplify/node249399313 1 Country http://dbpedia.org/resource/Ethiopia http://linkedgeodata.org/triplify/node2163914205 1 Country http://dbpedia.org/resource/German_Empire http://linkedgeodata.org/triplify/node1683325355 1 Country http://dbpedia.org/resource/Venezuela http://linkedgeodata.org/triplify/node424311890 1 Country http://dbpedia.org/resource/Greece http://linkedgeodata.org/triplify/node432424989 1 Country http://dbpedia.org/resource/Slovenia http://linkedgeodata.org/triplify/node208592566 1 Country http://dbpedia.org/resource/Spain http://linkedgeodata.org/triplify/node148332300 1 Country http://dbpedia.org/resource/Haiti http://linkedgeodata.org/triplify/node424297281 1 Country http://dbpedia.org/resource/Socialist_Republic_of_Slovenia http://linkedgeodata.org/triplify/node208592566 1 Country http://dbpedia.org/resource/Second_Bulgarian_Empire http://linkedgeodata.org/triplify/node424315709 1 Country http://dbpedia.org/resource/Duchy_of_Luxembourg http://linkedgeodata.org/triplify/node424298019 1 Country http://dbpedia.org/resource/Italian_Somaliland http://linkedgeodata.org/triplify/node424317039 1 Country http://dbpedia.org/resource/Cuba http://linkedgeodata.org/triplify/node249399313 1 Country http://dbpedia.org/resource/Luxembourg http://linkedgeodata.org/triplify/node424298019 1 Country http://dbpedia.org/resource/State_of_Palestine http://linkedgeodata.org/triplify/node2110225115 1 Country http://dbpedia.org/resource/Kingdom_of_Lithuania http://linkedgeodata.org/triplify/node424297773 1 Country http://dbpedia.org/resource/Vanuatu http://linkedgeodata.org/triplify/node424313466 1 Country http://dbpedia.org/resource/Paraguayan_architecture http://linkedgeodata.org/triplify/node1644764586 1 Country http://dbpedia.org/resource/Norway_in_1814 http://linkedgeodata.org/triplify/node424297198 1 Country http://dbpedia.org/resource/Emirate_of_Granada http://linkedgeodata.org/triplify/node424316074 1 Country http://dbpedia.org/resource/Second_Republic_of_South_Korea http://linkedgeodata.org/triplify/node249399362 1 Country http://dbpedia.org/resource/Cricket_Samoa http://linkedgeodata.org/triplify/node1529444079 1 Country http://dbpedia.org/resource/Commemorative_coins_of_Bulgaria http://linkedgeodata.org/triplify/node424315709 1 Country http://dbpedia.org/resource/Empire_of_Brazil http://linkedgeodata.org/triplify/node424313734 1 Country http://dbpedia.org/resource/General_Governorate_of_Belgium http://linkedgeodata.org/triplify/node1684793666 1 Country http://dbpedia.org/resource/Bhutan http://linkedgeodata.org/triplify/node424311028 1 Country http://dbpedia.org/resource/Interim_Government_of_India http://linkedgeodata.org/triplify/node424314145 1 Country http://dbpedia.org/resource/Transnistria http://linkedgeodata.org/triplify/node2043451343 1 Country http://dbpedia.org/resource/Provisional_Administrative_Entity_of_South_Ossetia http://linkedgeodata.org/triplify/node1696850886 1 Country http://dbpedia.org/resource/Uruguay http://linkedgeodata.org/triplify/node432425116 1 Country http://dbpedia.org/resource/Nagorno-Karabakh http://linkedgeodata.org/triplify/node1696853468 1 Country http://dbpedia.org/resource/Nagorno-Karabakh_Republic http://linkedgeodata.org/triplify/node1696853468 1 Country http://dbpedia.org/resource/French_Madagascar http://linkedgeodata.org/triplify/node432425028 1 Country http://dbpedia.org/resource/Brunei http://linkedgeodata.org/triplify/node1694010854 1 Country http://dbpedia.org/resource/Swedish_Empire http://linkedgeodata.org/triplify/node424313277 1 Country http://dbpedia.org/resource/Kingdom_of_Hungary http://linkedgeodata.org/triplify/node249399347 1 Country http://dbpedia.org/resource/Poland http://linkedgeodata.org/triplify/node432425060 1 Country http://dbpedia.org/resource/List_of_Zambians http://linkedgeodata.org/triplify/node432425123 1 Country http://dbpedia.org/resource/Sudan http://linkedgeodata.org/triplify/node424297973 1 Country http://dbpedia.org/resource/Democratic_Republic_of_Afghanistan http://linkedgeodata.org/triplify/node26847706 1 Country http://dbpedia.org/resource/Guinea http://linkedgeodata.org/triplify/node424297586 1 Country http://dbpedia.org/resource/Bosnia_and_Herzegovina http://linkedgeodata.org/triplify/node1983711862 1 Country http://dbpedia.org/resource/Francia http://linkedgeodata.org/triplify/node1363947712 1 Country http://dbpedia.org/resource/Nigeria http://linkedgeodata.org/triplify/node424316663 1 Country http://dbpedia.org/resource/Nasrid_dynasty http://linkedgeodata.org/triplify/node424316074 1 Country http://dbpedia.org/resource/Uzbek_Soviet_Socialist_Republic http://linkedgeodata.org/triplify/node424317428 1 Country http://dbpedia.org/resource/Papua_New_Guinea http://linkedgeodata.org/triplify/node424297931 1 Country http://dbpedia.org/resource/Central_African_Republic http://linkedgeodata.org/triplify/node432424956 1 Country http://dbpedia.org/resource/Federated_States_of_Micronesia http://linkedgeodata.org/triplify/node424311244 1 Country http://dbpedia.org/resource/Suriname http://linkedgeodata.org/triplify/node432425087 1 Country http://dbpedia.org/resource/Kingdom_of_Bulgaria http://linkedgeodata.org/triplify/node424315709 1 Country http://dbpedia.org/resource/Centralist_Republic_of_Mexico http://linkedgeodata.org/triplify/node432425036 1 Country http://dbpedia.org/resource/Grand_Duchy_of_Finland http://linkedgeodata.org/triplify/node432424981 1 Country http://dbpedia.org/resource/Mauritania http://linkedgeodata.org/triplify/node424316493 1 Country http://dbpedia.org/resource/Colony_of_Niger http://linkedgeodata.org/triplify/node424311675 1 Country http://dbpedia.org/resource/Thailand http://linkedgeodata.org/triplify/node437146363 1 Country http://dbpedia.org/resource/Brazilian_military_government http://linkedgeodata.org/triplify/node424313734 1 Country http://dbpedia.org/resource/Lesotho http://linkedgeodata.org/triplify/node3491215388 1 Country http://dbpedia.org/resource/Kingdom_of_Iceland http://linkedgeodata.org/triplify/node424314784 1 Country http://dbpedia.org/resource/United_Kingdom_of_Portugal,_Brazil_and_the_Algarves http://linkedgeodata.org/triplify/node2377028247 1 Country http://dbpedia.org/resource/Nazi_Germany http://linkedgeodata.org/triplify/node1683325355 1 Country http://dbpedia.org/resource/Hungary http://linkedgeodata.org/triplify/node249399347 1 Country http://dbpedia.org/resource/Japan http://linkedgeodata.org/triplify/node424313451 1 Country http://dbpedia.org/resource/Cayman_Islands http://linkedgeodata.org/triplify/node249399364 1 Country http://dbpedia.org/resource/Jersey http://linkedgeodata.org/triplify/node433483540 1 Country http://dbpedia.org/resource/Seychelles http://linkedgeodata.org/triplify/node424298277 1 Country http://dbpedia.org/resource/Vietnam http://linkedgeodata.org/triplify/node424298150 1 Country http://dbpedia.org/resource/Maldives http://linkedgeodata.org/triplify/node424311641 1 Country http://dbpedia.org/resource/Republic_of_Bosnia_and_Herzegovina http://linkedgeodata.org/triplify/node1983711862 1 Country http://dbpedia.org/resource/North_Korea http://linkedgeodata.org/triplify/node249399361 1 Country http://dbpedia.org/resource/San_Marino http://linkedgeodata.org/triplify/node424317025 1 Country http://dbpedia.org/resource/Viceroyalty_of_New_Granada http://linkedgeodata.org/triplify/node424316074 1 Country http://dbpedia.org/resource/Italian_Social_Republic http://linkedgeodata.org/triplify/node26347360 1 Country http://dbpedia.org/resource/Independent_Albania http://linkedgeodata.org/triplify/node424310601 1 Country http://dbpedia.org/resource/Islamic_Emirate_of_Afghanistan http://linkedgeodata.org/triplify/node26847706 1 Country http://dbpedia.org/resource/Socialist_Soviet_Republic_of_Abkhazia http://linkedgeodata.org/triplify/node466507355 1 Country http://dbpedia.org/resource/United_States_Protectorate_over_Cuba http://linkedgeodata.org/triplify/node249399313 1 Country http://dbpedia.org/resource/Czech_Republic http://linkedgeodata.org/triplify/node424313803 1 Country http://dbpedia.org/resource/Zambia http://linkedgeodata.org/triplify/node432425123 1 Country http://dbpedia.org/resource/The_Bahamas http://linkedgeodata.org/triplify/node249399296 1 Country http://dbpedia.org/resource/Lebanon http://linkedgeodata.org/triplify/node424297742 1 Country http://dbpedia.org/resource/Denmark http://linkedgeodata.org/triplify/node432424968 1 Country http://dbpedia.org/resource/Ivory_Coast http://linkedgeodata.org/triplify/node424314691 1 Country http://dbpedia.org/resource/Saudi_Arabia http://linkedgeodata.org/triplify/node249399419 1 Country http://dbpedia.org/resource/Germany http://linkedgeodata.org/triplify/node1683325355 1 Country http://dbpedia.org/resource/Saint_Vincent_and_the_Grenadines http://linkedgeodata.org/triplify/node432425070 1 Country http://dbpedia.org/resource/Hungarian_Soviet_Republic http://linkedgeodata.org/triplify/node249399347 1 Country http://dbpedia.org/resource/Paraguay http://linkedgeodata.org/triplify/node1644764586 1 Country http://dbpedia.org/resource/Eritrea http://linkedgeodata.org/triplify/node424316022 1 Country http://dbpedia.org/resource/Norfolk_Island http://linkedgeodata.org/triplify/node424311679 1 Country http://dbpedia.org/resource/Euro_gold_and_silver_commemorative_coins http://linkedgeodata.org/triplify/node208592566 1 Country http://dbpedia.org/resource/Euro_gold_and_silver_commemorative_coins http://linkedgeodata.org/triplify/node1684793666 1 Country http://dbpedia.org/resource/Euro_gold_and_silver_commemorative_coins http://linkedgeodata.org/triplify/node424311883 1 Country http://dbpedia.org/resource/Euro_gold_and_silver_commemorative_coins http://linkedgeodata.org/triplify/node432424981 1 Country http://dbpedia.org/resource/Euro_gold_and_silver_commemorative_coins http://linkedgeodata.org/triplify/node424313406 1 Country http://dbpedia.org/resource/Switzerland http://linkedgeodata.org/triplify/node1504546320 1 Country http://dbpedia.org/resource/Government_of_the_Autonomous_Republic_of_Abkhaziahttp://linkedgeodata.org/triplify/node466507355 1 Country http://dbpedia.org/resource/Bulgaria http://linkedgeodata.org/triplify/node424315709 1 Country http://dbpedia.org/resource/Swaziland http://linkedgeodata.org/triplify/node424310680 1 Country http://dbpedia.org/resource/Bahrain http://linkedgeodata.org/triplify/node424297207 1 Country http://dbpedia.org/resource/Lithuanian_Soviet_Socialist_Republic http://linkedgeodata.org/triplify/node424297773 1 Country http://dbpedia.org/resource/Madagascar http://linkedgeodata.org/triplify/node432425028 1 Country http://dbpedia.org/resource/Mauritius_(Commonwealth_realm) http://linkedgeodata.org/triplify/node1708980402 1 Country http://dbpedia.org/resource/Croatia_in_the_union_with_Hungary http://linkedgeodata.org/triplify/node424311353 1 Country http://dbpedia.org/resource/British_Solomon_Islands http://linkedgeodata.org/triplify/node823750101 1 Country http://dbpedia.org/resource/Kingdom_of_Croatia_(Habsburg) http://linkedgeodata.org/triplify/node424311353 1 Country http://dbpedia.org/resource/British_Mauritius http://linkedgeodata.org/triplify/node1708980402 1 Country http://dbpedia.org/resource/Colony_of_Jamaica http://linkedgeodata.org/triplify/node424313591 1 Country http://dbpedia.org/resource/Ba'athist_Iraq http://linkedgeodata.org/triplify/node424316205 1 Country http://dbpedia.org/resource/El_Salvador http://linkedgeodata.org/triplify/node424311802 1 Country http://dbpedia.org/resource/Republic_of_the_Congo http://linkedgeodata.org/triplify/node432425063 1 Country http://dbpedia.org/resource/Kuwait http://linkedgeodata.org/triplify/node424313936 1 Country http://dbpedia.org/resource/Kingdom_of_the_Netherlands http://linkedgeodata.org/triplify/node424297217 1 Country http://dbpedia.org/resource/Finland http://linkedgeodata.org/triplify/node432424981 1 Country http://dbpedia.org/resource/United_Kingdom http://linkedgeodata.org/triplify/node838090640 1 Country http://dbpedia.org/resource/Dominica http://linkedgeodata.org/triplify/node424297496 1 Country http://dbpedia.org/resource/Isle_of_Man http://linkedgeodata.org/triplify/node1602889766 1 Country http://dbpedia.org/resource/Greenland http://linkedgeodata.org/triplify/node424312026 1 Country http://dbpedia.org/resource/Falkland_Islands http://linkedgeodata.org/triplify/node424311237 1 Country http://dbpedia.org/resource/State_of_Bahrain http://linkedgeodata.org/triplify/node424297207 1 Country http://dbpedia.org/resource/Guam http://linkedgeodata.org/triplify/node607976629 1 Country http://dbpedia.org/resource/Romania http://linkedgeodata.org/triplify/node432425064 1 Country http://dbpedia.org/resource/Sovereign_Principality_of_the_United_Netherlands http://linkedgeodata.org/triplify/node424297217 1 Country http://dbpedia.org/resource/Kingdom_of_Israel_(Samaria) http://linkedgeodata.org/triplify/node424297647 1 Country http://dbpedia.org/resource/Moldova http://linkedgeodata.org/triplify/node424316417 1 Country http://dbpedia.org/resource/Senegal http://linkedgeodata.org/triplify/node432425075 1 Country http://dbpedia.org/resource/Belize http://linkedgeodata.org/triplify/node332779764 1 Country http://dbpedia.org/resource/Swedish_overseas_colonies http://linkedgeodata.org/triplify/node424313277 1 Country http://dbpedia.org/resource/Peru http://linkedgeodata.org/triplify/node249399406 1 Country http://dbpedia.org/resource/Granada http://linkedgeodata.org/triplify/node424316074 1 Country http://dbpedia.org/resource/South_Ossetia http://linkedgeodata.org/triplify/node1696850886 1 Country http://dbpedia.org/resource/Algeria http://linkedgeodata.org/triplify/node432424926 1 Country http://dbpedia.org/resource/Vatican_City http://linkedgeodata.org/triplify/node424311883 1 Country http://dbpedia.org/resource/Russian_Soviet_Federative_Socialist_Republic http://linkedgeodata.org/triplify/node424314830 1 Country http://dbpedia.org/resource/Burkina_Faso http://linkedgeodata.org/triplify/node249399288 1 Country http://dbpedia.org/resource/South_Georgia_and_the_South_Sandwich_Islands http://linkedgeodata.org/triplify/node249399339 1 Country http://dbpedia.org/resource/Abkhazia http://linkedgeodata.org/triplify/node466507355 1 Country http://dbpedia.org/resource/Malaysia http://linkedgeodata.org/triplify/node249399394 1 Country http://dbpedia.org/resource/Malaysia http://linkedgeodata.org/triplify/node2134362766 1 Country http://dbpedia.org/resource/Estonia http://linkedgeodata.org/triplify/node424313832 1 Country http://dbpedia.org/resource/Occupation_of_Japan http://linkedgeodata.org/triplify/node424313451 1 Country http://dbpedia.org/resource/Captaincy_General_of_Guatemala http://linkedgeodata.org/triplify/node1644732479 1 Country http://dbpedia.org/resource/Barbados http://linkedgeodata.org/triplify/node249399285 1 Country http://dbpedia.org/resource/Kazakhstan http://linkedgeodata.org/triplify/node424311521 1 Country http://dbpedia.org/resource/Oman_proper http://linkedgeodata.org/triplify/node424315291 1 Country http://dbpedia.org/resource/Serbia http://linkedgeodata.org/triplify/node369137525 1 Country http://dbpedia.org/resource/Kingdom_of_Serbia http://linkedgeodata.org/triplify/node369137525 1 Country http://dbpedia.org/resource/Cook_Islands http://linkedgeodata.org/triplify/node249399308 1 Country http://dbpedia.org/resource/Second_Federal_Republic_of_Mexico http://linkedgeodata.org/triplify/node432425036 1 Country http://dbpedia.org/resource/India http://linkedgeodata.org/triplify/node424314145 1 Country http://dbpedia.org/resource/Jordan http://linkedgeodata.org/triplify/node424316216 1 Country http://dbpedia.org/resource/Singapore http://linkedgeodata.org/triplify/node424313428 1 Country http://dbpedia.org/resource/Yemen http://linkedgeodata.org/triplify/node249399459 1 Country http://dbpedia.org/resource/Transitional_Federal_Government http://linkedgeodata.org/triplify/node424317039 1 Country http://dbpedia.org/resource/Irish_Republic http://linkedgeodata.org/triplify/node1420871007 1 Country http://dbpedia.org/resource/Bagratid_Armenia http://linkedgeodata.org/triplify/node249399277 1 Country http://dbpedia.org/resource/Republic_of_Estonia_(1918-1940) http://linkedgeodata.org/triplify/node424313832 1 Country http://dbpedia.org/resource/Third_Republic_of_South_Korea http://linkedgeodata.org/triplify/node249399362 1 Country http://dbpedia.org/resource/Uzbekistan http://linkedgeodata.org/triplify/node424317428 1 Country http://dbpedia.org/resource/Russia http://linkedgeodata.org/triplify/node424314830 1 Country http://dbpedia.org/resource/United_Arab_Emirates http://linkedgeodata.org/triplify/node838090640 1 Country http://dbpedia.org/resource/United_Arab_Emirates http://linkedgeodata.org/triplify/node424310703 1 Country http://dbpedia.org/resource/Pridnestrovian_Moldavian_Soviet_Socialist_Republic http://linkedgeodata.org/triplify/node2043451343 1 Country http://dbpedia.org/resource/Kingdom_of_Burundi http://linkedgeodata.org/triplify/node424310982 1 Country http://dbpedia.org/resource/British_Raj http://linkedgeodata.org/triplify/node424314145 1 Country http://dbpedia.org/resource/Tanzania http://linkedgeodata.org/triplify/node424314335 1 Country http://dbpedia.org/resource/Tuvalu http://linkedgeodata.org/triplify/node424317372 1 Country http://dbpedia.org/resource/July_Monarchy http://linkedgeodata.org/triplify/node1363947712 1 Country http://dbpedia.org/resource/Guyana http://linkedgeodata.org/triplify/node424311328 1 Country http://dbpedia.org/resource/American_Samoa http://linkedgeodata.org/triplify/node506535828 1 Country http://dbpedia.org/resource/Anguilla http://linkedgeodata.org/triplify/node433483517 1 Country http://dbpedia.org/resource/French_Chad http://linkedgeodata.org/triplify/node424298076 1 Country http://dbpedia.org/resource/Antigua_and_Barbuda http://linkedgeodata.org/triplify/node432424929 1 Country http://dbpedia.org/resource/Armenia http://linkedgeodata.org/triplify/node249399277 1 Country http://dbpedia.org/resource/Montenegro http://linkedgeodata.org/triplify/node445970763 1 Country http://dbpedia.org/resource/Grand_Duchy_of_Lithuania http://linkedgeodata.org/triplify/node424297773 1 Country http://dbpedia.org/resource/Socialist_Republic_of_Montenegro http://linkedgeodata.org/triplify/node445970763 1 Country http://dbpedia.org/resource/Mexico http://linkedgeodata.org/triplify/node432425036 1 Country http://dbpedia.org/resource/Austrian_Empire http://linkedgeodata.org/triplify/node26847709 1 Country http://dbpedia.org/resource/Latvia http://linkedgeodata.org/triplify/node432425019 1 Country http://dbpedia.org/resource/Brazil http://linkedgeodata.org/triplify/node424313734 1 Country http://dbpedia.org/resource/Egyptian_First_Republic http://linkedgeodata.org/triplify/node424316018 1 Country http://dbpedia.org/resource/United_States_of_Belgium http://linkedgeodata.org/triplify/node1684793666 1 Country http://dbpedia.org/resource/Austria http://linkedgeodata.org/triplify/node26847709 1 Country http://dbpedia.org/resource/Cambodia http://linkedgeodata.org/triplify/node432424952 1 Country http://dbpedia.org/resource/Democratic_Republic_of_the_Congo http://linkedgeodata.org/triplify/node435981993 1 Country http://dbpedia.org/resource/Ecuador http://linkedgeodata.org/triplify/node424298227 1 Country http://dbpedia.org/resource/South_Africa http://linkedgeodata.org/triplify/node424298167 1 Country http://dbpedia.org/resource/Tokelau http://linkedgeodata.org/triplify/node433483572 1 Country http://dbpedia.org/resource/British_Virgin_Islands http://linkedgeodata.org/triplify/node424311891 1 Country http://dbpedia.org/resource/Iceland http://linkedgeodata.org/triplify/node424314784 1 Country http://dbpedia.org/resource/Post-war_Singapore http://linkedgeodata.org/triplify/node424313428 1 Country http://dbpedia.org/resource/Honduras http://linkedgeodata.org/triplify/node424297629 1 Country http://dbpedia.org/resource/Sweden http://linkedgeodata.org/triplify/node424313277 1 Country http://dbpedia.org/resource/Bangladesh http://linkedgeodata.org/triplify/node432424937 1 Country http://dbpedia.org/resource/Kiribati http://linkedgeodata.org/triplify/node432425015 1 Country http://dbpedia.org/resource/Fourth_Republic_of_South_Korea http://linkedgeodata.org/triplify/node249399362 1 Country http://dbpedia.org/resource/Portugal http://linkedgeodata.org/triplify/node2377028247 1 Country http://dbpedia.org/resource/Andorra http://linkedgeodata.org/triplify/node148874198 1 Country http://dbpedia.org/resource/Saint_Lucia http://linkedgeodata.org/triplify/node424316319 1 Country http://dbpedia.org/resource/Russian_Democratic_Federative_Republic http://linkedgeodata.org/triplify/node424314830 1 Country http://dbpedia.org/resource/Dominion_of_Fiji http://linkedgeodata.org/triplify/node424313823 1 Country http://dbpedia.org/resource/Commemorative_coins_of_Romania http://linkedgeodata.org/triplify/node432425064 1 Country http://dbpedia.org/resource/Dutch_government-in-exile http://linkedgeodata.org/triplify/node424297217 1 Country http://dbpedia.org/resource/Principality_of_Sealand http://linkedgeodata.org/triplify/way363076217 1 Country http://dbpedia.org/resource/Montserrat http://linkedgeodata.org/triplify/node424314005 1 Country http://dbpedia.org/resource/Kingdom_of_Afghanistan http://linkedgeodata.org/triplify/node26847706 1 Country http://dbpedia.org/resource/Solomon_Islands http://linkedgeodata.org/triplify/node823750101 1 Country http://dbpedia.org/resource/Bolivia http://linkedgeodata.org/triplify/node424313730 1 Country http://dbpedia.org/resource/Hungarian_Democratic_Republic http://linkedgeodata.org/triplify/node249399347 1 Country http://dbpedia.org/resource/Nauru http://linkedgeodata.org/triplify/node424297917 1 Country http://dbpedia.org/resource/Palestine http://linkedgeodata.org/triplify/node2110225115 1 Country http://dbpedia.org/resource/Tunisia http://linkedgeodata.org/triplify/node424317315 1 Country http://dbpedia.org/resource/History_of_Mahdist_Sudan http://linkedgeodata.org/triplify/node424297973 1 Country http://dbpedia.org/resource/Philippines http://linkedgeodata.org/triplify/node26036352 1 Country http://dbpedia.org/resource/Saint_Helena,_Ascension_and_Tristan_da_Cunha http://linkedgeodata.org/triplify/node1602894836 1 Country http://dbpedia.org/resource/Zimbabwe http://linkedgeodata.org/triplify/node1640880787 1 Country http://dbpedia.org/resource/Emirate_of_Afghanistan http://linkedgeodata.org/triplify/node26847706 1 Country http://dbpedia.org/resource/4th_of_August_Regime http://linkedgeodata.org/triplify/node432424989 1 Country http://dbpedia.org/resource/Australia http://linkedgeodata.org/triplify/node424315584 1 Country http://dbpedia.org/resource/Australia http://linkedgeodata.org/triplify/node26847717 1 Country http://dbpedia.org/resource/Sheikhdom_of_Kuwait http://linkedgeodata.org/triplify/node424313936 1 Country http://dbpedia.org/resource/Andora http://linkedgeodata.org/triplify/node148874198 1 Country http://dbpedia.org/resource/Niue http://linkedgeodata.org/triplify/node433483552 1 Country http://dbpedia.org/resource/Israel http://linkedgeodata.org/triplify/node424297647 1 Country http://dbpedia.org/resource/Fiji http://linkedgeodata.org/triplify/node424313823 1 Country http://dbpedia.org/resource/Mozambique http://linkedgeodata.org/triplify/node424311658 1 Country http://dbpedia.org/resource/Guernsey http://linkedgeodata.org/triplify/node1607605918 1 Country http://dbpedia.org/resource/Republic_of_Ireland http://linkedgeodata.org/triplify/node1420871007 1 Country http://dbpedia.org/resource/Turks_and_Caicos_Islands http://linkedgeodata.org/triplify/node424311814 1 Country http://dbpedia.org/resource/Kingdom_of_Croatia-Slavonia http://linkedgeodata.org/triplify/node424311353 1 Country http://dbpedia.org/resource/Second_Occupation_of_Cuba http://linkedgeodata.org/triplify/node249399313 1 Country http://dbpedia.org/resource/Turkmenistan http://linkedgeodata.org/triplify/node249399440 1 Country http://dbpedia.org/resource/Jamaica http://linkedgeodata.org/triplify/node424313591 1 Country http://dbpedia.org/resource/Kenya http://linkedgeodata.org/triplify/node424313903 1 Country http://dbpedia.org/resource/Panama http://linkedgeodata.org/triplify/node424316758 1 Country http://dbpedia.org/resource/State_of_Burma http://linkedgeodata.org/triplify/node424316478 1 Country http://dbpedia.org/resource/Mali http://linkedgeodata.org/triplify/node424297816 1 Country http://dbpedia.org/resource/Guinea-Bissau http://linkedgeodata.org/triplify/node424297609 1 Country http://dbpedia.org/resource/South_Sudan http://linkedgeodata.org/triplify/node1356260631 1 Country http://dbpedia.org/resource/Mongolia http://linkedgeodata.org/triplify/node432425040 1 Country http://dbpedia.org/resource/Republic_of_Afghanistan http://linkedgeodata.org/triplify/node26847706 1 Country http://dbpedia.org/resource/Gabon http://linkedgeodata.org/triplify/node424311252 1 Country http://dbpedia.org/resource/Dominion_of_Pakistan http://linkedgeodata.org/triplify/node424314088 1 Country http://dbpedia.org/resource/Dominican_Republic http://linkedgeodata.org/triplify/node246422158 1 Country http://dbpedia.org/resource/Principality_of_Abkhazia http://linkedgeodata.org/triplify/node466507355 1 Country http://dbpedia.org/resource/Burundi http://linkedgeodata.org/triplify/node424310982 1 Country http://dbpedia.org/resource/New_Zealand http://linkedgeodata.org/triplify/node248120384 1 Country http://dbpedia.org/resource/Transitional_National_Government http://linkedgeodata.org/triplify/node424317039 1 Country http://dbpedia.org/resource/Italy http://linkedgeodata.org/triplify/node26347360 1 Country http://dbpedia.org/resource/Islamic_State_of_Afghanistan http://linkedgeodata.org/triplify/node26847706 1 Country http://dbpedia.org/resource/Moldavia http://linkedgeodata.org/triplify/node424316417 1 Country http://dbpedia.org/resource/Liechtenstein http://linkedgeodata.org/triplify/node3696525426 1 Country http://dbpedia.org/resource/Transitional_Government_of_Ethiopia http://linkedgeodata.org/triplify/node2163914205 1 Country http://dbpedia.org/resource/Libya http://linkedgeodata.org/triplify/node432425023 1 Country http://dbpedia.org/resource/Netherlands http://linkedgeodata.org/triplify/node424297217 1 Country http://dbpedia.org/resource/Turkey http://linkedgeodata.org/triplify/node424298326 1 Country http://dbpedia.org/resource/Third_Republic_of_Madagascar http://linkedgeodata.org/triplify/node432425028 1 Country http://dbpedia.org/resource/Dominion_of_India http://linkedgeodata.org/triplify/node424314145 1 Country http://dbpedia.org/resource/Occupation_of_the_Falkland_Islands http://linkedgeodata.org/triplify/node424311237 1 Country http://dbpedia.org/resource/Botswana http://linkedgeodata.org/triplify/node424313751 1 Country http://dbpedia.org/resource/United_States_of_Colombia http://linkedgeodata.org/triplify/node424314631 1 Country http://dbpedia.org/resource/Sultanate_of_Egypt http://linkedgeodata.org/triplify/node424316018 1 Country http://dbpedia.org/resource/Captaincies_of_Brazil http://linkedgeodata.org/triplify/node424313734 1 Country http://dbpedia.org/resource/Pakistan http://linkedgeodata.org/triplify/node424314088 1 Country http://dbpedia.org/resource/Indonesia http://linkedgeodata.org/triplify/node424311942 1 Country http://dbpedia.org/resource/First_Republic_of_Armenia http://linkedgeodata.org/triplify/node249399277 1 Country http://dbpedia.org/resource/Egypt_Eyalet http://linkedgeodata.org/triplify/node424316018 1 Country http://dbpedia.org/resource/Malagasy_Protectorate http://linkedgeodata.org/triplify/node432425028 1 Country http://dbpedia.org/resource/French_Fifth_Republic http://linkedgeodata.org/triplify/node1363947712 1 Country http://dbpedia.org/resource/Costa_Rica http://linkedgeodata.org/triplify/node424297465 1 Country http://dbpedia.org/resource/New_Sweden http://linkedgeodata.org/triplify/node424313277 1 Country http://dbpedia.org/resource/Somalia http://linkedgeodata.org/triplify/node424317039 1 Country http://dbpedia.org/resource/Somali_Republic http://linkedgeodata.org/triplify/node424317039 1 Country http://dbpedia.org/resource/Tsardom_of_Russia http://linkedgeodata.org/triplify/node424314830 1 Country http://dbpedia.org/resource/Euro_gold_and_silver_commemorative_coins_(Belgium) http://linkedgeodata.org/triplify/node1684793666 1 Country http://dbpedia.org/resource/Georgia_(country) http://linkedgeodata.org/triplify/node432424985 1 Country http://dbpedia.org/resource/Spanish_West_Indies http://linkedgeodata.org/triplify/node148332300 1 Country http://dbpedia.org/resource/Albania http://linkedgeodata.org/triplify/node424310601 1 Country http://dbpedia.org/resource/Cameroon http://linkedgeodata.org/triplify/node424315932 1 Country http://dbpedia.org/resource/Belgium http://linkedgeodata.org/triplify/node1684793666 1 Country http://dbpedia.org/resource/Socialist_Republic_of_Macedonia http://linkedgeodata.org/triplify/node424297810 1 Country http://dbpedia.org/resource/Rwanda http://linkedgeodata.org/triplify/node424311738 1 Country http://dbpedia.org/resource/Canada http://linkedgeodata.org/triplify/node424313760 1 Country http://dbpedia.org/resource/Croatia http://linkedgeodata.org/triplify/node424311353 1 Country http://dbpedia.org/resource/Government_of_National_Unity_(Hungary) http://linkedgeodata.org/triplify/node249399347 1 Country http://dbpedia.org/resource/South_Korea http://linkedgeodata.org/triplify/node249399362 1 Country http://dbpedia.org/resource/Palau http://linkedgeodata.org/triplify/node1511934680 1 Country http://dbpedia.org/resource/Euro_gold_and_silver_commemorative_coins_(Slovenia) http://linkedgeodata.org/triplify/node208592566 1 Country http://dbpedia.org/resource/Iran http://linkedgeodata.org/triplify/node424298311 1 Country http://dbpedia.org/resource/Argentina http://linkedgeodata.org/triplify/node249399280 1 Country http://dbpedia.org/resource/Ukraine http://linkedgeodata.org/triplify/node1648342273 1 Country http://dbpedia.org/resource/Iraq http://linkedgeodata.org/triplify/node424316205 1 Country http://dbpedia.org/resource/Khedivate_of_Egypt http://linkedgeodata.org/triplify/node424316018 1 Country http://dbpedia.org/resource/Hungarian_State_(1849) http://linkedgeodata.org/triplify/node249399347 1 Country http://dbpedia.org/resource/Samoa http://linkedgeodata.org/triplify/node1529444079 1 Country http://dbpedia.org/resource/Suriname_(Kingdom_of_the_Netherlands) http://linkedgeodata.org/triplify/node432425087 1 Country http://dbpedia.org/resource/Ghana http://linkedgeodata.org/triplify/node424313836 1 Country http://dbpedia.org/resource/Democratic_Republic_of_Georgia http://linkedgeodata.org/triplify/node432424985 1 Country http://dbpedia.org/resource/Namibia http://linkedgeodata.org/triplify/node424311662 1 Country http://dbpedia.org/resource/Faroe_Islands http://linkedgeodata.org/triplify/node424312044 1 Country http://dbpedia.org/resource/Bourbon_Restoration http://linkedgeodata.org/triplify/node1363947712 1 Country http://dbpedia.org/resource/Aruba http://linkedgeodata.org/triplify/node1770467684 1 Country http://dbpedia.org/resource/Chad http://linkedgeodata.org/triplify/node424298076 1 Country http://dbpedia.org/resource/Morocco http://linkedgeodata.org/triplify/node432425041 1 Country http://dbpedia.org/resource/Democratic_Republic_of_Madagascar http://linkedgeodata.org/triplify/node432425028 1 Country http://dbpedia.org/resource/Socialist_Republic_of_Serbia http://linkedgeodata.org/triplify/node369137525 1 Country http://dbpedia.org/resource/Cyprus http://linkedgeodata.org/triplify/node424313406 1 Country http://dbpedia.org/resource/Hamburg http://linkedgeodata.org/triplify/node20833623 1 City http://dbpedia.org/resource/Munich http://linkedgeodata.org/triplify/node1700534808 1 City http://dbpedia.org/resource/Maribor http://linkedgeodata.org/triplify/node36667529 1 City http://dbpedia.org/resource/Wolverhampton http://linkedgeodata.org/triplify/node25472086 1 City http://dbpedia.org/resource/Nijmegen http://linkedgeodata.org/triplify/node867975219 1 City http://dbpedia.org/resource/Dortmund http://linkedgeodata.org/triplify/node25293125 1 City http://dbpedia.org/resource/Essen http://linkedgeodata.org/triplify/node27350363 1 City http://dbpedia.org/resource/Podgorica http://linkedgeodata.org/triplify/node123860883 1 City http://dbpedia.org/resource/Zalaegerszeg http://linkedgeodata.org/triplify/node183813583 1 City http://dbpedia.org/resource/Hagen http://linkedgeodata.org/triplify/node240110547 1 City http://dbpedia.org/resource/The_Hague http://linkedgeodata.org/triplify/node235857686 1 City http://dbpedia.org/resource/Haarlem http://linkedgeodata.org/triplify/node296054949 1 City http://dbpedia.org/resource/Belgrade http://linkedgeodata.org/triplify/node60571493 1 City http://dbpedia.org/resource/City_Municipality_of_Ljubljana http://linkedgeodata.org/triplify/node6968827 1 City http://dbpedia.org/resource/Sheffield http://linkedgeodata.org/triplify/node266155142 1 City http://dbpedia.org/resource/Oxford http://linkedgeodata.org/triplify/node12805909 1 City http://dbpedia.org/resource/Leeds http://linkedgeodata.org/triplify/node358309 1 City http://dbpedia.org/resource/Banja_Luka http://linkedgeodata.org/triplify/node248407024 1 City http://dbpedia.org/resource/Budapest http://linkedgeodata.org/triplify/node85788293 1 City http://dbpedia.org/resource/Uzhhorod http://linkedgeodata.org/triplify/node111353560 1 City http://dbpedia.org/resource/Southampton http://linkedgeodata.org/triplify/node2478628079 1 City http://dbpedia.org/resource/Skopje http://linkedgeodata.org/triplify/node170792214 1 City http://dbpedia.org/resource/Kiel http://linkedgeodata.org/triplify/node24487450 1 City http://dbpedia.org/resource/Split,_Croatia http://linkedgeodata.org/triplify/node1467604275 1 City http://dbpedia.org/resource/Split,_Croatia http://linkedgeodata.org/triplify/way177736372 1 City http://dbpedia.org/resource/Rotterdam http://linkedgeodata.org/triplify/node44410812 1 City http://dbpedia.org/resource/Tuzla http://linkedgeodata.org/triplify/node275005833 1 City http://dbpedia.org/resource/Worcester http://linkedgeodata.org/triplify/node20971097 1 City http://dbpedia.org/resource/Satu_Mare http://linkedgeodata.org/triplify/node146881926 1 City http://dbpedia.org/resource/Leverkusen http://linkedgeodata.org/triplify/node1651889631 1 City http://dbpedia.org/resource/Erfurt http://linkedgeodata.org/triplify/node240038130 1 City http://dbpedia.org/resource/Wrocław http://linkedgeodata.org/triplify/node418392093 1 City http://dbpedia.org/resource/Derby http://linkedgeodata.org/triplify/node21904579 1 City http://dbpedia.org/resource/Tallinn http://linkedgeodata.org/triplify/node29787492 1 City http://dbpedia.org/resource/Dresden http://linkedgeodata.org/triplify/node240076989 1 City http://dbpedia.org/resource/Chemnitz http://linkedgeodata.org/triplify/node20830918 1 City http://dbpedia.org/resource/Tilburg http://linkedgeodata.org/triplify/node1206763375 1 City http://dbpedia.org/resource/Bergen http://linkedgeodata.org/triplify/node21261083 1 City http://dbpedia.org/resource/Bergen http://linkedgeodata.org/triplify/node893278148 1 City http://dbpedia.org/resource/Newcastle_upon_Tyne http://linkedgeodata.org/triplify/node20910685 1 City http://dbpedia.org/resource/Manchester http://linkedgeodata.org/triplify/node294001443 1 City http://dbpedia.org/resource/Sarajevo http://linkedgeodata.org/triplify/node2021709163 1 City http://dbpedia.org/resource/Dublin http://linkedgeodata.org/triplify/node3473474851 1 City http://dbpedia.org/resource/Oradea http://linkedgeodata.org/triplify/node2691094601 1 City http://dbpedia.org/resource/Cologne http://linkedgeodata.org/triplify/node20953083 1 City http://dbpedia.org/resource/Toledo,_Spain http://linkedgeodata.org/triplify/node1679058496 1 City http://dbpedia.org/resource/Nottingham http://linkedgeodata.org/triplify/node24913081 1 City http://dbpedia.org/resource/'s-Hertogenbosch http://linkedgeodata.org/triplify/node60400458 1 City http://dbpedia.org/resource/Belz http://linkedgeodata.org/triplify/node266185307 1 City http://dbpedia.org/resource/Swansea http://linkedgeodata.org/triplify/node21063632 1 City http://dbpedia.org/resource/Alicante http://linkedgeodata.org/triplify/node21323935 1 City http://dbpedia.org/resource/Nicosia http://linkedgeodata.org/triplify/node1893015330 1 City http://dbpedia.org/resource/Kassel http://linkedgeodata.org/triplify/node1583294719 1 City http://dbpedia.org/resource/Breda http://linkedgeodata.org/triplify/node867998189 1 City http://dbpedia.org/resource/Barcelona http://linkedgeodata.org/triplify/node152364165 1 City http://dbpedia.org/resource/Wuppertal http://linkedgeodata.org/triplify/node240090728 1 City http://dbpedia.org/resource/Groningen http://linkedgeodata.org/triplify/node3125538288 1 City http://dbpedia.org/resource/Tarragona http://linkedgeodata.org/triplify/node152377920 1 City http://dbpedia.org/resource/Zwolle http://linkedgeodata.org/triplify/node235886826 1 City http://dbpedia.org/resource/Amsterdam http://linkedgeodata.org/triplify/node268396336 1 City http://dbpedia.org/resource/Bucharest http://linkedgeodata.org/triplify/node96209423 1 City http://dbpedia.org/resource/Zadar http://linkedgeodata.org/triplify/way177758144 1 City http://dbpedia.org/resource/Zadar http://linkedgeodata.org/triplify/node1832917478 1 City http://dbpedia.org/resource/Leeuwarden http://linkedgeodata.org/triplify/node235278210 1 City http://dbpedia.org/resource/Burgos http://linkedgeodata.org/triplify/node166920565 1 City http://dbpedia.org/resource/Potsdam http://linkedgeodata.org/triplify/node1695218178 1 City http://dbpedia.org/resource/Eindhoven http://linkedgeodata.org/triplify/node42616340 1 City http://dbpedia.org/resource/Osijek http://linkedgeodata.org/triplify/node253924853 1 City http://dbpedia.org/resource/Bălți http://linkedgeodata.org/triplify/node266185307 1 City http://dbpedia.org/resource/Ljubljana http://linkedgeodata.org/triplify/node6968827 1 City http://dbpedia.org/resource/Nuremberg http://linkedgeodata.org/triplify/node1569338041 1 City http://dbpedia.org/resource/Amersfoort http://linkedgeodata.org/triplify/node667889917 1 City http://dbpedia.org/resource/Arnhem http://linkedgeodata.org/triplify/node44728830 1 City http://dbpedia.org/resource/Delft http://linkedgeodata.org/triplify/node274247451 1 City http://dbpedia.org/resource/Newport,_Wales http://linkedgeodata.org/triplify/node26700977 1 City http://dbpedia.org/resource/Utrecht http://linkedgeodata.org/triplify/node235861650 1 City http://dbpedia.org/resource/Bochum http://linkedgeodata.org/triplify/node240099833 1 City http://dbpedia.org/resource/Birmingham http://linkedgeodata.org/triplify/node17861291 1 City http://dbpedia.org/resource/Brno http://linkedgeodata.org/triplify/node1601566699 1 City http://dbpedia.org/resource/Cambridge http://linkedgeodata.org/triplify/node20971094 1 City http://dbpedia.org/resource/Leiden http://linkedgeodata.org/triplify/node235865586 1 City http://dbpedia.org/resource/Mostar http://linkedgeodata.org/triplify/node29976450 1 City http://dbpedia.org/resource/Zagreb http://linkedgeodata.org/triplify/node1551865419 1 City http://dbpedia.org/resource/Cluj-Napoca http://linkedgeodata.org/triplify/node32591050 1 City http://dbpedia.org/resource/Hanover http://linkedgeodata.org/triplify/node1651888734 1 City http://dbpedia.org/resource/Riga http://linkedgeodata.org/triplify/node26707967 1 City http://dbpedia.org/resource/Pula http://linkedgeodata.org/triplify/node1830216906 1 City http://dbpedia.org/resource/Bonn http://linkedgeodata.org/triplify/node26373169 1 City http://dbpedia.org/resource/Thessaloniki http://linkedgeodata.org/triplify/node57554537 1 City http://dbpedia.org/resource/North_College_Thessaloniki__Thessaloniki__1 http://linkedgeodata.org/triplify/node57554537 1 City http://dbpedia.org/resource/Lviv http://linkedgeodata.org/triplify/node26150791 1 City http://dbpedia.org/resource/Bristol http://linkedgeodata.org/triplify/node959199214 1 City http://dbpedia.org/resource/Prague http://linkedgeodata.org/triplify/node1601837931 1 City http://dbpedia.org/resource/Grenada http://linkedgeodata.org/triplify/node240423025 1 City http://dbpedia.org/resource/Granada http://linkedgeodata.org/triplify/node240423025 1 City http://dbpedia.org/resource/Bratislava http://linkedgeodata.org/triplify/node530544342 1 City http://dbpedia.org/resource/Leicester http://linkedgeodata.org/triplify/node10021976 1 City http://dbpedia.org/resource/Herne,_North_Rhine-Westphalia http://linkedgeodata.org/triplify/node21315295 1 City http://dbpedia.org/resource/Georgian_Theatre_Royal http://linkedgeodata.org/triplify/node3215997829 1 Theatre http://dbpedia.org/resource/Stamford_Center_for_the_Arts http://linkedgeodata.org/triplify/node1684145430 1 Theatre http://dbpedia.org/resource/Brockley_Jack_Theatre http://linkedgeodata.org/triplify/node2715385289 1 Theatre http://dbpedia.org/resource/St._James_Theatre http://linkedgeodata.org/triplify/node2693036336 1 Theatre http://dbpedia.org/resource/Little_Theatre_(Leicester) http://linkedgeodata.org/triplify/node2450234535 1 Theatre http://dbpedia.org/resource/Little_Theatre_(Leicester) http://linkedgeodata.org/triplify/node3392761211 1 Theatre http://dbpedia.org/resource/Little_Theatre_(Leicester) http://linkedgeodata.org/triplify/node1518354242 1 Theatre http://dbpedia.org/resource/Mayakovsky_Theatre http://linkedgeodata.org/triplify/node1684243150 1 Theatre http://dbpedia.org/resource/Teatro_del_Silenzio http://linkedgeodata.org/triplify/node1590068388 1 Theatre http://dbpedia.org/resource/Isala_Theater http://linkedgeodata.org/triplify/node2333673289 1 Theatre http://dbpedia.org/resource/Mill_Theatre_Dundrum http://linkedgeodata.org/triplify/node256171098 1 Theatre http://dbpedia.org/resource/People's_Theatre,_Newcastle_upon_Tyne http://linkedgeodata.org/triplify/node137496729 1 Theatre http://dbpedia.org/resource/Moulin_Rouge http://linkedgeodata.org/triplify/node252969214 1 Theatre http://dbpedia.org/resource/Palace_Theatre_(New_York_City) http://linkedgeodata.org/triplify/node1684145430 1 Theatre http://dbpedia.org/resource/Le_Divan_du_Monde http://linkedgeodata.org/triplify/node1697073915 1 Theatre http://dbpedia.org/resource/New_Theatre_Royal http://linkedgeodata.org/triplify/node249759194 1 Theatre http://dbpedia.org/resource/New_Theatre_Royal http://linkedgeodata.org/triplify/node349212537 1 Theatre http://dbpedia.org/resource/Plaza_Theatre_(El_Paso) http://linkedgeodata.org/triplify/node1128833022 1 Theatre http://dbpedia.org/resource/Plaza_Theatre_(Bangalore) http://linkedgeodata.org/triplify/node1128833022 1 Theatre http://dbpedia.org/resource/Plaza_Theatre_(Atlanta) http://linkedgeodata.org/triplify/node1128833022 1 Theatre http://dbpedia.org/resource/Her_Majesty's_Theatre,_Melbourne http://linkedgeodata.org/triplify/node1907704058 1 Theatre http://dbpedia.org/resource/Alhambra,_Blackpool http://linkedgeodata.org/triplify/node2195758631 1 Theatre http://dbpedia.org/resource/Jersey_Opera_House http://linkedgeodata.org/triplify/node1765756723 1 Theatre http://dbpedia.org/resource/Lyceum_Theatre_(Broadway) http://linkedgeodata.org/triplify/node245569130 1 Theatre http://dbpedia.org/resource/Theatre_Royal,_Dumfries http://linkedgeodata.org/triplify/node249759194 1 Theatre http://dbpedia.org/resource/Theatre_Royal,_Dumfries http://linkedgeodata.org/triplify/node349212537 1 Theatre http://dbpedia.org/resource/Theatre_Royal,_Manchester http://linkedgeodata.org/triplify/node249759194 1 Theatre http://dbpedia.org/resource/Theatre_Royal,_Manchester http://linkedgeodata.org/triplify/node349212537 1 Theatre http://dbpedia.org/resource/The_Theatre,_Leeds http://linkedgeodata.org/triplify/node2357638497 1 Theatre http://dbpedia.org/resource/The_Theatre,_Leeds http://linkedgeodata.org/triplify/node1141134316 1 Theatre http://dbpedia.org/resource/The_Theatre,_Leeds http://linkedgeodata.org/triplify/node2327112298 1 Theatre http://dbpedia.org/resource/The_Theatre,_Leeds http://linkedgeodata.org/triplify/node1917586915 1 Theatre http://dbpedia.org/resource/Alhambra_(Paris) http://linkedgeodata.org/triplify/node2195758631 1 Theatre http://dbpedia.org/resource/Leas_Cliff_Hall http://linkedgeodata.org/triplify/node253231965 1 Theatre