Semantic Web 0 (0) 1 1 IOS Press

1 1 2 2 3 3 4 Ontology Engineering: Current State, 4 5 5 6 Challenges, and Future Directions 6 7 7 8 Tania Tudorache 8 9 Stanford Center for Biomedical Informatics Research, Stanford University, CA, USA 9 10 E-mail: [email protected] 10 11 11 12 12 Editors: Pascal Hitzler, Kansas State University, Manhattan, KS, USA; Krzysztof Janowicz, University of California, Santa Barbara, USA 13 Solicited reviews: Aldo Gangemi, LIPN Université Paris13-CNRS-Sorbonne Cité, France and ISTC-CNR Roma, Italy; Oscar Corcho, 13 14 Universidad Politécnica de Madrid, Spain 14 15 15 16 16 17 17 18 18 19 Abstract. 19 20 In the last decade, ontologies have become widely adopted in a variety of fields ranging from biomedicine, to finance, engi- 20 21 neering, law, and cultural heritage. The ontology engineering field has been strengthened by the adoption of several standards 21 22 pertaining to ontologies, by the development or extension of ontology building tools, and by a wider recognition of the impor- 22 23 tance of standardized vocabularies and formalized semantics. Research into ontology engineering has also produced methods 23 24 and tools that are used more and more in production settings. Despite all these advancements, ontology engineering is still a 24 difficult process, and many challenges still remain to be solved. This paper gives an overview of how the ontology engineering 25 25 field has evolved in the last decade and discusses some of the unsolved issues and opportunities for future research. 26 26 27 Keywords: Ontologies, Ontology Engineering, Methods, Standards, Tooling, Patterns, Challenges, Future Research 27 28 28 29 29 30 30 31 1. Ontologies Make an Impact much more widely adopted in academia, industry and 31 32 government environments, and are finally making an 32 33 The research on ontologies in computer science impact in many domains. 33 34 started in the early 1990s. Ontologies were proposed Biomedicine has widely adopted ontologies since 34 35 as a way to enable people and software agents to seam- their beginnings. The Gene Ontology (GO) [3]—a 35 36 lessly share information about a domain of interest. comprehensive ontology describing the function of 36 37 An ontology was defined as a conceptual representa- genes—is the poster child for a successful ontology 37 38 tion of the entities, their properties and relationships development project that has produced a big impact in 38 39 in a domain [1]. The ultimate goal of using ontolo- biomedical research. Indeed, GO is routinely used in 39 40 gies was to make the knowledge in a domain compu- the computational analysis of large-scale molecular bi- 40 41 tationally useful [2]. The initial research period was ology and genetics experiments [4]. Researchers have 41 42 followed by a time of great excitement about using also used ontologies in biomedicine to standardize 42 43 ontologies to solve a wide range of problems. How- terminology in particular domains, to annotate large 43 44 ever, the enthusiasm dwindled in the early 2000s, as biomedical datasets, to integrate data, and to aid struc- 44 45 the methods and infrastructures for building and us- tured data mining and machine learning [5, 6]. 45 46 ing ontologies were not mature enough at that time. One notable example of the impact ontologies are 46 47 Nonetheless, significant changes have taken place in making in healthcare is the development of the 11th 47 48 the last decade: The research and development on on- revision of the International Classification of Diseases 48 49 tologies had a big boost, more standardization efforts (ICD-11). ICD—developed by the World Health Or- 49 50 were on the way, and industry started to buy into se- ganization (WHO)—is the international standard for 50 51 mantic technologies. As a result, ontologies are now reporting diseases and health conditions, and is used 51

1570-0844/0-1900/$35.00 © 0 – IOS Press and the authors. All rights reserved 2 T. Tudorache / Ontology Engineering

1 to identify health trends and statistics on a global allowing the exchange of cultural heritage data be- 1 2 scale [7]. ICD-11 is now using OWL to encode the for- tween institutions. 2 3 mal representation of diseases, their properties, and re- Other fields have also adopted ontologies more 3 4 lations, as well as mappings to other terminologies [8]. widely. Researchers have used ontologies in the legal 4 5 The financial industry has embraced the use of on- domain to formally represent laws and regulations, to 5 6 tologies. The most prominent example is the Finan- simulate legal actions, or for semantic searching and 6 7 cial Industry Business Ontology (FIBO)—the industry indexing [21]. In the geographical domain, the ISO 7 8 standard resource for the definitions of business con- 19150-1:2012 [22] defines a high-level model of the 8 9 cepts in the financial services industry [9]. FIBO is components required to handle semantics in the ISO 9 10 developed by the Enterprise Data Management Coun- geographic information standards with ontologies. The 10 11 cil (EDMC) and it is standardized through the Object second part of the standard, ISO 19150-2:2015 [23] 11 12 Management Group (OMG). FIBO is built as a series defines rules to convert the UML models used in the 12 13 of OWL ontologies and is developed using a rigorous ISO geographic information standards into OWL. 13 14 and well-defined process, known as the “Build-Test- The examples we mentioned above are not meant 14 15 Deploy-Maintain” methodology. to be comprehensive. They show how ontologies have 15 16 Engineering is another field that has adopted on- been embraced by a wide range of fields in the last 16 17 tologies from the early 1990s, long before the stan- decade and how they are making an impact. 17 18 dardization of the current Semantic Web languages, This paper is meant to give a retrospective overview 18 19 such as OWL and RDF [10, 11]. In the last decade, of how the ontology landscape and ontology engineer- 19 20 we have witnessed significant efforts around using on- ing have evolved in the last decade, current challenges, 20 21 tologies to cover different aspects of engineering rang- and prospects for future research. This paper can hope- 21 22 ing from defining requirements [12], to integrating dif- fully also serve as an introduction for newcomers in 22 23 ferent engineering models [13], to detecting incon- the field. We briefly discuss standards relevant to on- 23 24 sistencies in models in multidisciplinary engineering tology engineering that have been adopted in the last 24 25 projects [14]. Sabou et. al [15] provide a comprehen- decade (Section 2), highly visible and influential on- 25 26 sive overview into how ontologies and Semantic Web tologies and knowledge bases that are constructed by 26 27 27 technologies can assist in building intelligent engineer- large communities (Section 3), trends in ontology en- 28 gineering from the last ten years (Section 4), and cur- 28 ing applications. 29 rent challenges (Section 5) and opportunities for future 29 A lot of work has gone into developing methods 30 research (Section 6). 30 and tools for publishing linked datasets of the vast cul- 31 31 tural heritage field in the last decade [16]. One out- 32 32 standing example is the linked open dataset of the Eu- 33 2. New Standards 33 ropeana Collections1 [17] which provides access to 34 34 millions of artworks, artefacts, books, films and mu- 35 The significant standardization efforts on ontolo- 35 sic from European museums, galleries, libraries and 36 gies and Semantic Web languages in the last decade 36 archives. Another example comes from the scholarly 37 also prove the maturation of the field. Figure 1 shows 37 publishing domain, in which the SPAR (Semantic Pub- 38 some of ontology-related standards that the World 38 lishing and Referencing) Ontologies [18] are having 39 Wide Web Consortium (W3C) has adopted in the past 39 a high impact since they were released in 2015. The 40 decade. Several ontologies and vocabularies have be- 40 recently released Italian Cultural Heritage knowledge 41 come W3C recommendations: The Time Ontology 41 graph, ArCO2 [19], consists of a network of seven 42 (OWL-Time) [24]—describing the temporal properties 42 43 high-quality ontologies, modeling the cultural her- of resources; the Semantic Sensors Network Ontol- 43 44 itage domain, and contains over 169 million triples ogy (SSN) [25]—representing sensors and their obser- 44 45 about 820 thousand cultural entities. The testament vations; the Provenance Ontology (PROV-O) [26]— 45 46 to the importance of ontologies in the cultural her- describing provenance information from different sys- 46 47 itage field is shown also by the adoption of the ISO tems; or the RDF Data Cube [27]—enabling publish- 47 48 21127:2014 [20] standard that prescribes an ontology ing of multi-dimensional data on the Web. 48 49 Ontology and knowledge representation languages 49 50 1https://data.europeana.eu have also evolved as proved by the adoption of new 50 51 2http://dati.beniculturali.it/arco/index.php versions of the standards: RDF 1.1 was adopted in 51 T. Tudorache / Ontology Engineering 3

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 12 13 13 14 14 Fig. 1. The timeline of W3C recommendations related to ontologies and vocabularies in the last decade (2010-2019). The years that do not have 15 15 any recommendations are skipped. 16 16 17 February 2014, and introduced identifiers as IRIs, RDF from the Wikipedia infoboxes. The English version 17 18 datasets, and new serialization formats, such as RDFa3 of the DBpedia knowledge base describes more than 18 19 and Turtle.4 OWL 2.05 was adopted in December 4.5 million things to date.8 The DBpedia ontology is 19 20 2012, and introduced several new features, such as rather small—it contains 685 classes and 2,795 prop- 20 21 support for keys and property chains, richer datatypes erties9— and it is manually built through a commu- 21 22 and data ranges, qualified cardinality restrictions, and nity effort using a wiki. The DBpedia ontology con- 22 23 23 enhanced annotation capabilities. tains the most commonly used information in the 24 24 Another notable W3C recommendation adopted Wikipedia infoboxes. To extract the information from 25 25 in July 2017 is the Shapes Constraints Language the Wikipedia infoboxes, the DBpedia community 26 26 (SHACL)6 that provides a mechanism for validating manually creates the mappings between the infoboxes 27 27 constraints against RDF graphs, a feature that was and the ontology in the DBpedia Mappings Wiki.10 28 28 sorely lacking from the current knowledge represen- DBpedia is widely used both within and outside the 29 7 29 tation standards. ShEx is an alternative way of val- Semantic Web research community, with many appli- 30 30 idating RDF and OWL and is backed by an active cations and tools being built around it. Most impor- 31 31 user community. SHACL and ShEx are considered by tantly, DBpedia has become a central hub within the 32 some as a simpler knowledge representation languages 32 33 web of Open Linked Data, as many RDF data sets link 33 that might provide an alternative to the more complex back to DBpedia entities. The Linked Open Data ini- 34 OWL representation. 34 35 tiative would not have been successful without DBpe- 35 36 dia. 36 37 A project with a similar goal, but significantly dif- 37 3. Large-Scale Community-Driven Creation of ferent approach is YAGO [29]. YAGO builds a large- 38 Knowledge 38 39 scale ontology also from Wikipedia infoboxes. It uses 39 40 the Wikipedia categories to find a type for each en- 40 Another area of substantial growth in the last decade 41 tity, which is then mapped into the WordNet taxon- 41 is the development of community-authored ontolo- 42 omy [30]. In this way, YAGO creates a high-quality 42 gies and knowledge bases. One of the most visible 43 taxonomy which is used not only in the YAGO-driven 43 community-driven projects in the Semantic Web is 44 applications, but also to perform consistency checks on 44 DBpedia [28]. DBpedia is a crowd-sourced commu- 45 the automatically extracted information. The YAGO2 45 nity effort to extract structured, multilingual content 46 ontology [31] extends the YAGO model with spatial 46 47 and temporal dimensions. The spatial information is 47 3 48 https://www.w3.org/TR/rdfa-core/ 48 4https://www.w3.org/TR/turtle/ 49 49 5https://www.w3.org/TR/owl2-overview/ 8https://wiki.dbpedia.org/about/facts-figures 50 6https://www.w3.org/TR/shacl/ 9https://wiki.dbpedia.org/services-resources/ontology 50 51 7http://shex.io/ 10http://mappings.dbpedia.org 51 4 T. Tudorache / Ontology Engineering

1 obtained by integration with GeoNames,11 a geograph- searchers have built knowledge networks before, the 1 2 ical database covering all countries and which contains phrase “knowledge graph” started catching on once 2 3 over eleven million placenames. YAGO2 contains 447 Google announced their Google KG in May 2012. 3 4 million facts about 9.8 million entities and has a high Since then, we have seen a flourishing of KGs. In- 4 5 accuracy of 95% for the facts stored in YAGO. YAGO3 deed, most large companies, including Amazon, Net- 5 6 extends the YAGO content with information from the flix, Pintrest, LinkedIn, Microsoft, Uber, NASA, IBM, 6 7 Wikipedias in multiple languages which is fused to and Alibaba are developing their own KGs. Gartner 7 8 create a coherent knowledgebase. also identified knowledge graphs as an emerging tech- 8 9 Another notable large-scale, community-driven knowl- nology trend in their 2018 technology report [33]. 9 10 edge base is Wikidata12—a free and open knowl- Even with this high adoption, there is no single widely 10 11 edge base that serves as the central storage of struc- adopted definition of a KG. A common denomina- 11 12 tured data for several Wikimedia projects, including tor is that KGs contain entities that are inter-related, 12 13 Wikipedia [32]. The Wikidata project started in Octo- and are usually at the data level. The level of for- 13 14 ber 2012. Initially, it was conceived as a central place mality varies a lot: While some use RDF and OWL 14 15 for storing inter-languages links between Wikipedia and a schema, others are schema-less and use property 15 16 articles about the same topic in different languages, graphs.17 Some KGs are built bottom-up using Ma- 16 17 and nowadays, Wikidata provides the structured data chine Learning (ML) and Natural Language Process- 17 18 for almost 60% of Wikipedia pages.13 Wikidata’s data ing (NLP) techniques, while others are built top-down. 18 19 model is centered around items with unique identifiers Their uses range widely from intelligent search, to an- 19 20 that contain statements—basically, key-value pairs— alytics, cataloging, data integration, and more. 20 21 which can be qualified (e.g., with provenance infor- Community curation is not only used in building 21 22 mation). One distinguishing feature of Wikidata is its large-scale knowledge bases like the ones presented 22 23 collaborative authoring model: Both humans and pro- above, but also for building ontologies in different do- 23 24 grammable bots can contribute content, with a major- mains. For example, the Gene Ontology (GO) is cre- 24 25 ity of the edits (about 90%) coming from bots. The ated by a community-driven workflow, in which com- 25 26 project is highly active, containing more than 63 mil- munity members suggest new entities, or changes to 26 27 lion entities, and over 900 million edits as of April current entities using the Gene Ontology GitHub issue 27 28 2019. tracker.18 Only a handful of editors actually edit the 28 29 Another high-impact project for creating vocabu- GO ontology file based on the community feedback. 29 30 laries for structured data to be used on Web con- The GO changes rapidly with nightly releases. Simi- 30 31 tent is Schema.org.14 Started in 2011 by Google, Mi- larly, the ICD-11 ontology project solicits community 31 32 crosoft, Yahoo and Yandex, the Schema.org vocab- feedback in the form of comments, structured propos- 32 33 ularies enable Web content creators to add struc- als, or translations through its public Web platform.19 33 34 tured metadata to their Web pages, so that search en- The proposed changes are discussed by the ICD-11 34 35 gines can better understand the content of the page. editorial committee, and if approved, they are imple- 35 36 The Schema.org vocabularies are developed by an mented in a Web-based collaborative ontology devel- 36 37 37 open community process using W3C mailing lists and opment environment, called iCAT [34]. The ICD-11 38 38 GitHub.15 Schema.org also offers an extension mech- project also employs another level of scientific review 39 39 anism that communities have used to create domain- by sending a PDF extracted from parts of the ontol- 40 40 specific vocabularies, for example, for bibliographic or ogy to domain experts in different medical specialties. 41 41 auto extensions. As of April 2019, these extensions are Even though several large-scale ontology projects use 42 42 folded back into the main Schema.org vocabulary.16 collaboration processes and involve the larger commu- 43 43 Certainly, knowledge graphs (KG) are one of the nity in the authoring process, there are no two collabo- 44 44 leading topics of the last decade. Even though re- ration workflows that are the same. For more informa- 45 tion, we refer the reader to the review by Simperl and 45 46 46 47 11http://www.geonames.org/ 47 12 17 48 https://www.wikidata.org A property graph is a graph for which the edges are labeled, and 48 13http://wdcm.wmflabs.org/WD_percentUsageDashboard/ both vertices and edges can have any number of key/value properties 49 49 14https://schema.org/ associated with them. 50 15https://www.w3.org/community/schemaorg/ 18https://github.com/geneontology/go-ontology/issues 50 51 16https://schema.org/docs/extension.html 19https://icd.who.int/dev11/l-m/en 51 T. Tudorache / Ontology Engineering 5

1 Luczak-Rösch [35] of the current collaborative ontol- idated in several large-scale ontology development 1 2 ogy engineering practices. projects [40, 41]. The OBO Foundry defines many 2 3 of the principles20 by which OBO Foundry ontolo- 3 4 gies, including the GO, should abide, such as ver- 4 5 4. Ontology Engineering sioning, naming conventions, defining relations, lo- 5 6 cus of authority, documentation, collaboration pro- 6 7 Ontology engineering did not change significantly cess, orthogonality of ontologies, and reuse [42]. The 7 8 in the last decade. Even though there has been progress OBO Foundry is a good example of how a commu- 8 9 in specific areas, which we will briefly discuss in nity can develop ontologies for a specific domain. Mo- 9 10 this section, the work on new ontology engineering tivated by the need to manage ontologies that are be- 10 11 methodologies did not seem to progress much. Even to coming more modular and inter-dependent, the GO 11 12 date, the most cited ontology engineering method, ac- project developed a continuous integration process us- 12 13 cording to Google Scholar, is the Ontology 101 guide ing Jenkins and Hudson for building ontologies that 13 14 by Noy and McGuiness [36] from 2001. became a model for the development of other on- 14 15 The NeOn project (2006-2010) produced the most tologies [43]. ROBOT21 is a generic command-line 15 16 comprehensive methodology for building networked tool and Java library for performing common ontol- 16 17 ontologies [37]. The NeOn methodology describes a ogy tasks, such as, computing differences between on- 17 18 set of nine scenarios for building ontologies focusing tology versions, merging, extracting ontology mod- 18 19 on reuse of ontological and non-ontological resources, ules, reasoning, explanation, materializing inferences, 19 20 merging, re-engineering, and also accounting for col- etc. The commands in ROBOT can be chained to- 20 21 laboration. In addition, the methodology also publishes gether to create a powerful, repeatable workflow. An- 21 22 a Glossary of Processes and Activities to support col- other generic tool that was developed as part of the 22 23 laboration, and methodological guidelines for different GO project is TermGenie [44], a Web-based class sub- 23 24 processes and activities involved in ontology engineer- mission form that can generate new classes, once the 24 25 ing. Even though the NeOn methodology had modest submission passes a suite of logical, lexical and struc- 25 26 adoption, the work in the NeOn project produced im- tural checks. TermGenie is generic and customizable 26 27 portant research that advanced the field. and has been deployed in the development of several 27 28 In the last decade, researchers have developed other biomedical ontologies. 28 29 ontology engineering methodologies that have been The adoption of ontologies into mainstream is also 29 30 deployed in specific projects, but are still yet to be proven by the recent publication of several books fo- 30 31 widely adopted. For example, the UPON Lite method- cusing on ontology engineering, such as, “Demystify- 31 32 ology [38] supports the rapid prototyping of trial on- ing OWL for the Enterprise” in 2018 by Uschold [45], 32 33 tologies, while trying to enhance the role of domain ex- the “Ontology Engineering” in 2019 by Kendall and 33 34 perts and minimize the need for ontology experts. The McGuiness [46], and the "An Introduction to Ontology 34 35 methodology uses a socially-oriented approach and fa- Engineering" in 2018 by Keet [47]. 35 36 miliar tools, such as spreadsheets, to make the engi- 36 37 neering process more accessible to domain experts. 4.1. Patterns, Templating, and Automation 37 38 The eXtreme Design (XD) methodology [39] uses an 38 39 agile approach to ontology engineering that is focused As ontology engineering became more broadly 39 40 on the reuse of ontology design patterns. The method- used, knowledge engineers needed ways to optimize 40 41 ology is inspired by the principles of extreme program- and accelerate parts of the ontology development pro- 41 42 ming and uses a divide-and-conquer approach. XD is cess. One of the approaches was employing ontology 42 43 iterative and incremental, and tries to address one mod- design patterns—small, modular, and reusable solu- 43 44 eling issue at a time. The modeling issue, defined by a tions to recurrent modeling problems—and templates 44 45 set of competency questions, is mapped to one or more based on these patterns or other representation regular- 45 46 ODPs, which are then integrated into the ontology, and ities in the ontology. Another approach was to use au- 46 47 tested using unit tests. tomation, such as bulk imports, or scripts to accelerate 47 48 The Gene Ontology (GO)—arguably, the most vis- ontology population. 48 49 ible and successful ontology project—has generated 49 50 several ontology engineering methods and tools that 20http://www.obofoundry.org/principles/fp-000-summary.html 50 51 are generic, reusable, and that have already been val- 21http://robot.obolibrary.org/ 51 6 T. Tudorache / Ontology Engineering

1 The initial work on ontology design patterns (ODP) 4.2. Better Tooling Is Available 1 2 dates back to 2005 [48]. The research on ODPs has 2 3 only intensified in the last decade. One of the sta- The tooling for building ontologies has also evolved 3 4 ples of this area is the ODP repository (http://www. considerably in the last decade. The open-source Pro- 4 5 ontologydesignpatterns.org), which was developed as tégé ontology editor [57] has grown its active large 5 6 part of the NeOn project and it is still actively used. community to more than 360,000 registered users. 6 7 The Workshop on Ontology Design Patterns (WOP) WebProtégé [58] is a Web-based editor for OWL 2.0 7 8 that attracts researchers working on ODPs, as well as with a simplified user interface [59] that supports col- 8 9 users trying to apply them, is already in its 10th edi- laboration. WebProtégé also supports tagging, multi- 9 10 10 tion.22 In their book, “Ontology engineering with on- linguality, querying and visualization. The Stanford- 11 11 tology design patterns: Foundations and applications”, hosted WebProtégé server (https://webprotege.stanford. 12 12 Hitzler and colleagues [49] provide a current assess- edu) hosts more than 60,000 ontology projects that 13 13 ment of the research and application of ontology pat- users have created or uploaded to the server. 14 The OnToology [60]—an open-source project that 14 15 terns. Some of the new work on ODPs include the def- 15 automates part of the collaborative ontology devel- 16 inition of a language for the representation of ontology 16 opment process—will generate different types of re- 17 patterns and of their relationships [50]. 17 sources for a GitHub ontology, such as documen- 18 Several biomedical projects adopted the Dead Sim- 18 23 tation using Widoco [61]; class and taxonomy dia- 19 ple OWL Design Patterns (DOS-DPs) —a lightweight, 19 24 grams using the AR2DTool26; and an evaluation re- 20 YAML -based syntax for specifying design patterns [51]. 20 port for common pitfalls using the OOPS! frame- 21 DOS-DPs support the generation of OWL axioms and 21 work [62]. VocBench [63] is another open-source 22 user-facing documentation using a simple format that 22 Web-based SKOS editor that focuses on collabora- 23 can be parsed using out-of-the-box parsers. With DOS- 23 tion. There are also other active ontology engineering 24 DP, users can quickly generate new classes, or change 24 tools, of which we would like to mention: the Live 25 existing ones when a design pattern changes. 25 OWL Documentation Environment (LODE) [64]— 26 Other mechanisms for specifying patterns and gen- 26 an OWL ontology documentation tool that generates 27 erating axioms are the Ontology PreProcessing Lan- 27 documentation in human-readable HTML format, and 28 guage (OPPL) [52] and the Tawny OWL [53]. OPPL is 28 the WebVOWL [65]—a Web-based tool for visual- 29 a macro language based on the Manchester OWL Syn- 29 izing ontologies using a force-directed graph layout, 30 tax [54] that contains instructions for adding or remov- 30 and based on the Visual Notation for OWL Ontologies 31 ing entities and axioms to an OWL ontology. Tawny 31 (VOWL) [66]. 32 OWL, which is built in Clojure25 and backed by the 32 Commercial ontology engineering tools have also 33 OWL API [55], provides a programmatic way to build 33 34 proliferated and gained wide adoption in the last 34 ontologies. Tawny OWL allows ontology engineers to 35 decade. Some of the commercial offerings include the 35 use a wide range of tools available for software de- 27 36 TopQuadrant’s tool suite for vocabulary and meta- 36 velopment, including versioning, distributed develop- 37 data management, the PoolParty Semantic Suite,28 or 37 ment, building, testing and continuous integration. 38 Mondeca’s Intelligent Topic Manager (ITM),29 just to 38 Another approach that adopts widely-used technolo- 39 name a few. Gra.fo30 is the most recent addition of 39 40 gies from software engineering to ontology devel- commercial ontology tools that was launched in late 40 41 opment is OntoMaven [56]. OntoMaven adapts the 2018. Gra.fo is a visual, collaborative, and real-time 41 42 Maven development process to ontology engineering ontology and knowledge graph schema editor that sup- 42 43 in distributed ontology repositories. It supports the ports both OWL/RDF and property graphs. Several 43 44 modular reuse of ontologies, versioning, the life cycle other commercial ontology tools have morphed re- 44 45 and dependency management. cently into knowledge graph solutions. 45 46 46 47 47 22 26 48 http://ontologydesignpatterns.org/wiki/WOP:2019 https://github.com/idafensp/ar2dtool 48 23https://github.com/INCATools/dead_simple_owl_design_ 27https://www.topquadrant.com 49 49 patterns 28https://www.poolparty.biz/ 50 24https://yaml.org/ 29https://mondeca.com/itm/ 50 51 25https://clojure.org/ 30https://gra.fo 51 T. Tudorache / Ontology Engineering 7

1 5. Challenges in Ontology Engineering research is needed to tackle the challenges of bridging 1 2 GraphQL, RDF and SPARQL [70]. 2 3 Semantic Web languages, and especially OWL, The adoption of ontologies is also hindered by 3 4 have a steep learning curve [67, 68] and require a competing approaches that are simpler to use than 4 5 change of perspective, especially for people coming OWL. Microdata and microformats were much more 5 6 from software engineering, object-oriented program- wider found in the 2013 Common Crawl dataset 6 7 ming, or relational database backgrounds. Newcom- than RDFa [71]. Schema.org chose to use an exten- 7 8 ers are faced not only with the daunting task of creat- sion of RDFS Schema as its data model,33 rather 8 9 ing a new type of model for their problem or domain, than OWL, and introduced domainIncludes and 9 10 but also trying to find the right tool—ontology editor, rangeIncludes (as alternatives for the rdfs:domain 10 11 visualization, reasoner— and workflow/development and rdfs:range) for pragmatic reasons, so that it 11 12 process, while the resources for making an informed is easier to encode multiple values for the domain and 12 13 decision are scarce. ranges of properties. ShACL and ShEx, while being 13 14 Many newcomers to semantic technologies start developed to validate graph-based data against a set 14 15 with simpler modeling languages, such as SKOS, of conditions, is seen by some in the community as a 15 16 for defining their vocabularies, but then struggle to simpler knowledge representation language that might 16 17 upgrade their model into a more expressive lan- one day replace OWL. Many commercial companies 17 18 guage, such as OWL, as there is no straightfor- are choosing labeled property graphs to encode their 18 19 ward path between the two languages [69]. Simi- knowledge graphs rather than RDF triplestores as the 19 20 larly, many application developers start with other property graphs are perceived to be simpler to under- 20 21 types of representations—UML diagrams, mind maps, stand, and usually offer better query performance. 21 22 XML Schemas, spreadsheets—which then need to be One key aspect of ontology engineering is reuse, 22 23 converted into OWL. Existing conversion algorithms both of ontologies and of parts of ontologies. Even 23 24 provide mostly structural transformations based on though, several ontology repositories exist [72]— Bio- 24 25 apriori-defined mapping rules. The conversion results Portal [73] for biomedicine, AgroPortal [74] for agri- 25 26 often need to be further processed manually to try culture, or general-purpose repositories, such as the 26 27 to capture the intended semantics of the data source. Linked Open Vocabularies (LOV) [75] and Onto- 27 28 Although the bootstrapping of ontologies represents hub [76]—finding the right ontology for a particular 28 29 a crucial aspect of the ontology development pro- task is still difficult. The testament to this challenge 29 30 cess, and is also the first encounter of newcomers to are the countless postings on the Semantic Web mail- 30 31 ontologies—often a make or break issue—it is not as ing lists from interested users trying to find an ontol- 31 32 well researched as other related areas, and has less ogy for a particular domain or task. A common sce- 32 33 methodological and tool support. Simplifying and bet- nario is finding an ontology that can be used to anno- 33 34 ter supporting the initial phases of ontology develop- tate a corpus of text. The BioPortal Ontology Recom- 34 35 ment would encourage a wider adoption of ontologies. mender34 [77] uses several criteria to make ontology 35 36 The inherent complexity of OWL also makes it chal- recommendations for the biomedical domain, however 36 37 lenging to build developer-friendly APIs for access- a general-purpose recommender is not available. One 37 38 ing and handling OWL ontologies. The most compre- of the bigger research challenges in finding the “right” 38 39 hensive Java API for OWL 2.0 ontologies, the OWL- ontology is coming up with a set of metrics that can 39 40 API [55], requires good knowledge of the OWL spec- measure how suitable an ontology is for a task or a 40 41 ification and it can be intimidating for developers. At domain—a topic that has been one of the main sub- 41 42 the same time, developer-friendly approaches, such as jects of ontology evaluation for a long time, but it is 42 31 43 JSON-LD —a lightweight syntax to serialize Linked still not solved. While a review of current ontology 43 32 44 Data in JSON, or GraphQL —a highly popular data evaluation methods is not covered here, the reader can 44 45 query and manipulation language—if used properly, refer to the review by Sabou and Fernandez for fur- 45 46 could bolster the adoption of ontologies. There are al- ther details [78], and to a more recent description of 46 47 ready several academic and commercial approaches the state of the art in ontology evaluation by Poveda 47 48 that use GraphQL to query RDF graphs. Even so, more Villalon [79]. 48 49 49 50 31https://json-ld.org/ 33https://schema.org/docs/datamodel.html 50 51 32https://graphql.org/ 34https://bioportal.bioontology.org/recommender 51 8 T. Tudorache / Ontology Engineering

1 Reusing parts of an ontology—single entities, sub- plexity of the languages and tooling, as we discussed 1 2 trees, or sets of axioms—is often necessary during on- in the previous section. There is a stringent need to 2 3 tology development, however enacting reuse in prac- simplify, if not the actual Semantic Web languages and 3 4 tice is difficult. Several published studies have shown standards, but at least the presentation and interaction 4 5 that the level of ontological reuse is low [80–82]. For of the users with the Semantic Web languages and tool- 5 6 example, in a recent study, Kamdar et. al [81] show ing. The Semantic Web community can benefit signif- 6 7 that the term reuse is less than 9% in biomedical on- icantly from a tighter collaboration with the Human 7 8 tologies, even though the term overlap is between 25– Computer Interaction (HCI) communities and from ap- 8 9 31%, with most ontologies reusing fewer than 5% of plying HCI techniques in the design of their tooling. 9 10 their terms from a small set of popular ontologies. The Two encouraging, albeit rare examples of such collab- 10 11 challenges related to reuse come for a wide range: orations, are the paper by Fu et. al [89] that investi- 11 12 from finding the ontology to reuse, to extracting the gates ontology visualization techniques in the context 12 13 subset to reuse (although several module-extraction al- of class mappings and the paper by Vigo et. al [68] that 13 14 gorithms exist [83]), to maintaining the extracted sub- provides design guidelines for ontology editors. Dis- 14 15 set as the source ontology evolves. playing class hierarchies, which are so central to on- 15 16 Due to space limitations, the challenges described in tology development but are still cumbersome to use, is 16 17 this section do not represent a comprehensive listing of another area where HCI already provides several solu- 17 18 challenges in ontology engineering, but rather repre- tions [90]. 18 19 sent issues that have been encountered all too often by At the same time, the user interfaces for eliciting the 19 20 the author and her collaborators. The interested reader content of an ontology have not evolved much. Cur- 20 21 can learn more about the current state and challenges rent ontology editors, including Protégé, offer an in- 21 22 in using ontology design pattern from Blomqvist et. timidating view of all features that OWL offers. It is 22 23 al [84], about the state of ontology evolution from no wonder that newcomers are scared away. We need 23 24 Zablith et. al [85], and about ontology matching from role-based user interfaces that would enable users with 24 25 Otero-Cerdeira et. al [86]. Shaviko et. al [87] cover the different expertise to contribute effectively. These user 25 26 state and challenges of ontology learning, while a more interfaces could be automatically generated based on a 26 27 introductory and comprehensive view can be found in user profile, and at the same time the interfaces need 27 28 the book “Perspectives on Ontology Learning” edited to enforce certain editing rules (e.g., all classes need to 28 29 by Lehmann and Völker [88]. have a rdfs:label) that are paramount in any real- 29 30 world ontology development project. 30 31 Another opportunity for simplifying the presenta- 31 32 6. Opportunities for Future Research tion and consumption of ontologies is to continue the 32 33 research on ontology summarization. Although a few 33 34 Even though the research topics in Semantic Web summarization approaches already exist [91, 92], there 34 35 have evolved in the last decade, ontologies and their are still several open issues. Most notably it is not clear 35 36 engineering were always present among them. In Fig- what is the best way to evaluate such approaches. Fu- 36 37 ure 2, we generated the word clouds from the titles of ture directions in ontology summarization include the 37 38 the accepted papers at the International Semantic Web customization of the summarization by allowing the 38 39 Conference (ISWC) in three different years of the last user to tune the model to generate different summaries 39 40 decade, which arguably represent a fairly accurate im- based on different requirements, as well as, using non- 40 41 age of the research topics at the time. An investigation extractive techniques, in which the summary is not us- 41 42 of the topics in the call for papers for larger Seman- ing necessarily terms extracted from the ontology [91]. 42 43 tic Web conferences also confirms that ontologies and Effective summarization techniques could have a great 43 44 their engineering have always been present in the last impact in helping disseminate ontologies to other com- 44 45 decade. On one hand, this constancy points to how im- munities, and to help our community better find and 45 46 portant ontologies are for the Semantic Web, and on reuse ontologies. 46 47 the other hand, it is a sign that ontology engineering is Knowledge graphs are gaining in popularity, and 47 48 still an active research topic that needs to evolve sig- they will likely become even more widespread in the 48 49 nificantly in the next decade. near future. Knowledge graphs can be seen both as a 49 50 One of main show stoppers for a wider adoption of challenge and as an opportunity with respect to ontolo- 50 51 ontologies is their steep learning curve and the com- gies. Knowledge graphs are now widely adopted in in- 51 T. Tudorache / Ontology Engineering 9

1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 11 11 12 Fig. 2. The word clouds generated from the titles of the papers accepted at the International Semantic Web Conference (ISWC) in the last decade, 12 which are representative for the research topics in the Semantic Web in the respective years. 13 13 14 14 35 15 dustry, however many of these KGs do not use RDF, ity co-located with ISWC 2019, and a similar session 15 16 triplestores or semantics, but rather ML techniques for which took place at the 2nd US Semantic Technologies 16 36 17 automatic building and property graphs or other graph Symposium (US2TS 2019). Explainable AI is try- 17 18 stores for storing. Now it is a turning point, in which ing to produce techniques that will allow human users 18 19 ontologies and semantics may become important, if to understand how complex AI and ML-based systems 19 20 in the short-term, new research and methods will be reach a decision, i.e., to find an explanation of a deci- 20 21 able to bridge the gap between property graphs repre- sion, to help us produce more explainable models, and 21 37 22 sentation and RDF, and between the different graph- to enable us to debug the decisions. Ontologies have 22 23 database query languages and SPARQL, making se- the potential of playing an important role into making 23 24 mantics more accessible to developers. One active area AI systems explainable as they already provide a user’s 24 25 of research is the knowledge graph embeddings [93] in conceptualization of a domain, which could be used as 25 26 which KG entities and relationships are embedded into part of the explanation or debugging process. However, 26 27 continuous vector spaces, which can then be further exactly how to achieve this is a difficult and open issue. 27 28 used to perform efficiently many types of tasks, such as We will need new design patterns (some initial work 28 29 knowledge graph completion—finding missing triples already exists [96]) and even new methodologies for 29 building ontologies that can support explainable sys- 30 in the graph; relation extraction—extracting relations 30 tems. We will need to define the interplay of ontologies 31 from text using a KG; entity resolution—verifying 31 with the different AI techniques, such as deep learn- 32 that two entities refer to the same object; or question 32 ing methods. There have already been some efforts in 33 answering—trying to answer a natural language ques- 33 this direction, referred to as neural-symbolic integra- 34 tion using the information in a KG. However most 34 tion [97], but it is still a new field of investigation that 35 of these approaches work on the data/instance level. 35 36 if successful, will have a high impact on society. 36 The ontology engineering field could benefit greatly 37 37 by adapting some of these techniques to ontology- 38 38 specific tasks, such as, ontology learning and ontology 39 7. Conclusions 39 matching. There are already promising approaches, 40 40 with OWL2Vec* that computes embeddings for OWL 41 The goal of the paper is to give an overview of 41 ontologies [94], or the DeepAlignment that performs 42 how the field of ontology engineering has evolved in 42 unsupervised ontology matching uses pre-trained word 43 the last decade. Due to space limitations, we could 43 44 vectors to derive ontological entity descriptions tai- only cover some of the main topics in ontology en- 44 45 lored to the ontology matching task, and obtain signif- gineering. We hope that the paper can serve as an 45 46 icantly better results than the state-of-the-art matching entry point for a newcomer in the ontology field, 46 47 approaches [95]. 47 48 Another area that has a lot of growth now is Ex- 48 35http://www.semantic-explainability.com/ 49 plainable AI, and more recently, the intersection of Ex- 49 36https://semanticsforxai.github.io/ 50 plainable AI with semantics and ontologies, as demon- 37https://www.darpa.mil/program/ 50 51 strated also by the Workshop on Semantic Explainabil- explainable-artificial-intelligence 51 10 T. Tudorache / Ontology Engineering

1 and as a quick reference for the more knowledge- P.T. Groth, C. Biemann, J.X. Parreira, L. Aroyo, N.F. Noy, 1 2 able researchers. As a result of the research and de- C. Welty and K. Janowicz, eds, Lecture Notes in Computer 2 3 velopment efforts in the last ten years, ontologies Science, Vol. 8219, Springer, 2013, pp. 195–211. ISBN 978- 3 3-642-41337-7. doi:10.1007/978-3-642-41338-4_13. 4 4 are now adopted in wide range of domains, from [9] M. Bennett, The financial industry business ontology: Best 5 biomedicine to engineering and finance. The infras- practice for big data, Journal of Banking Regulation 14(3–4) 5 6 tructures for storing, finding and building ontologies (2013), 255–268. doi:10.1057/jbr.2013.1. 6 7 have also evolved significantly. Several standards per- [10] W. Borst, Construction of engineering ontology, Ph. D. Dis- 7 8 taining to ontology engineering have been adopted sertation, Institute for Telematica and Information Technol- 8 ogy, University of Twente, Enschede, the Netherlands (1997). 9 9 in the last decade, and highly-visible efforts to build [11] T.R. Gruber and G.R. Olsen, An ontology for engineering 10 large-scale ontologies and knowledge bases are well mathematics, in: Principles of Knowledge Representation and 10 11 underway. Even though the ontology engineering field Reasoning, Elsevier, 1994, pp. 258–269. 11 12 still faces several challenges—many of them long- [12] S. Feldmann, S. Rösch, C. Legat and B. Vogel-Heuser, 12 13 standing—we have also identified many opportunities Keeping requirements and test cases consistent: Towards an 13 ontology-based approach, in: 12th IEEE International Con- 14 14 for future research and development, and exciting new ference on Industrial Informatics, INDIN 2014, Porto Alegre, 15 opportunities from synergies with other domains that RS, Brazil, July 27-30, 2014, M. Wollschlaeger and J. Barata, 15 16 can drive the ontology engineering field even further. eds, IEEE, 2014, pp. 726–732. ISBN 978-1-4799-4905-2. 16 17 doi:10.1109/INDIN.2014.6945603. 17 18 [13] W. Terkaj and M. Urgo, Ontology-based modeling of 18 production systems for design and performance evalu- 19 8. Acknowledgements 19 ation, in: 12th IEEE International Conference on In- 20 dustrial Informatics, INDIN 2014, Porto Alegre, RS, 20 21 The author would like to thank the reviewers for pro- Brazil, July 27-30, 2014, M. Wollschlaeger and J. Barata, 21 22 viding helpful feedback, and to Jennifer Vendetti and eds, IEEE, 2014, pp. 748–753. ISBN 978-1-4799-4905-2. 22 23 Csongor Nyulas for proof-reading the paper. doi:10.1109/INDIN.2014.6945606. 23 [14] O. Kovalenko, E. Serral, M. Sabou, F.J. Ekaputra, D. Win- 24 24 kler and S. Biffl, Automating Cross-Disciplinary Defect 25 Detection in Multi-disciplinary Engineering Environments, 25 26 References in: Knowledge Engineering and Knowledge Management 26 27 - 19th International Conference, EKAW 2014, Linköping, 27 28 [1] T.R. Gruber, A translation approach to portable ontology Sweden, November 24-28, 2014. Proceedings, K. Janow- 28 specifications., Knowledge Acquisition 5(2) (1993), 199–220. icz, S. Schlobach, P. Lambrix and E. Hyvönen, eds, Lec- 29 29 doi:10.1006/knac.1993.1008. ture Notes in Computer Science, Vol. 8876, Springer, 2014, 30 [2] R. Stevens, A. Rector and D. Hull, What is an ontology?, On- pp. 238–249. ISBN 978-3-319-13703-2. doi:10.1007/978-3- 30 31 togenesis (2010). http://ontogenesis.knowledgeblog.org/66. 319-13704-9_19. 31 32 [3] M. Ashburner, C.A. Ball, J.A. Blake, D. Botstein, H. Butler, [15] S. Biffl and M. Sabou, Semantic web technologies for intel- 32 33 J.M. Cherry, A.P. Davis, K. Dolinski, S.S. Dwight, J.T. Ep- ligent engineering applications, Springer, Cham, 2016. ISBN 33 pig et al., Gene ontology: tool for the unification of biology, 978-3-319-41488-1. doi:10.1007/978-3-319-41490-4. 34 34 Nature genetics 25(1) (2000), 25. doi:10.1038/75556. [16] E. Hyvönen, Publishing and Using Cultural Heritage Linked 35 [4] P. Khatri, M. Sirota and A.J. Butte, Ten years of path- Data on the Semantic Web, Synthesis Lectures on the 35 36 way analysis: current approaches and outstanding chal- Semantic Web, Morgan & Claypool Publishers, 2012. 36 37 lenges, PLoS computational biology 8(2) (2012), e1002375. doi:10.2200/S00452ED1V01Y201210WBE003. 37 38 doi:10.1371/journal.pcbi.1002375. [17] A. Isaac and B. Haslhofer, Europeana Linked Open Data 38 [5] R. Hoehndorf, P.N. Schofield and G.V. Gkoutos, The role of - data.europeana.eu, Semantic Web 4(3) (2013), 291–297. 39 39 ontologies in biological and biomedical research: a functional doi:10.3233/SW-120092. 40 perspective, Briefings in bioinformatics 16(6) (2015), 1069– [18] S. Peroni and D.M. Shotton, The SPAR Ontologies, in: 40 41 1080. doi:10.1093/bib/bbv011. The Semantic Web - ISWC 2018 - 17th International Se- 41 42 [6] M. Ivanovic´ and Z. Budimac, An overview of on- mantic Web Conference, Monterey, CA, USA, October 8- 42 43 tologies and data resources in medical domains, Ex- 12, 2018, Proceedings, Part II, D. Vrandecic, K. Bontcheva, 43 pert Systems with Applications 41(11) (2014), 5158–5166. M.C. Suárez-Figueroa, V. Presutti, I. Celino, M. Sabou, 44 44 doi:10.1016/j.eswa.2014.02.045. L. Kaffee and E. Simperl, eds, Lecture Notes in Computer 45 [7] W.H. Organization, International Classification of Diseases Science, Vol. 11137, Springer, 2018, pp. 119–136. ISBN 978- 45 46 (ICD) Information Sheet, 2019, [Online; accessed 05-May- 3-030-00667-9. doi:10.1007/978-3-030-00668-6_8. 46 47 2019]. https://www.who.int/classifications/icd/factsheet/en/. [19] V.A. Carriero, A. Gangemi, M.L. Mancinelli, L. Marinucci, 47 48 [8] T. Tudorache, C. Nyulas, N.F. Noy and M.A. Musen, Using A.G. Nuzzolese, V. Presutti and C. Veninata, ArCo: The Ital- 48 Semantic Web in ICD-11: Three Years Down the Road, in: ian Cultural Heritage Knowledge Graph, in: The Semantic 49 49 The Semantic Web - ISWC 2013 - 12th International Seman- Web - ISWC 2019 - 18th International Semantic Web Con- 50 tic Web Conference, Sydney, NSW, Australia, October 21-25, ference, Auckland, New Zealand, October 26-30, 2019, Pro- 50 51 2013, Proceedings, Part II, H. Alani, L. Kagal, A. Fokoue, ceedings, Part II, C. Ghidini, O. Hartig, M. Maleshkova, 51 T. Tudorache / Ontology Engineering 11

1 V. Svátek, I.F. Cruz, A. Hogan, J. Song, M. Lefrançois [34] T. Tudorache, S.M. Falconer, N.F. Noy, C. Nyulas, 1 2 and F. Gandon, eds, Lecture Notes in Computer Science, T.B. Üstün, M.D. Storey and M.A. Musen, Ontology De- 2 3 Vol. 11779, Springer, 2019, pp. 36–52. ISBN 978-3-030- velopment for the Masses: Creating ICD-11 in WebPro- 3 30795-0. doi:10.1007/978-3-030-30796-7_3. tégé, in: Knowledge Engineering and Management by the 4 4 [20] International Organization for Standardization (ISO), ISO Masses - 17th International Conference, EKAW 2010, Lis- 5 21127:2014 Information and documentation - A reference on- bon, Portugal, October 11-15, 2010. Proceedings, P. Cimi- 5 6 tology for the interchange of cultural heritage information, ano and H.S. Pinto, eds, Lecture Notes in Computer Sci- 6 7 2014, [Online; accessed 08-Oct-2019]. https://www.iso.org/ ence, Vol. 6317, Springer, 2010, pp. 74–89. ISBN 978-3-642- 7 8 standard/57832.html. 16437-8. doi:10.1007/978-3-642-16438-5_6. 8 [21] C.M. de Oliveira Rodrigues, F.L.G. de Freitas, E.F.S. Bar- 9 [35] E. Simperl and M. Luczak-Rösch, Collaborative ontology en- 9 reiros, R.R. de Azevedo and A.T. de Almeida Filho, Legal on- gineering: a survey, Knowledge Eng. Review 29(1) (2014), 10 10 tologies over time: A systematic mapping study, Expert Syst. 101–131. doi:10.1017/S0269888913000192. 11 Appl. 130 (2019), 12–30. doi:10.1016/j.eswa.2019.04.009. [36] N.F. Noy, D.L. McGuinness et al., Ontology devel- 11 12 [22] International Organization for Standardization (ISO), ISO/TS opment 101: A guide to creating your first ontol- 12 13 19150-1:2012 Geographic information - Ontology - Part 1: ogy, Stanford knowledge systems laboratory techni- 13 Framework, 2012, [Online; accessed 08-Oct-2019]. https:// 14 cal report KSL-01-05, 2001, [Online: accessed 22-Oct- 14 www.iso.org/standard/57465.html. 2019]. http://www.ksl.stanford.edu/people/dlm/papers/ 15 15 [23] International Organization for Standardization (ISO), ISO ontology-tutorial-noy-mcguinness-abstract.html. 16 19150-2:2015 Geographic information - Ontology - Part 2: [37] M.C. Suárez-Figueroa, A. Gómez-Pérez and M. Fernández- 16 17 Rules for developing ontologies in the Web Ontology Lan- López, The NeOn Methodology for Ontology Engineer- 17 18 guage (OWL), 2015, [Online; accessed 08-Oct-2019]. https: ing, in: Ontology Engineering in a Networked World., 18 //www.iso.org/standard/57466.html. 19 M.C. Suárez-Figueroa, A. Gómez-Pérez, E. Motta and 19 [24] World Wide Web Consortium (WWW), Time Ontology in 20 A. Gangemi, eds, Springer, 2012, pp. 9–34. ISBN 978-3-642- 20 OWL, W3C Recommendation, 2017, [Online; accessed 05- 24793-4. doi:10.1007/978-3-642-24794-1_2. 21 May-2019]. https://www.w3.org/TR/owl-time/. 21 [38] A.D. Nicola and M. Missikoff, A lightweight methodology [25] World Wide Web Consortium (WWW), Semantic Sensor Net- 22 for rapid ontology engineering, Commun. ACM 59(3) (2016), 22 work Ontology, W3C Recommendation, 2017, [Online; ac- 23 79–86. doi:10.1145/2818359. 23 cessed 05-May-2019]. https://www.w3.org/TR/vocab-ssn/. 24 [39] E. Blomqvist, V. Presutti, E. Daga and A. Gangemi, Ex- 24 [26] World Wide Web Consortium (WWW), PROV-O: The PROV perimenting with eXtreme Design, in: Knowledge Engineer- 25 Ontology, W3C Recommendation, 2013, [Online; accessed 25 ing and Management by the Masses - 17th International 26 05-May-2019]. https://www.w3.org/TR/prov-o/. 26 Conference, EKAW 2010, Lisbon, Portugal, October 11-15, [27] World Wide Web Consortium (WWW), The RDF Data 27 2010. Proceedings, P. Cimiano and H.S. Pinto, eds, Lec- 27 Cube Vocabulary, W3C Recommendation, 2014, [On- 28 ture Notes in Computer Science, Vol. 6317, Springer, 2010, 28 line; accessed 05-May-2019]. https://www.w3.org/TR/2014/ 29 pp. 120–134. ISBN 978-3-642-16437-8. doi:10.1007/978-3- 29 REC-vocab-data-cube-20140116/. 642-16438-5_9. 30 [28] J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, 30 [40] Gene Ontology Consortium, Gene Ontology Consortium: 31 P.N. Mendes, S. Hellmann, M. Morsey, P. van Kleef, S. Auer 31 going forward, Nucleic Acids Research 43(Database–Issue) 32 and C. Bizer, DBpedia - A large-scale, multilingual knowl- 32 (2015), 1049–1056. doi:10.1093/nar/gku1179. 33 edge base extracted from Wikipedia, Semantic Web 6(2) 33 (2015), 167–195. doi:10.3233/SW-140134. [41] C. Mungall, H. Dietze and D. Osumi-Sutherland, Use of 34 34 [29] F.M. Suchanek, G. Kasneci and G. Weikum, Yago: a core OWL within the Gene Ontology, in: Proceedings of the 11th 35 of semantic knowledge, in: Proceedings of the 16th In- International Workshop on OWL: Experiences and Directions 35 36 ternational Conference on World Wide Web, WWW 2007, (OWLED 2014) co-located with 13th International Seman- 36 tic Web Conference on (ISWC 2014), Riva del Garda, Italy, 37 Banff, Alberta, Canada, May 8-12, 2007, C.L. Williamson, 37 October 17-18, 2014., C.M. Keet and V.A.M. Tamma, eds, 38 M.E. Zurko, P.F. Patel-Schneider and P.J. Shenoy, eds, 38 ACM, 2007, pp. 697–706. ISBN 978-1-59593-654-7. CEUR Workshop Proceedings, Vol. 1265, CEUR-WS.org, 39 39 doi:10.1145/1242572.1242667. 2014, pp. 25–36. http://ceur-ws.org/Vol-1265/owled2014_ 40 [30] G.A. Miller, WordNet: A Lexical Database for submission_5.pdf. 40 41 English, Commun. ACM 38(11) (1995), 39–41. [42] B. Smith, M. Ashburner, C. Rosse, J. Bard, W. Bug, 41 42 doi:10.1145/219717.219748. W. Ceusters, L.J. Goldberg, K. Eilbeck, A. Ireland, 42 C.J. Mungall et al., The OBO Foundry: coordinated evolution 43 [31] J. Hoffart, F.M. Suchanek, K. Berberich and G. Weikum, 43 YAGO2: A spatially and temporally enhanced knowledge of ontologies to support biomedical data integration, Nature 44 44 base from Wikipedia, Artif. Intell. 194 (2013), 28–61. biotechnology 25(11) (2007), 1251. doi:10.1038/nbt1346. 45 doi:10.1016/j.artint.2012.06.001. [43] C.J. Mungall, H. Dietze, S.J. Carbon, A. Ireland, S. Bauer and 45 46 [32] D. Vrandecic and M. Krötzsch, Wikidata: a free collabora- S. Lewis, Continuous integration of open biological ontology 46 47 tive knowledgebase, Commun. ACM 57(10) (2014), 78–85. libraries, in: Proceedings of the 15th Annual Bio-Ontologies 47 Meeting, July 13-14th, 2012, Co-located with ISMB 2012, 48 doi:10.1145/2629489. 48 [33] Gartner, Gartner Identifies Five Emerging Technology Trends Long Beach, CA, USA, 2012. 49 49 That Will Blur the Lines Between Human and Machine, [44] H. Dietze, T.Z. Berardini, R.E. Foulger, D.P. Hill, J. Lo- 50 2018, [Online; accessed 05-May-2019]. https://tinyurl.com/ max, D. Osumi-Sutherland, P. Roncaglia and C.J. Mungall, 50 51 yxbxdka3. TermGenie–a web-application for pattern-based ontology 51 12 T. Tudorache / Ontology Engineering

1 class generation, Journal of biomedical semantics 5(1) and J. Baumeister, eds, Advances in Intelligent Sys- 1 2 (2014), 48. doi:10.1186/2041-1480-5-48. tems and Computing, Vol. 626, Springer, 2018, pp. 251– 2 3 [45] M. Uschold, Demystifying OWL for the En- 273, doi:10.1007/978-3-319-64161-4_12. ISBN 978-3-319- 3 terprise, Synthesis Lectures on Semantic Web: 64160-7. doi:10.1007/978-3-319-64161-4_12. 4 4 Theory and Technology 8(1) (2018), i–237. [57] M.A. Musen, The protégé project: a look back 5 doi:10.2200/S00824ED1V01Y201801WBE017. and a look forward, AI Matters 1(4) (2015), 4–12. 5 6 [46] E.F. Kendall and D. McGuiness, Ontology En- doi:10.1145/2757001.2757003. 6 7 gineering, Synthesis Lectures on Semantic Web: [58] M. Horridge, R.S. Gonçalves, C.I. Nyulas, T. Tudorache and 7 8 Theory and Technology 18(1) (2019), i–102. M.A. Musen, WebProtégé: A Cloud-Based Ontology Edi- 8 doi:10.2200/S00834ED1V01Y201802WBE018. tor, in: Companion of The 2019 World Wide Web Confer- 9 9 [47] M. Keet, An Introduction to Ontology Engineering, 2018, ence, WWW 2019, San Francisco, CA, USA, May 13-17, 10 Accessed online on May 25, 2019. http://www.meteck.org/ 2019., S. Amer-Yahia, M. Mahdian, A. Goel, G. Houben, 10 11 teaching/OEbook/. K. Lerman, J.J. McAuley, R. Baeza-Yates and L. Zia, 11 12 [48] A. Gangemi, Ontology Design Patterns for Semantic Web eds, ACM, 2019, pp. 686–689. ISBN 978-1-4503-6675-5. 12 13 Content, in: The Semantic Web - ISWC 2005, 4th Interna- doi:10.1145/3308560.3317707. 13 tional Semantic Web Conference, ISWC 2005, Galway, Ire- [59] M. Horridge, T. Tudorache, J. Vendetti, C. Nyulas, 14 14 land, November 6-10, 2005, Proceedings, Y. Gil, E. Motta, M.A. Musen and N.F. Noy, Simplified OWL Ontology Edit- 15 V.R. Benjamins and M.A. Musen, eds, Lecture Notes in Com- ing for the Web: Is WebProtégé Enough?, in: The Semantic 15 16 puter Science, Vol. 3729, Springer, 2005, pp. 262–276. ISBN Web - ISWC 2013 - 12th International Semantic Web Con- 16 17 3-540-29754-5. doi:10.1007/11574620_21. ference, Sydney, NSW, Australia, October 21-25, 2013, Pro- 17 18 [49] P. Hitzler, A. Gangemi, K. Janowicz, A. Krisnadhi and V. Pre- ceedings, Part I, H. Alani, L. Kagal, A. Fokoue, P.T. Groth, 18 sutti (eds), Ontology Engineering with Ontology Design Pat- C. Biemann, J.X. Parreira, L. Aroyo, N.F. Noy, C. Welty 19 19 terns - Foundations and Applications, Studies on the Seman- and K. Janowicz, eds, Lecture Notes in Computer Science, 20 tic Web, Vol. 25, IOS Press, 2016. ISBN 978-1-61499-675-0. Vol. 8218, Springer, 2013, pp. 200–215. ISBN 978-3-642- 20 21 [50] P. Hitzler, A. Gangemi, K. Janowicz, A.A. Krisnadhi and 41334-6. doi:10.1007/978-3-642-41335-3_13. 21 22 V. Presutti, Towards a Simple but Useful Ontology Design [60] A. Alobaid, D. Garijo, M. Poveda-Villalón, I. Santana-Pérez, 22 23 Pattern Representation Language, in: Proceedings of the 8th A. Fernández-Izquierdo and Ó. Corcho, Automating ontol- 23 Workshop on Ontology Design and Patterns (WOP 2017) ogy engineering support activities with OnToology, J. Web 24 24 co-located with the 16th International Semantic Web Con- Semant. 57 (2019). doi:10.1016/j.websem.2018.09.003. 25 ference (ISWC 2017), Vienna, Austria, October 21, 2017., [61] D. Garijo, WIDOCO: A Wizard for Documenting Ontolo- 25 26 E. Blomqvist, Ó. Corcho, M. Horridge, D. Carral and gies, in: The Semantic Web - ISWC 2017 - 16th Interna- 26 27 R. Hoekstra, eds, CEUR Workshop Proceedings, Vol. 2043, tional Semantic Web Conference, Vienna, Austria, October 27 28 CEUR-WS.org, 2017. http://ceur-ws.org/Vol-2043/paper-09. 21-25, 2017, Proceedings, Part II, C. d’Amato, M. Fernán- 28 pdf. dez, V.A.M. Tamma, F. Lécué, P. Cudré-Mauroux, J.F. Se- 29 29 [51] D. Osumi-Sutherland, M. Courtot, J.P. Balhoff and queda, C. Lange and J. Heflin, eds, Lecture Notes in Com- 30 C. Mungall, Dead simple OWL design patterns, Journal of puter Science, Vol. 10588, Springer, 2017, pp. 94–102. ISBN 30 31 biomedical semantics 8(1) (2017), 18. doi:10.1186/s13326- 978-3-319-68203-7. doi:10.1007/978-3-319-68204-4_9. 31 32 017-0126-0. [62] M. Poveda-Villalón, A. Gómez-Pérez and M.C. Suárez- 32 33 [52] M. Egana, R. Stevens and E. Antezana, Transforming the Ax- Figueroa, OOPS! (OntOlogy Pitfall Scanner!): An On-line 33 iomisation of Ontologies: The Ontology Pre-Processor Lan- Tool for Ontology Evaluation, Int. J. Semantic Web Inf. Syst. 34 34 guage., OWLED (Spring) 496 (2008). 10(2) (2014), 7–34. doi:10.4018/ijswis.2014040102. 35 [53] P. Lord, The Semantic Web takes Wing: Programming On- [63] A. Stellato, S. Rajbhandari, A. Turbati, M. Fiorelli, C. Carac- 35 36 tologies with Tawny-OWL, in: Proceedings of the 10th In- ciolo, T. Lorenzetti, J. Keizer and M.T. Pazienza, VocBench: 36 37 ternational Workshop on OWL: Experiences and Directions A Web Application for Collaborative Development of Mul- 37 38 (OWLED 2013) co-located with 10th Extended Semantic Web tilingual Thesauri, in: The Semantic Web. Latest Advances 38 Conference (ESWC 2013), Montpellier, France, May 26-27, and New Domains - 12th European Semantic Web Confer- 39 39 2013., M. Rodriguez-Muro, S. Jupp and K. Srinivas, eds, ence, ESWC 2015, Portoroz, Slovenia, May 31 - June 4, 2015. 40 CEUR Workshop Proceedings, Vol. 1080, CEUR-WS.org, Proceedings, F. Gandon, M. Sabou, H. Sack, C. d’Amato, 40 41 2013. http://ceur-ws.org/Vol-1080/owled2013_16.pdf. P. Cudré-Mauroux and A. Zimmermann, eds, Lecture 41 42 [54] M. Horridge and P.F. Patel-Schneider, OWL 2 Web Ontology Notes in Computer Science, Vol. 9088, Springer, 2015, 42 43 Language Manchester Syntax, W3C Working Group Note pp. 38–53. ISBN 978-3-319-18817-1. doi:10.1007/978-3- 43 (2009), [Online accessed on 22-10-2019]. https://www.w3. 319-18818-8_3. 44 44 org/2009/pdf/NOTE-owl2-manchester-syntax-20091027. [64] S. Peroni, D.M. Shotton and F. Vitali, Tools for the Auto- 45 pdf. matic Generation of Ontology Documentation: A Task-Based 45 46 [55] M. Horridge and S. Bechhofer, The OWL API: A Java Evaluation, Int. J. Semantic Web Inf. Syst. 9(1) (2013), 21–44. 46 47 API for OWL ontologies, Semantic Web 2(1) (2011), 11–21. doi:10.4018/jswis.2013010102. 47 48 doi:10.3233/SW-2011-0025. [65] S. Lohmann, V. Link, E. Marbach and S. Negru, WebVOWL: 48 [56] A. Paschke and R. Schäfermeier, OntoMaven - Maven-Based Web-based Visualization of Ontologies, in: Knowledge En- 49 49 Ontology Development and Management of Distributed gineering and Knowledge Management - EKAW 2014 Satel- 50 Ontology Repositories, in: Synergies Between Knowl- lite Events, VISUAL, EKM1, and ARCOE-Logic, Linköping, 50 51 edge Engineering and Software Engineering, G.J. Nalepa Sweden, November 24-28, 2014. Revised Selected Papers., 51 T. Tudorache / Ontology Engineering 13

1 P. Lambrix, E. Hyvönen, E. Blomqvist, V. Presutti, G. Qi, ers and Electronics in Agriculture 144 (2018), 126–143. 1 2 U. Sattler, Y.Ding and C. Ghidini, eds, Lecture Notes in Com- doi:10.1016/j.compag.2017.10.012. 2 3 puter Science, Vol. 8982, Springer, 2014, pp. 154–158. ISBN [75] P. Vandenbussche, G. Atemezing, M. Poveda-Villalón and 3 978-3-319-17965-0. doi:10.1007/978-3-319-17966-7_21. B. Vatant, Linked Open Vocabularies (LOV): A gateway to 4 4 [66] S. Lohmann, S. Negru, F. Haag and T. Ertl, Visualizing on- reusable semantic vocabularies on the Web, Semantic Web 5 tologies with VOWL, Semantic Web 7(4) (2016), 399–419. 8(3) (2017), 437–452. doi:10.3233/SW-160213. 5 6 doi:10.3233/SW-150200. [76] M. Codescu, E. Kuksa, O. Kutz, T. Mossakowski and 6 7 [67] P. Warren, P. Mulholland, T.D. Collins and E. Motta, The F. Neuhaus, Ontohub: A semantic repository engine for het- 7 8 Usability of Description Logics - Understanding the Cog- erogeneous ontologies, 2017, pp. 275–298. doi:10.3233/AO- 8 9 nitive Difficulties Presented by Description Logics, in: The 170190. 9 Semantic Web: Trends and Challenges - 11th International [77] M.M. Romero, C. Jonquet, M.J. O’Connor, J. Graybeal, 10 10 Conference, ESWC 2014, Anissaras, Crete, Greece, May 25- A. Pazos and M.A. Musen, NCBO Ontology Recommender 11 29, 2014. Proceedings, V. Presutti, C. d’Amato, F. Gan- 2.0: an enhanced approach for biomedical ontology recom- 11 12 don, M. d’Aquin, S. Staab and A. Tordai, eds, Lecture mendation, J. Biomedical Semantics 8(1) (2017), 21:1–21:22. 12 13 Notes in Computer Science, Vol. 8465, Springer, 2014, doi:10.1186/s13326-017-0128-y. 13 14 pp. 550–564. ISBN 978-3-319-07442-9. doi:10.1007/978-3- [78] M. Sabou and M. Fernández, Ontology (Network) Eval- 14 319-07443-6_37. uation, in: Ontology Engineering in a Networked World., 15 15 [68] M. Vigo, S. Bail, C. Jay and R.D. Stevens, Overcoming the M.C. Suárez-Figueroa, A. Gómez-Pérez, E. Motta and 16 pitfalls of ontology authoring: Strategies and implications for A. Gangemi, eds, Springer, 2012, pp. 193–212. ISBN 978-3- 16 17 tool design, Int. J. Hum.-Comput. Stud. 72(12) (2014), 835– 642-24793-4. doi:10.1007/978-3-642-24794-1_9. 17 18 845. doi:10.1016/j.ijhcs.2014.07.005. [79] M.P. Villalon, Ontology evaluation: a pitfall-based approach 18 19 [69] S. Jupp, S. Bechhofer and R. Stevens, SKOS with OWL: to ontology diagnosis, PhD thesis, Universidad Politeecnica 19 20 Don’t be Full-ish!, in: Proceedings of the Fifth OWLED de Madrid, 2016. 20 Workshop on OWL: Experiences and Directions, collocated [80] A. Ghazvinian, N.F. Noy and M.A. Musen, How orthog- 21 21 with the 7th International Semantic Web Conference (ISWC- onal are the OBO Foundry ontologies?, J. Biomedical Se- 22 2008), Karlsruhe, Germany, October 26-27, 2008, C. Dol- mantics 2(S–2) (2011), S2. doi:10.1186/2041-1480-2-S2-S2. 22 23 bear, A. Ruttenberg and U. Sattler, eds, CEUR Workshop Pro- http://www.jbiomedsem.com/content/2/S2/S2. 23 24 ceedings, Vol. 432, CEUR-WS.org, 2008. http://ceur-ws.org/ [81] M.R. Kamdar, T. Tudorache and M.A. Musen, A sys- 24 25 Vol-432/owled2008eu_submission_22.pdf. tematic analysis of term reuse and term overlap across 25 [70] R. Taelman, M.V. Sande and R. Verborgh, GraphQL-LD: biomedical ontologies, Semantic Web 8(6) (2017), 853–871. 26 26 Linked Data Querying with GraphQL, in: Proceedings of the doi:10.3233/SW-160238. 27 27 ISWC 2018 Posters & Demonstrations, Industry and Blue [82] M. Poveda Villalón, M.C. Suárez-Figueroa and A. Gómez- 28 Sky Ideas Tracks co-located with 17th International Seman- Pérez, The landscape of ontology reuse in linked data, 28 29 tic Web Conference (ISWC 2018), Monterey, USA, October in: Proceedings of Ontology Engineering in a Data-driven 29 30 8th - to - 12th, 2018., M. van Erp, M. Atre, V. López, World (OEDW 2012), Galway, Ireland, Informatica, 2012, 30 31 K. Srinivas and C. Fortuna, eds, CEUR Workshop Proceed- http://oa.upm.es/14385/. http://oa.upm.es/14385/. 31 ings, Vol. 2180, CEUR-WS.org, 2018. http://ceur-ws.org/ [83] M. d’Aquin, Modularizing Ontologies, in: Ontology 32 32 Vol-2180/paper-65.pdf. Engineering in a Networked World., M.C. Suárez- 33 [71] R. Meusel, P. Petrovski and C. Bizer, The WebDataCommons Figueroa, A. Gómez-Pérez, E. Motta and A. Gangemi, eds, 33 34 Microdata, RDFa and Microformat Dataset Series, in: The Springer, 2012, pp. 213–233. ISBN 978-3-642-24793-4. 34 35 Semantic Web - ISWC 2014 - 13th International Semantic doi:10.1007/978-3-642-24794-1_10. 35 36 Web Conference, Riva del Garda, Italy, October 19-23, 2014. [84] E. Blomqvist, P. Hitzler, K. Janowicz, A. Krisnadhi, 36 Proceedings, Part I, P. Mika, T. Tudorache, A. Bernstein, T. Narock and M. Solanki, Considerations regarding On- 37 37 C. Welty, C.A. Knoblock, D. Vrandecic, P.T. Groth, N.F. Noy, tology Design Patterns, Semantic Web 7(1) (2016), 1–7. 38 38 K. Janowicz and C.A. Goble, eds, Lecture Notes in Computer doi:10.3233/SW-150202. 39 Science, Vol. 8796, Springer, 2014, pp. 277–292. ISBN 978- [85] G. Antoniou, M. d’Aquin, G. Flouris, H. Kondylakis, 39 40 3-319-11963-2. doi:10.1007/978-3-319-11964-9_18. E. Motta, D. Plexousakis and M. Sabou, Ontology evolu- 40 41 [72] M. d’Aquin and N.F. Noy, Where to publish and find on- tion: a process-centric survey, Knowledge Eng. Review 30(1) 41 42 tologies? A survey of ontology libraries, J. Web Semant. 11 (2015), 45–75. doi:10.1017/S0269888913000349. 42 (2012), 96–111. doi:10.1016/j.websem.2011.08.005. [86] L. Otero-Cerdeira, F.J. Rodríguez-Martínez and 43 43 [73] P.L. Whetzel, N.F. Noy, N.H. Shah, P.R. Alexander, C. Nyu- A. Gómez-Rodríguez, Ontology matching: A litera- 44 las, T. Tudorache and M.A. Musen, BioPortal: enhanced func- ture review, Expert Syst. Appl. 42(2) (2015), 949–971. 44 45 tionality via new Web services from the National Center for doi:10.1016/j.eswa.2014.08.032. 45 46 Biomedical Ontology to access and use ontologies in software [87] P. Shvaiko and J. Euzenat, Ontology Matching: State of the 46 47 applications, Nucleic Acids Research 39(Web–Server–Issue) Art and Future Challenges, IEEE Trans. Knowl. Data Eng. 47 (2011), 541–545. doi:10.1093/nar/gkr469. 25 48 (1) (2013), 158–176. doi:10.1109/TKDE.2011.253. 48 [74] C. Jonquet, A. Toulet, E. Arnaud, S. Aubin, E.D.Y. Kaboré, [88] J. Lehmann and J. Völker, Perspectives on Ontology Learn- 49 49 V. Emonet, J. Graybeal, M. Laporte, M.A. Musen, ing, Studies on the Semantic Web, Vol. 18, IOS Press, 2014. 50 V. Pesce and P. Larmande, AgroPortal: A vocabu- ISBN 978-1-61499-378-0. doi:10.3233/978-1-61499-379-7- 50 51 lary and ontology repository for agronomy, Comput- i. 51 14 T. Tudorache / Ontology Engineering

1 [89] B. Fu, N.F. Noy and M.D. Storey, Indented Tree or Graph? Vectors, in: Proceedings of the 2018 Conference of the 1 2 A Usability Study of Ontology Visualization Techniques in North American Chapter of the Association for Computa- 2 3 the Context of Class Mapping Evaluation, in: The Semantic tional Linguistics: Human Language Technologies, NAACL- 3 Web - ISWC 2013 - 12th International Semantic Web Con- HLT 2018, New Orleans, Louisiana, USA, June 1-6, 4 4 ference, Sydney, NSW, Australia, October 21-25, 2013, Pro- 2018, Volume 1 (Long Papers), M.A. Walker, H. Ji and 5 ceedings, Part I, H. Alani, L. Kagal, A. Fokoue, P.T. Groth, A. Stent, eds, Association for Computational Linguistics, 5 6 C. Biemann, J.X. Parreira, L. Aroyo, N.F. Noy, C. Welty 2018, pp. 787–798. ISBN 978-1-948087-27-8. https://www. 6 7 and K. Janowicz, eds, Lecture Notes in Computer Science, aclweb.org/anthology/N18-1072/. 7 8 Vol. 8218, Springer, 2013, pp. 117–134. ISBN 978-3-642- [96] I. Tiddi, M. d’Aquin and E. Motta, An Ontology Design Pat- 8 41334-6. doi:10.1007/978-3-642-41335-3_8. 9 tern to Define Explanations, in: Proceedings of the 8th In- 9 [90] H. Schulz, Treevis.net: A Tree Visualization Reference, IEEE ternational Conference on Knowledge Capture, K-CAP 2015, 10 10 Computer Graphics and Applications 31(6) (2011), 11–15. Palisades, NY, USA, October 7-10, 2015, K. Barker and 11 doi:10.1109/MCG.2011.103. J.M. Gómez-Pérez, eds, ACM, 2015, pp. 3:1–3:8. ISBN 978- 11 12 [91] S. Pouriyeh, M. Allahyari, Q. Liu, G. Cheng, H.R. Arabnia, 1-4503-3849-3. doi:10.1145/2815833.2815844. 12 13 M. Atzori and K. Kochut, Graph-Based Methods for Ontol- [97] T.R. Besold, A.S. d’Avila Garcez, S. Bader, H. Bowman, 13 ogy Summarization: A Survey, in: First IEEE International 14 P.M. Domingos, P. Hitzler, K. Kühnberger, L.C. Lamb, 14 Conference on Artificial Intelligence and Knowledge Engi- D. Lowd, P.M.V. Lima, L. de Penning, G. Pinkas, H. Poon 15 neering, AIKE 2018, Laguna Hills, CA, USA, September 26- 15 and G. Zaverucha, Neural-Symbolic Learning and Reasoning: 16 28, 2018, IEEE Computer Society, 2018, pp. 85–92. ISBN 16 A Survey and Interpretation, CoRR abs/1711.03902 (2017). 978-1-5386-9555-5. doi:10.1109/AIKE.2018.00020. 17 http://arxiv.org/abs/1711.03902. 17 [92] N. Li and E. Motta, Evaluations of User-Driven Ontology 18 [98] P. Cimiano and H.S. Pinto (eds), Knowledge Engineering and 18 Summarization, in: Knowledge Engineering and Manage- 19 19 ment by the Masses - 17th International Conference, EKAW Management by the Masses - 17th International Conference, 20 2010, Lisbon, Portugal, October 11-15, 2010. Proceedings, EKAW 2010, Lisbon, Portugal, October 11-15, 2010. Pro- 20 21 P. Cimiano and H.S. Pinto, eds, Lecture Notes in Computer ceedings, in Lecture Notes in Computer Science, Vol. 6317, 21 Springer, 2010. ISBN 978-3-642-16437-8. doi:10.1007/978- 22 Science, Vol. 6317, Springer, 2010, pp. 544–553. ISBN 978- 22 3-642-16438-5. 23 3-642-16437-8. doi:10.1007/978-3-642-16438-5_44. 23 [93] Q. Wang, Z. Mao, B. Wang and L. Guo, Knowledge Graph [99] M. Wollschlaeger and J. Barata (eds), 12th IEEE Interna- 24 24 Embedding: A Survey of Approaches and Applications, tional Conference on Industrial Informatics, INDIN 2014, 25 IEEE Trans. Knowl. Data Eng. 29(12) (2017), 2724–2743. Porto Alegre, RS, Brazil, July 27-30, 2014, IEEE, 2014. ISBN 25 26 doi:10.1109/TKDE.2017.2754499. 978-1-4799-4905-2. https://ieeexplore.ieee.org/xpl/conhome/ 26 27 [94] O.M. Holter, E.B. Myklebust, J. Chen and E. Jiménez- 6926648/proceeding. 27 [100] M.C. Suárez-Figueroa, A. Gómez-Pérez, E. Motta and 28 Ruiz, Embedding OWL Ontologies with OWL2Vec, in: 28 Proceedings of the ISWC 2019 Satellite Tracks (Posters A. Gangemi (eds), Ontology Engineering in a Net- 29 29 & Demonstrations, Industry, and Outrageous Ideas) co- worked World, Springer, 2012. ISBN 978-3-642-24793-4. 30 located with 18th International Semantic Web Conference doi:10.1007/978-3-642-24794-1. 30 31 (ISWC 2019), Auckland, New Zealand, October 26-30, 2019., [101] H. Alani, L. Kagal, A. Fokoue, P.T. Groth, C. Biemann, 31 32 M.C. Suárez-Figueroa, G. Cheng, A.L. Gentile, C. Guéret, J.X. Parreira, L. Aroyo, N.F. Noy, C. Welty and K. Janowicz 32 33 C.M. Keet and A. Bernstein, eds, CEUR Workshop Pro- (eds), The Semantic Web - ISWC 2013 - 12th International 33 ceedings, Vol. 2456, CEUR-WS.org, 2019, pp. 33–36. http: Semantic Web Conference, Sydney, NSW, Australia, October 34 34 //ceur-ws.org/Vol-2456/paper9.pdf. 21-25, 2013, Proceedings, Part I, in Lecture Notes in Com- 35 35 [95] P. Kolyvakis, A. Kalousis and D. Kiritsis, DeepAlign- puter Science, Vol. 8218, Springer, 2013. ISBN 978-3-642- 36 ment: Unsupervised Ontology Matching with Refined Word 41334-6. doi:10.1007/978-3-642-41335-3. 36 37 37 38 38 39 39 40 40 41 41 42 42 43 43 44 44 45 45 46 46 47 47 48 48 49 49 50 50 51 51