Human Computation and Crowdsourcing Meet the Semantic Web: a Survey

Undeﬁned 1 (2009) 1–5 1 IOS Press

Human Computation and Crowdsourcing meet the Semantic Web: A Survey

Editor(s): Name Surname, University, Country Solicited review(s): Name Surname, University, Country Open review(s): Name Surname, University, Country

Amna Basharat a,∗, I. Budak Arpinar a and Khaled Rasheed a a Department of Computer Science, University of Georgia, Athens, GA, USA E-mail: {amnabash,budak, khaled}@uga.edu

Abstract. Challenges associated with large-scale adoption of semantic web technologies continue to confront the researchers in the ﬁeld. Researchers have recognized the need for human intelligence in the process of semantic content creation and analytics, which forms the backbone of any semantic application. Realizing the potential that human computation, collective intelligence and the ﬁelds of the like such as crowdsourcing and social computation have offered, semantic web researchers have effectively taken up the synergy to solve the bottlenecks of human experts and the needed human contribution in the semantic web development processes. In this paper, we present a comprehensive survey of the intersection of semantic web and the human computation paradigm. We adopt a two fold approach towards understanding this intersection. As the primary focus, we analyze how the semantic web domain has adopted the dimensions of human computation to solve the inherent problems. We present an in-depth analysis of the need for human computation in semantic web tasks such as ontology engineering and linked data management. We provide a ’collective intelligence genome’ adapted for the semantic web as means to analyze the threads of composing semantic web applications using human computation methods. As a secondary contribution we also analyze existing research efforts through which the human computation domain has been better served with the use of semantic technologies. We present a comprehensive view of the promises and challenges offered by the successful synergy of semantic web and human computation. In conclusion, we discuss several key outstanding challenges and propose some open research directions.

Keywords: semantic web, ontologies, crowdsourcing, human computation

1. Introduction tion capabilities in an effort to solve the inherent problems. Semantic web visionists have put forth a number 1.1. Research Context of visionary ideas for the road ahead for the success of the semantic web. The notion of ’The Global Brain After more than a decade of semantic web research, Semantic Web’ [10] - a semantic Web interleaving a researchers remain challenged by the large scale adop- large number of human and machine computation - has tion of the semantic technologies. Semantic technolo- come to be seen as a vision with great potential to over- gies have been deployed in the context of a wide range come some of the issues of the current semantic web of information management tasks, for which machine- . This idea of interleaved human-machine computa- driven algorithmic techniques aiming at full automation has already resulted in successful systems that are tion do not reach a level of accuracy and reliability to able to solve problems in manner and ways unthink- ensure usable systems. Researchers have started aug- able for either computers or machines to be able to menting automatic techniques with human computa- solve alone. The domain of human computation, collective intelligence, social computing and crowdsourc- *Corresponding author. E-mail: [email protected]. ing have all contributed to this successful synergy of

0000-0000/09/$00.00 © 2009 – IOS Press and the authors. All rights reserved 2 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey humans and machines and contribute to the constantly mantic web and human compution will bear its own set evolving metaphor of the ’Global Brain’ [36]. Much of challenges. Both semantic web and human compu- like the concept of the programming needed for the tation seem to have a long way to go before being fully ’Global Brain’ [11], the ’Global Brain Semantic Web’ able to reap the benefits promised by the intersection will need new strands of programming, workflows and with the other [23]. challenges to be accomplished. The challenge for the semantic web community, is 1.2. Contributions to rethink the original semantic web vision, which was largely built on the vision of computers populating the The first contribution of this paper is to provide a web of machines [10]. Researchers have recognized review of the challenges that the semantic web domain the need for human intelligence in the process of se- has faced especially in terms of the need for human in- mantic content creation [98] which forms the back- tervention. Secondly, we analyze the intersection of se- bone of any semantic application. The entrance bar- mantic web and the human computation paradigm. We rier for many semantic applications is said to be high, adopt a two fold approach towards understanding this given the dependence on expertise in knowledge engi- intersection. As the primary focus, we analyze how neering, logics and more. In short, semantic web lacks the semantic web domain has adopted the dimensions the sufficient user involvement in various aspects. Hu- of human computation to solve the inherent problems. mans are simply considered indispensable[23] for the We present a review of studies and applications span- semantic web to realize its full potential. ning the two most common genres in human compu- Realizing the potential that human computation, tation namely Games With A Purpose (GWAP) and collective intelligence and the fields of the like such Micro-Task Crowdsourcing. We also adopt a classifi- as crowdsourcing and social computation have offered, cation scheme in the form of ’collective intelligence semantic web researchers have attempted to effectively genome’ and apply it to some of the key studies and taken up the synergy to solve the bottlenecks of hu- approaches that combine semantic web, human com- man experts and the needed human contribution in se- putation and crowdsourcing. This genome is adapted mantic web development processes. Semantic web re- from the original collective intelligence genome pro- search can be seen as experiencing a shift from increas- posed by [60]. We apply it specifically to the context ingly expert driven to one embracing the larger community and the users involved in the semantic content of the semantic web with the aim to provide useful in- creation process. Some early efforts that led to the evo- sights in analyzing the various threads that constitute lution of this approach includes myOntology [99] and the design of studies for creating semantic web driven inPho [75]. by human computation. These threads are considered Two major genres of research may be seen emerg- useful for possible further investigation to be taken up ing in the last few years, in an attempt to bring human be researchers. At the same time, it is also meant to computation methods to the semantic web: 1) Mech- serve as means of analyzing the strengths and weak- anized Labour and 2) Games with a Purpose for the nesses of existing researches. Semantic Web. In this paper, we not only take a de- Recent research in crowdsourcing and semantic web tailed look at the challenges leading to the adoption of has also seen the emergence of some workflow sys- the human computation methods for the semantic web, tems designed to meet the need of providing a generic we also provide a comprehensive coverage of the ap- framework for automating human-machine computa- proaches in these mentioned genres. On a parallel note, tion workflows. We undertake a comparative analysis human computation systems can also potentially bene- of few of the most prominent studies to this end, and fit from the promises offered by the semantic web. The highlight the essential dimensions constituting these next generation of human computation systems are be- workflow systems. We give special mention to these ing envisioned that go beyond the platform they were systems for they seem closer to the directions that we built on, offering data reusability in ways unintended expect to see in the future that awaits this emerging by their creators [23]. Semantic web may be seen as research domain. means of providing better user continuity and platform As a secondary contribution, we also analyze exist- consistency across human computation systems. ing research efforts through which the human compu- While the potential is clearly evident in going about tation domain has been better served with the use of such a synergy, effectively realizing the synergy of se- semantic technologies. Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 3

We also present a comprehensive view of the promises highlighting the distinctions and similarities based on and challenges offered by the successful synergy of several factors such as motivation, quality control, ag- semantic web and human computation. gregation, process order and task request cardinality We hope that this review will serve as basis for ex- [81]. ploring newer threads of synergy between the seman- What is Collective Intelligence? Collective intelli- tic web and human computation research, resulting in gence is an encompassing term to broadly refer to the creation of better applications and approaches that groups of individuals doing things collectively that advance both domains. seem intelligent [60,61]. This idea of loosely orga- nized group of people accomplishing more than what individuals can alone has been garnering a lot of inter- 2. Background est within the research community. What is Social Computing? Technologies such as In this section, we provide a short introduction on blogs, wikis, and online communities are examples of some aspects of the theoretical foundations and basics social computing [81]. The scope is broad, but always of human computation and crowdsourcing, semantic includes humans in a social role where communication web, and define concepts that will be used throughout is mediated by technology. The purpose is not usually the paper. We also examine the need of human contri- to perform a computation. bution in the domain of semantic web which provides What is Crowdsourcing? Crowdsourcing is a term the grounds for a successful confluence of human com- used for a range of activities that take can take dif- putation and the semantic web. ferent forms as examined by the authors of [27]. The term was originally coined by Jeff Howe [37,38]. Es- 2.1. Human Computation and Collective sentially, a crowdsourcing system engages a multitude Intelligence: Some Theoretical Foundations of humans to help solve a wide variety of problems. It is worth distinguishing between crowdsourcing Researchers consider human computation similar to systems (platforms) vs. crowdsourcing applications. but not synonymous with terms such as collective in- The former refers to those systems such as Amazon telligence, crowdsourcing and social computing. This Mechanical Turk1, Turkit [54], CrowdFlower2, Cloud- distinction has been argued in more detail by Quinn Crowd3 to name a few that help build crowdsourcing and Bederson [81]. While subtle differences may be applications in various domains. The latter is a more present, it is evident that these domains are closely knit encompassing term to include any application or sys- with one another. tem that may incorporate an element of crowdsourc- 2.1.1. What is Human Computation? ing. Some detailed surveys may be found for further details [24,59,117]. Most widely adopted definition of human computa- While most obvious forms of crowdsourcing sys- tion in state of the art research has been adopted in- tems engage a crowd of users to explicitly collabo- spired by von Ahn’s dissertation titled "Human Com- rate to build a meaningful and useful artifact, other putation" [108]. That thesis defines the term as: "...a less obvious forms employ implicit means to gather paradigm for utilizing human processing power to user contributions such as the ’Games With A Pur- solve problems that computers cannot yet solve". From pose’ (GWAP) paradigm. This includes many exam- the body of work that self-identifies as human compu- ples where the users implicitly collaborate. This dis- tation, Quinn and Bederson [81] present a consensus tinction is provided in the classification by Doan et al. that emerges as to what constitutes human computa- [24]. tion: a) The problems fit the general paradigm of com- There are several fundamental issues in crowdsourc- putation, and as such might someday be solvable by ing, however, most prominent of thse are: nature of computers, b) The human participation is directed by tasks that can be crowdsourced, reliability of crowd- the computational system or process. sourcing, crowdsourcing workflows. Accoring to Doan 2.1.2. Classification of Human Computation Systems et al. [24], crowdsourcing systems face four key chal- With a plethora of contributions from several research domains coming together under the umbrella 1www.crowdsource.com of human computation, researchers have attempted to 2www.crowdflower.com distinguish and classify human computation systems 3www.crowdsource.com 4 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

Fig. 1. Ontology Development Process (simplified adaptation from [80]) lenges: 1) How to recruit contributors, 2) What they unlike the traditional AI researchers working to rebuild can do, 3) How to combine their contributions, and some aspects of the brain. 4) How to manage abuse. In addition, the balance be- 2.2.1. Ontologies and Ontology Engineering tween openness and quality also needs to be ensured. Ontologies form the backbone of the semantic web. A number of surveys on existing crowdsourcing sys- An ontology is an explicit, formal specification of tems exist [24,59,117], which may be referred to for a shared conceptualization of a domain of interest an indepth analysis of the issues involved. [31,32]. This famous definition by Gruber implies that the ontology should be machine-readable (formal 2.2. Semantic Web Preliminaries specification), while at the same time it is accepted by a group or community (shared). Most ontologies are Tim Berners-Lee envisioned a ’semantic web’, capa- also considered to be restricted to a given domain of in- ble of providing automated information access based terest and therefore model concepts and relations that on machine-processable semantics of data and heuris- are relevant to a particular task or application domain tics that utilize this metadata. A new kind of web, [15]. Ontologies are said to formalize the intensional driven by explicit representation of the semantics of aspects of a domain (also known as the TBox schema), the data, accompanied with formalized knowledge whereas the extensional part is provided by a knowl- models, vocabularies in the form of ontologies to en- edge base that contains assertions about instances (also able services operate a greater level of quality [28]. known as the ABox schema) of concepts and relations Semantic web was aimed at solving the problems that as defined by the ontology. originally the AI researchers faced in knowledge acquisition, engineering, modeling, representation and Understanding the Ontology Development and reasoning upon knowledge. Given the web enabling a Knowledge Engineering Processes: workforce of millions of knowledge ’acquisitioners’ Figure 1 gives an overview of the activities involved working nearly for free providing tons of information in the Ontology Development process adapted from and knowledge, semantic web, as the pioneers of se- [80] . This process clearly distinguishes between three mantic web resaerch claimed, was envisioned as the key groups of activities namely Management, Devel- enabling force to build a brain of and for mankind [28], opment and Support. Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 5

Ontology Management: This refers to primarily the structured data on the Web’ [12]. Linked data initiative activities of scheduling, controlling and quality assur- claims to provide two primary advantages by expos- ance. Scheduling involves activities that pertain to the ing data in the form of linked data. These are data dis- coordination and management of an ontology devel- covery and data integration at web-scale, in a uniform opment project such as resource and time manage- and homogenous manner. Linked open data is based ment. Controlling deals with ensuring that the sched- on two simple ideas [35] : First, to employ RDF to uled tasks are accomplished as planned. Quality assur- publish structured data sources. Second, to explicitly ance process is involved in evaluating the quality of interlink different data sources. Tremendous amount the outcomes of each activity, in particular of the de- of data is published on the Web according to the linked veloped ontology. data principles [12,42]. LOD has generated remark- Ontology Development: This is further divided into able threads of arguments among research communi- three phases: pre-development, development and post- ties. The number of participants in the LOD cloud has development. In the pre-development phase, an envi- been significantly growing and publishing, consuming ronmental study investigates the intended usage and and reuse of structured data on the web has been al- the competencies of the envisioned ontology. The fea- tered dramatically. Apparently, managing data in such sibility study ensures that the ontology can actually highly distributed and heterogeneous environment is a be built within the time and resources assigned to the critical issue and hence has opened up many research project. The development stage of ontology develop- opportunities. We examine the issues that require spe- ment includes specification, which defines the usage, cial input from humans in the next sub-section, thus users and the scope of the ontology. This is usually fol- making LOD management a key area subject to human lowed by conceptualization in which domain knowl- computation. edge is structured into meaningful models, followed by 2.3. Human contribution in the process of semantic formalization and implementation. During formaliza- content creation tion, the conceptual model is formalized to prepare the implementation in a given ontology language. In the We focus on two strands of semantic content cre- post-development phase, the ontology is subjected to ation: 1) Ontology Development and 2) Linked Data use, updates, and maintenance as required. Management. We comprehensively analyze and sum- Ontology Support: This covers a broad range of ac- marize the need for human intelligence in contribution tivities which are considered to be most crucial and in both these contexts. cover many areas of semantic content creation and maintenance. No order is determined in which these 2.3.1. Need for Human Contribution in the Ontology activities must be undertaken. Typical support activ- Engineering Process ities include knowledge acquisition (specification of In order to highlight the role of human intelligence the knowledge needed for ontology), learning (pro- in semantic web research, Siorpaes and Simperl [98] cess of automatically creating an ontology or a part of surveyed methodologies, methods and tools covering it), evaluation (analyzing the quality of the developed various activities in the ontology engineering lifecycle ontology), integration (for reusing other ontologies to in order to learn about the types of processes knowl- build a new ontology), documentation (for providing edge and ontology engineering projects conform to, detailed description of the ontology and main activ- and the extent and reasons they might rely on human ities and tasks of the development process), merging intervention. We summarize their findings here with (produces a new ontology by combining existing on- an overview shown in Figure 2 as derived from their tologies), configuration management (tracks versions analysis. They analyze and classify human contribu- of the ontology) and alignment (establishes relations tion falling in three key stages of Ontology Engineer- among related or complementary ontologies). ing stages described in Section 2.2.1 earlier. These are: Researchers have primarily focused on the develop- Ontology Development, Semantic Annotation and On- ment and support activities because they are specific to tology Support. In the figure, the broad task categories are classified according to the level of automation and the scope of the semantic content creation. the extent of human contribution usually required by 2.2.2. Linked Open Data these activities. Linked Open Data (LOD) has recently been emerged Ontology Development: A further detailed break- as ’a set of best practices for publishing and connecting down of ontology development tasks is given in Fig- 6 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey ure 3. It is obvious, that pretty much all tasks require human intervention, while only few are supported with automation support. Conceptual modeling, which is one of the most essential tasks in ontology development, is essentially a human-driven task. Only very few tasks can be automated and the final modeling decision is always taken by human actors. Automation support is partly possible for collecting relevant terms or for detecting properties of concepts based on semi- structured knowledge corpora such as folksonomies. Semantic Annotation Automation has more role in the annotation of several types of media such as text annotation that leverages on natural language processing techniques and has been investigated upon in a plethora of research. However, human supervision is needed to ensure reliability. The annotation of multimedia content is more challenging, whereby only low level semantic content may be automatically derived. Much of the high level semantics is derived using human effort. The annotation of web services is very much manual task. Ontology Learning: Although, it supported with tools that may be automatically executed, require human effort in terms of input and feedback. A variety of tools are available ranging from fully automatic, to semi-automatic, to those that heavily rely on user in- Fig. 2. Classification of Tasks in the Ontology Engineering Process tervention through out the process. (as summarized in [98]) Ontology Alignment: This is another activity that is said to be specific in nature and far from a stage where nological developments in the data management per- any successful generic methodologies have been seen. spective in the last decade [1]. However the seamless While there are numerous tools to match, merge, in- consumption and integration of linked open data is tegrate, map and align heterogeneous ontologies, very challenged by the several quality issues and problems few tools can be run fully automatically. Most tools re- that the linked data paradigm is facing. As researchers quire a human in the loop, to give feedback or input remark, many of these quality issues are not possible suggestions to the system. Some even depend on user to be fixed automatically rather, require manual human defined rules to carry out the mappings. effort. Ontology Evaluation: This is a broad topic, which is Emerging research is establishing and highlighting nevertheless primarily driven by human effort. Auto- the need to combine human and computational in- matic support is available to cross-check and validate telligence, instead of relying on fully automated ap- structural features of taxonomies. Some tools are de- proaches. Simperl et al., [91] argue that several tasks signed to facilitate the humans to carry out the eval- in the context of linked open data are inherently hu- uation. The notion of achieving an automated process man driven, given their highly contextual and often of ontology evaluation generic enough to be applied knowledge intensive nature, which imposes consid- across domains is hardly feasible, given the depth of erable challenges to the the algorithmic approaches. knowledge needed that is difficult to be captured in They provide a broad classification of tasks derived tools that run on their own. from the study of the architecture for applications that 2.3.2. Need for Human Contribution in Linked Data consume the linked data as suggested in [35]. [1], [2] Management and [8,9] also provide insights into the tasks manage- Linked data, fueled primarily by the semantic web ment for the linked data from the human intelligence vision, is considered to be one of most important tech- perspective. Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 7

fore this type of classification becomes dependent on human contribution. – Ordering or Ranking Facts: Often the linked open data facts need to be ranked depending on the needs for querying and browsing the data. This is again a task that is difficult to be automated and would require human intervention. – Generating new facts: Linked data suffers greatly from the lack of completion of information. For instance, a common need is for labelling in terms of multi-linguality. Providing non-english labels (Translation) is something human contribution can not only benefit from, rather it is indispensable in certain situations. – Creating Links with the LOD: Applications consuming the LOD will often attempt to create links with the entities in the LOD datasets. This too cannot be automated entirely and often requires human input. Quality Assessment of Linked Data: This includes the following: – Validation of Linked Data: Many aspects of linked data need manual validation in terms of completion, correctness and provenance. – Meta-data Completion: At times the extraction Fig. 3. Classification of Tasks in the Process of Ontology Develop- process used to create linked data results in the ment (as summarized in [98]) incomplete extraction of entities from the source datasets, resulting in incomplete meta data infor- We summarize the broad group of tasks that could mation, which needs to be manually inspected, be subject to semi-automation and human computation identified and corrected. based on what has been identified by the existing stud- – Meta-Data Correction: It is also possible that ies, and that are relevant to the linked data architecture. the meta-data information is extracted incorrectly. This is something that is difficult to be detected Linked Data Annotation and Production: automatically and requires human intervention – Identity Resolution: This involves creating owl: for identification and correction. sameAs links between different entities, either by – Link Evaluation: Links on the LOD often need to comparison of metadata or by the investigation of be manually inspected, for correctness and verilinks on the human web. fication, even though automated tools may help in identifying some problem areas. At times the – Entity Linking: This may involve linking two en- link, though valid, doesnot show any valid con- tities using a new relation between them. This is tent against the linked entity. Such problems are different from identity resolution since the new hard to be identified and corrected in an auto- link may not always be an owl: sameAs relation. mated manner. – Entity Classification: Unlike the emphasis of the traditional semantic web ontologies, the LOD Query Processing: Since any form of consumption tends to emphasize the relationships and links be- of linked open data requires some form of query pro- tween the entities, rather than classification of en- cessing, primarily carried out using the SPARQL lan- tities. Since the vocabularies used are generic, it guage, it is natural that this task will involve human is difficult to infer a classification using the estab- intelligence at various levels. lished reasoning methods and techniques. There- 8 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

We will revisit these tasks with details of how they cording to Difranzo and Hendler [23] who claim, while have been subjected to human computation methods, remarking on the mutual promise that the two fields of- in particular crowdsourcing in Section 3.2.2. fer, that human computation can be used to help curate the semantic web data in the linked open data cloud 2.4. Confluence of Semantic Web and Human while the semantic web can be used to provide better Computation user continuity and platform consistency across human computation systems. However, many challenges and After more than a decade of semantic web research, questions still remain. Leveraging the best of human researchers remain challenged by the large scale adop- computation and semantic web technologies is greatly tion of the semantic technologies. The question of needed to bring forth this next generation. why and where the element of human computation is needed in semantic web has been answered in the pre- vious sub-section. 3. Application of Human Computation Simperl et al. [94] argue that human computation Techniques to Semantic Web Research methods such as paid micro-tasks and games with a purpose could be used to advance the state of the art In this section, we first provide an overview of those in knowledge engineering research in general and on- human computation genres that have been applied to tology engineering in particular. They examine the lit- the semantic web research. We then present a classi- erature to note how human computation has received fication of the broad tasks and applications in the se- great attention from the semantic web research com- mantic web domain according to these genres. We also munity, resulting in a number of systems that tackle review tool support for closely integrating human com- essentially human driven tasks that require expertise putation into semantic web processes. such as ontology classification, annotation, labelling, ontology alignment and annotation of different types 3.1. Human Computation Genres for the Semantic of media to name a few. Web Looking into the reasons of barriers imposed by the adoption of semantic technologies, one finds that there Applications of human computation techniques to is considerable lack of useful semantic content as this the domain of semantic web falls primarily under two content cannot be created automatically but requires genres: 1) Games with a Purpose (GWAP) and 2) to a significant degree, human contribution. Ironically, Mechanized Labor or Micro-task Crowdsourcing. We there is not enough interest in creating semantic con- present a brief overview of both approaches and then tent from the humans. Abraham Bernstein, in his represent classification of existing approaches according cent vision towards the ’Global Brain Semantic Web’, to the two genres. highlights important differences between human computers and traditional computers [10]. He enlists three 3.1.1. Semantic GWAPs: Semantic Games With A major differences: Motivational Diversity, Cognitive Purpose Diversity and Error Diversity. These diversities give The philosophy that GWAPs capitalize upon is sim- rise to a range of issues when including people in se- ple yet elegantly effective: tasks that are difficult for mantic web. the computers but can be tackled easily by the humans Research clearly indicates that combining human are hidden behind entertaining games. The games are computation and semantic web is of mutual benefit to designed to engage regular users and not just ex- both domains i.e. its benefits are not uni-directional. perts. Users are usually unaware when they are play- Both domains are complementary to one another. Al- ing such games how they are indirectly helping build though, the benefits of one may have superseded the knowledge bases and create annotations useful for fur- other in terms of the current landscape of research. ther computation and to be used in algorithms. This More research is evident in terms of how semantic web is a significant step in combining human and ma- domain has benefitted from human computation. More chine intelligence. Siorpaes and Hepp [95] adopted the real examples of systems whereby the semantic web Lui von Ahn’s "games with a purpose" [107,109,110] has driven the human computation system towards a paradigm for creating the next generation of the se- better future are yet to be seen. The future of human mantic web. They proposed the idea of tapping on computation and semantic web holds great promise ac- the wisdom of the crowds by providing motivation Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 9 in terms of fun and intellectual challenge. Accord- ing to von Ahn, the key principle in GWAPS is the idea that people are not interested in solving a computational problem, rather they want to be engaged in something entertaining. The primary objective of semantic GWAPS is massive semantic content generation which entails that there will be massive user participation as well. Pe-Than et al. [79] review human computation games in general and provide a typology consisting of 12 dimensions and strategies employed by the games. Simko and M. Bielikova [123] present a classification of semantics acquisition games’ purposes, ranging from multimedia metadata acquisition, through text annotation, building common knowledge bases to ontology creation. Simko and his collegues have a series of work which may be refered to with respect to semantics discovery and acquisition using human computation games [121,124]. Recent work by Simperl et al., [88] have proposed an ontology learning approach using games with a purpose. 3.1.2. Micro-Task Crowdsourcing Several recent research prototypes have attempted to use micro-task crowdsourcing for solving semantic web tasks. Some of the most common platforms used for crowdsourcing include the Amazon Mechan- ical Turk4(referred alternatively as AMT or MTurk) and CrowdFlower5, amongst others. MTurk (and others) provides a platform where users (requesters) can post a given problem(task) that other users (turkers) can solve. To do so, the requester designs a problem, to a suitable degree of granularity, resulting in a number Fig. 4. The Typical Task Lifecycle in a Crowdsourcing Platform of so-called Human Intelligence Tasks (HITs), which can be tackled independently by multiple turkers in re- accepted submissions are further processed for retriev- turn for a financial reward. ing and synthesizing relevant results. The complete execution of a task (HIT) lifecycle us- There may be a number of additional configuration ing any particular crowdsourcing system involves sev- parameters when tasks are published or submitted such eral general stages, as shown in Figure 4: (i) A task as the number of assignments, the time to completion is defined, with input parameters and the needed data, for each HIT or restrictions on the profiles of the work- (ii) The task is published over the crowdsourcing plat- ers. Once the tasks are completed, and submitted the form, (iii) The task is searched for or discovered by the results are collected and aggregated and quality assur- crowdworkers (via search or direct notification), (iv) ance measures applied depending on the the nature of The crowdworkers perform and submit the tasks, (v) the task design. The evaluation may be carried out in The submitted tasks then need to go through a review a number of ways. Often, a common practice to al- process for obtaining the submissions, (vi) The sub- low assignments of the same task to multiple workers. missions are either accepted or rejected and (vii) The Therefore the results may be aggregated using majority voting or other sophisticated techniques such as a 4www.mturk.com probability distribution or by taking into account some 5www.crowdflower.com estimate of the expertise and skills of the works. 10 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

An important conceptual distinction has to be made There are tradeoffs to both approaches. There is also between the terms tasks, micro-tasks and complex evidence of using a hybrid-genre workflow [83] that tasks. This has been done by Luz et. al [59]. Micro- seeks to combine the two genres in a single workflow tasks are atomoic operations, whereas complex tasks to achieve some promising results. However, the hy- are sets of microtasks (e.g some workflow) with a spe- brid approach may not be feasible in all kinds of sce- cific purpose. In this section, we adopt a generic ap- narios and needs more dwelling into further. proach to crowdsourcing approaches inclusive of both Another example of a hybrid genre is the combina- types of tasks. We dedicate a separate discussion to tion of a ’contest’ and a paid micro-task approach used those systems that involve some form of workflow by Acosta et al. [1]. The contest targets the experts and based approach as the basis of their crowdsourcing linked data enthusiasts, whereas the paid microtasks model in Section 5. target faceless crowds on the Amazon mechanical turk. There are several factors known from existing re- The study is an interesting contribution to show how search which constrain the applicability of a micro- such a hybrid methodology, whereby both approaches task appraoch e.g. according to [91] there are three key used not only complement one another, they may be factors: optimally combined together to enhance linked data Decomposability: The tasks need to be decompos- curation processes. able into appropriate levels of granularity which can be executed independently. This also needs to be done 3.2. Classification of Human Computation Tasks and according to the platform upon which the task is be- Genres in the Semantic Web ing executed e.g. AMT. If the granularity of the task is A broad classification of tasks subject to human higher than the level at which it can be atomically pub- computation in the semantic web domain and respec- lished, then additional workflows need to be generated tive genres is given Figures 5 and 6 and detailed clas- and managed such as the approach defined in [50]. sification of related research and applications is pre- Verifiability: The performance of the workers need sented in Tables 1 and 2. The broad classification is to be measurable. This entails the need for meth- adapted from the reviews presented in [23,94,98,114] ods and techniques for the quality assessment of the and a study of the systems and applications enlisted. collected results. In addition, mapping to the input The classification is broadly classified in the categories from different task assignments are also needed. Open- of ontology engineering and linked data management ended tasks such as requiring the definition of a term or tasks. The classification of the studies show that the translation requires means to deal with iteration such role of GWAPs in the ontology engineering tasks is as one discussed in [54,56]. significantly greater than those of mechanized labor Expertise: The domain of the tasks being experi- and there is considerable room for more research. mented ought to be available to the workers. How- ever, some knowledge intensive tasks may require ad- 3.2.1. Ontology Engineering ditional expertise which the general purpose platforms Research in ontology engineering not only recog- such as AMT are not expected to provide. nizes and establishes the need for human intelligence and contribution [98], there have been considerable ef- 3.1.3. Hybrid Genres: Combining GWAPs and forts to develop ontologies in a collaborative and com- Micro-Task Crowdsourcing munity driven manner. A detailed review may be found There is evidence of research that combines the two at [90]. CrowdMap [85] attempt to solve ontology human computation genres in an attempt to cache upon alignment tasks using CrowdFlower using a workflow the relative benefits of both approaches. Thaler et. al framework designed to crowdsource ontology align- [104] compared these two prominent techniques to ment tasks. Mortensen et al., [69–73] and Noy and col- evaluate which approach is better with respect to costs legues [76] present a series of works with respect to and benefits. They used OntoPronto GWAP and repli- ontology validation and quality assurance in the do- cated the study by employing the Amazon Mechani- main of biomedical ontologies. The work serves to cal Turk using a similar approach. Their experimenta- establish strong grounds for the feasibility of crowd- tion showed the feasibility of both approaches in ac- sourcing in both generic and specialized domains. A complishing conceptual modeling tasks, however, they broad overview of these tasks is presented in Figure 5. concluded the microtask approach to be superior in A detailed classification of ontology engineering tasks terms of both development and problem solving tasks. using human computation in presented in Table 1. Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 11

Table 1 Broad Classiﬁcation of Human Computation Tasks in the Semantic Web (Ontology Engineering) and the Respective Genres

Task Category Task Description Genre Source Ontology Engineering: Ontology Creation/Ontology Alignment Concept and Instance Identifica- Decide if given entity forms a class or an in- GWAP OntoPronto [95] tion stance OntoGame [96] Identify class from given attributes GWAP GuessWhat?! [62] Given a set of attribute descriptions, identify the class Validate class labels GWAP GuessWhat?! [62] Evaluate if the class name fits the description Specification of Term Relatedness Check whether two terms (usually ontology MTurk InPhO [26] concepts) are equal Select from term pairs Select from a set of terms GWAP SpotTheLink [101,102] Voting on terms GWAP LittleGameSearch [87] Provide related terms GWAP FreeAssociation [106] Verification of Relation Correct- Equivalence CrowdFlower CrowdMap [85] ness Verification of a valid or invalid relation MTurk Conference v2.0 [19] Subsumption MTurk Noy et al. [76] CrowdFlower CrowdMap [85] Specification of Relation Type Equivalence CrowdFlower CrowdMap[85]

Select an appropriate relation SpotTheLink [101,102] from the set of given relation OntoPronto[95] types Subsumption MTurk InPhO [26] CrowdFlower CrowdMap [85] GWAP SpotTheLink [101,102] Categorilla [106] OntoPronto [95] Instantiation (Specify Instance Types) GWAP Categorilla [106] Disjointness GWAP OntoPronto [95] InstanceOf GWAP OntoPronto [95] Spatial, Purpose, Opposite of, IsRelated to GWAP Verbosity [111] Generic and Domain Specific Relations GWAP ClimateQuiz [86] Ontology Engineering: Relevant for Ontology Learning/Automatic Extraction of Ontologies Verification of Domain Relevance Verify if the given term is relevant to the do- GWAP OntoGalaxy [49] main Annotation of Multimedia Select from given choices and annotate a GWAP OntoTube [95] video CrowdFlower, MTurk CrowdTruth [40] Extraction of Text Annotations Medical Text Annotation GWAP Dr. Detective [25] Generic text annotation CrowdFlower, MTurk CrowdTruth [40] Annotating Web content eBay offereings GWAP OntoBay [95] Domain Specific Vocabulary and Collect goals and attributes GWAP Common Consensus [53] Relation Building Collect terms and relationships GWAP TermBlaster [122] 12 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

– Concept and Instance Identification/Collection: One of the most fundamental tasks in Ontology engineering is the process of identifying and collecting concepts and instances. Ontogame [96] and OntoPronto [95] adopt a game based approach to decide if given entity forms a class or an instant. Players are presented with a Wikipedia page of an entity and they have to judge if it de- notes a class or an instance and then relate it to the most specific concept of the PROTON ontology. GuessWhat?! [62] also includes the similar feature, whereby, using a GWAP, the player is sup- posed to identify class from the given attributes and description. A similar task also incorporated by GuessWhat?! [62] is that of validating class labels, in which the user is required to evaluate if the class name fits the description. – Specification of Term Relatedness: In this particular type of task, the crowd workers check whether two terms, which are usually concepts in an ontology, are equal. InPhO [26] is an example for such a study which employs MTurk for the purpose of this task, whereby workers select from a choice of term pairs given to them to select the pair that represent the equality in the best manner. Work- ers may be required to select from a set of given terms (SpotTheLink [101,102]), or vote on terms Fig. 5. Classification of Human Computation Tasks and Genres in (LittleGameSearch [87]). In some cases, workers Semantic Web (Ontology Engineering) or players provdie related terms (FreeAssociation [106]). – Verification of Domain Relevance : Some tasks – Verification of Relation Correctness: In this task, require the crowdworkers to verify if the given crowd workers or players are usually presented term is relevant to a specific domain such as On- with a pair of terms and a relation between these toGalaxy [49]. terms and they are required to determine if the re- – Annotation of Text and Multimedia: Several stud- lation is a valid or an applicable one. Most com- ies include the annotation of text (Crowd-Truth mon of these relation types include equivalence [40]) and multimedia (OntoTube[95], CrowdTruth (CrowdMap [85], Conference v2.0 [19]) and sub- [40]). sumption (Noy et al. [76], CrowdMap[85]). – Annotation of Web Content: Another aspect of se- – Specification of Relation Type: This task focus mantic annotation is the annotation of the web on the selection of an appropriate relation from content. OntoBay [95] for instance gets annota- the set of given relation types. Most common tions for eBayOfferings. relation types that have been crowdsourced so – Domain Specific Vocabulary and Relation Build- far are equivalence (CrowdMap [85], SpotThe- ing: There also exists evidence of studies that Link [101,102], OntoPronto [95]), subsumption work on domain specific relation and vocabulary (CrowdMap[85], InPhO [26]), instantiation (Cat- building. Common Consensus [53], for instance, egorilla [106]), disjointness (OntoPronto[95]) collects goals and attributes while TermBlaster and instanceOf (OntoPronto[95]). There are other [122] collects terms and relationships between examples such as spatial, purpose, oppositeOf, is- them. RelatedTo provided by Verbosity [111]. There are also efforts to crowdsource domain specific rela- It is obvious from Table 1 and Figure 5 that a wide tions such as ClimateQuiz [86]. range of tasks have been subjected to human contribu- Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 13

linking tools in order to highlight aspects of the interlinking process that crucially rely on human contributions and explain how these aspects could be subject to a semantically enabled human computation architecture that can be set-up by extending interlinking platforms such as Silk with direct interfaces to popular microtask platforms such as Amazon’s Mechanical Turk. Linked Data Annotation and Production Tasks: – Identity Resolution: This involves asking the crowd to help create owl: sameAs links between different entities, either by comparison of metadata or by the investigation of links on the human web. Simperl et al. [91] use the MTurk to map entity links. A similar approach for entity disambiguation has been adopted by CrowdLink [8,9] whereby the crowd disambiguates the researchers DBLP names and links using the information provided. – Entity Linking: This task requires the crowd to link two entities using a new relation between them. This is different from identity resolution since the new link may not always be an owl: sameAs relation. For instance, ZenCrowd [21,22] utilizes MTurk to involve the crowd in matching entities to most similar ones by picking links Fig. 6. Classification of Human Computation Tasks and Genres in (Matching URIs). Semantic Web (Linked Data Management) – Entity Classification: Simperl et al. [91] propose the approach of entity classification which in- tion using both mechanized labor and GWAPs. While volves classifying entities to pre-determined vo- the classification is not meant to be exhaustive, how- cabularies or ontologies. ever, it shows representative studies in broadly most – Ordering or Ranking Facts: Often the linked open categories of ontology engineering tasks reported in data facts need to be ranked depending on the literature. It is also evident, that interestingly, much needs for querying and browsing the data as pro- of ontology engineering tasks have so far been tack- posed by Simperl et al. [91]. RISQ! [115] uses a led using the GWAP approach, implying that there is GWAP approach to rank facts by asking players much potential and room for the mechanized labor to to play a jeopardy like quiz game. be taken up further towards a holistic ontology engi- – Generating new facts: Linked data suffers greatly neering approach. from the lack of completion of information. E.g. a common need is for labelling in terms of multi- 3.2.2. Linked Data Management linguality. Providing non-english labels (transla- The Linked Data initiative has come to be seen as a tion) is something human contribution can benefit significant step forward towards the large scale adop- from as proposed by Simperl et al. [91]. SeaFish tion of the semantic web. However, the linked data [103] uses a GWAP approach for creating an- management, which includes linked data creation, up- notations for images in a collaborative manner. dation, querying, retrieval, validation and quality as- CrowdLink [8,9] provides means to create or ac- sessment itself presents its own challenges. Linked quire new knowledge and facts for LOD sources data researchers have also capitalized upon the bene- using MTurk. WordBucket [82] uses a mobile fits of crowdsourcing, GWAPS and human computa- based GWAP approach and attempts to create tion to solve several challenges [97]. Simperl and Wol- linked data by asking the players to play word ger et al., [93] provide a survey of various data inter- games and thus adding word senses and trans- 14 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

Table 2 Broad Classification of Human Computation Tasks in Semantic Web (Linked Data Management) and Respective Genres Task Category Task Description Genre Source Linked Data Management Linked Data Annotation and Production Identity Resolution Mapping entity links Crowd (MTurk) Simperl et al. [91] Entity/Link disambiguation Crowd (MTurk) CrowdLink [8,9] Entity Linking Matching entities to most similar ones by MTurk ZenCrowd [21,22] picking links (Matching URIs) Entity Classification Classification Crowd (MTurk) Simperl et al. [91] Ranking Facts in LOD Ordering Crowd (MTurk) Simperl et al. [91] Play Jeopardy like quiz game to rank facts GWAP RISQ! [115] Generating New Facts Translation Crowd (MTurk) Simperl et al. [91] Collaborative image annotation GWAP SeaFish [103] Create new facts Crowd (MTurk) CrowdLink [8,9] Linguistic linked open data Production mGWAP WordBucket [82] Play word game and add word senses and translations Generating ground truth by anwering ques- GWAP WhoKnows?Movies! tions on a quiz [105] Create Links with LOD Link photographs with LOD concepts mGWAP UrbanMatch [17,18] Concepts Quality Assessment of Linked Data Validation of Linked Validating links Crowd (MTurk) CrowdLink [8,9] Data Review concepts in DBPedia while review- App Taste It! Try It! [97] ing restaurant reviews Link verification GWAP VeriLinks [52] Metadata-Correction Metadata-Correction Crowd (MTurk) Simperl et al. [91] Incorrect data type Contest (TCM Tool) & Acosta et al. [1] Crowd (MTurk) Metadata-Completion Incomplete object value Contest (TCM Tool) & Acosta et al. [1] Crowd (MTurk) Link Evaluation Incorrect links Contest (TCM Tool) & Acosta et al. [1] Crowd (MTurk) Link verification MTurk CrowdLink [8,9] Evaluating LOD heuristics using a quiz GWAP WhoKnows? [112] Query Processing Query processing MTurk CrowdSPARQL [2,92]

lations. WhoKnows?Movies! [105] also uses a use principle [44] demands the involvement of humans GWAP approach to generate ground truth by re- to ensure such quality. quiring the players to answer questions on a Quiz. – Validation of Linked Data: Many aspects of – Creating Links with the LOD: Applications con- linked data need manual validation in terms of suming the LOD will often attempt to create links completion, correctness and provenance. In this with the entities in the LOD datasets. Urban- respect, CrowdLink [8,9] provides a crowdsourc- Match [17,18], for instance, uses a mobile GWAP ing based workflow for verifying entity links from approach to link photographs with LOD concepts. LOD sources. Taste It! Try It! [97] is an application that aims to get users to review concepts in Quality Assessment of Linked Data: When it comes DBPedia while reviewing restaurant reviews. Ver- to quality assessment whether semantic web in gen- iLinks [52] is another example that uses GWAP eral or linked open data in particular, the fitness-for- approach for link verification. A relevant issue Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 15

when it comes to linked data curation using hu- 3.3. Automated Tool Support for Human man computation is the issue of data provenance Computation based Semantic Web Tasks and version control on linked data resources. A collaborative approach to curate linked data has Most applications described in Section 3.2.1 and been proposed by Knuth et al., [46]. At the core 3.2.2 are primarily independent, external systems or of their approach lies the design and usage of applications. There are relatively fewer efforts that aim the PatchR ontology [45] that allows to describe to combine and closely integrate crowdsourcing into patch requests including provenance information. ontology engineering practices. We briefly review such They extended WhoKnows? [112] to embed pub- efforts in both ontology engineering and LOD man- lishing prevenance information within a GWAP. agement perspective. Markovic et al, examine the role of provenance in social computation [67], and demonstrate the Automated Tool support for crowdsourcing On- importance and management of provenance of tology Engineering Tasks: To reduce the complexity crowdsourced disruption reports [67]. The pro- of ontology construction, the process is often boot- posed vocabulary SC-PROV, a provenance vocab- strapped by re-using existing or automatically derived ulary for social computation [63] is an important ontologies. Ontology learning methods are often used step in managing the provenance information and to automatically extract ontologies from structured and thus contributing to enhancing the transparency unstructured sources. Crowdsourcing is therefore seen in human computation systems. This is crucial as a useful method for seeking human contribution in for enabling decision making about the reliability various ontology engineering tasks. However, despite of participants and quality of the generated solu- the usefulness, the process of acquiring crowdsourced tions. inputs brings its own baggage. Hanika et al, [34] on – Meta-Data Correction: For identification and cor- remarking the need for automated tool based support, rection of incorrecte meta-data, Simperl et al. [91] highlight the high upfront investments(understanding and CrowdLink [8,9] provide MTurk based tasks techniques, task design) in setting up crowdsourcing which can easily post tasks to the crowd. for ontology engineers. – Meta-data Completion: Acosta et al. [1] provide Researchers in the domain of Natural Language means to verify and complete incomplete meta Processing (NLP), have adopted crowdsourcing tech- data information which needs to be manually in- niques [84]. A recent effort has successfully engi- spected, identified and corrected. They provide neereed a crowdsourcing plugin called GATE Crowd- both a MTurk approach and a custom tool called sourcing plugin [13], as a new component in the pop- TripleCheckMate. ulate GATE NLP system that allows seamless inte- – Link Evaluation: Acosta et al. [1] use their tool gration of crowdsourcing tasks into larger NLP work- TripleCheckMate to identify incorrect link and flows, from within the GATE’s user’s interface. Noy also use MTurk to do the same. CrowdLink [8,9] et al, [76] have also envisioned a similar tool to sup- also provide means for link verification using port ontology engineering tasks. Inspired from these their crowdsourcing framework for linked data ideas and efforts, the foremost effort to this end, is management. WhoKnows? [112] uses a GWAP the uComp Protégé plugin that aims to closely inte- approach for evaluating LOD heuristics using a grate typical crowdsourcing tasks into the ontology en- quiz. gineering work from within the Protégé ontology edit- ing environment [33,34,114] . Query Processing Tasks: The plugin allows for a number of ontology engi- CrowdSPARQL [2,92] is an approach proposed for neering tasks to be crowdsourced. The main stages in- carrying out a number of linked data management volved when using the plugin are shown in Figure 7. tasks using SPARQL query processing, and delegat- The efficiency of the plugin is compared to manual ef- ing part of the triples in the query to the crowd us- forts in terms of time and cost reductions, while en- ing MTurk. This is a novel approach indeed. There are suring good data quality. The findings indicate that in similar successful approaches in the database commu- scenarios where automatically extracted ontologies are nity such as CrowdDB [29]. verified and pruned, the use of the plugin significantly reduces the time spent by the ontology engineer and 16 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey leads to important cost reductions without a loss of 4.2. Key Driving Questions quality with respect to manual process. The key driving questions that drive the CI genome design are: What is being done? Who is doing it? Why are they doing it? And how is it being done? We provide an in-depth analysis of all the possibilities that these genes may take the shape of within semantic web research. We use the 16 genes as the basis from the original genome and map them to existing semantic web research. An overview is presented in Table 3. 4.2.1. What is being done? Fig. 7. Main Stages in using the uComp Plugin from [114] The first question that needs to be answered for any Automated Tool support for crowdsourcing Linked particular activity according to this genome model is: Data Tasks: Automated tool support for Linked data What is being done? The most obvious level of granu- management has been proposed using the crowd by larity it is the task which is to be performed. We con- TripleCheckMate: a tool for crowdsourcing the qual- sider the two genes proposed in the original genome: ity assessment of linked data [48]. The tool has been Create: The actors in the system create a new arti- successfully used to crowdsource issues of accuracy, fact. It could be a piece of knowledge such as a con- relevancy, representational consistency and interlink- cept, relation or classification. ing quality dimensions (a subset of those described in Decide: This translates to an evaluation task or a se- [120]) applied to DBPedia. lection which is to be performed based on certain alter- natives, such as selecting the most appropriate relation from the set of given ones. The original genome model 4. A Collective Intelligence Genome for the suggests that completion of a complete goal involves Semantic Web at least one of each of these create and decide genes. Create genes will generally be followed with a decide After having analyzed the classification of broad se- gene in order to make a decision about what items are mantic web engineering tasks and the respective hu- to be kept. And the decide gene usually needs a create man computation genres, we present the collective in- gene to generate the choices being considered. telligence genome for the semantic web, which aims to highlight the essential threads that constitute the suc- 4.2.2. Who undertakes the activity? cessful design considerations for conceiving an appli- The next critical question to be answered is: Who cation that successfully integrates human computation. undertakes the activity? Hierarchy or Management: Someone in authority 4.1. Overview makes a decision about who performs a task. Crowd: Using this gene anyone who is part of a large Malone et al. in their seminal work present a user's group may take up a task who chooses to do so. This is guide to building blocks of Collective Intelligence (CI) a central gene in most collective intelligence systems. [60,61] . They try to answer the fundamental question Expert: This gene is not part of the original genome, of how to get the crowds to do what needs to be done. however We have included this as an exclusive gene. They claim that combining or recombining the right We consider it to be different from the hierarchy or genes or building blocks would result in the formu- management gene because the distinction is impor- lation of the right kind of system suited to the needs tant especially within the semantic web research. The of collective intelligence. We adopt this genome and semantic web research community recognizes the re- apply it to the existing research limiting our focus to liance on experts for certain domain specific tasks. primarily those studies that attempt to apply collective The choice between who performs when task is a intelligence, human computation or crowdsourcing of critical one. The choice of crowds is usually to bene- some sort in the context of semantic web research. fit from the skills and resources of a larger group of We analyze how well these studies comply with this people at a much larger scale. Experts are usually hard genome and identify any possible gaps, limitations and to find. Crowds are usually suited for tasks that do not opportunities in this light. require any particular skills or domain specific knowl- Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 17

Table 3 Overview of the Collective Intelligence Genome Question Gene Manifestion What Create Creation of new artifact Decide Evaluation or Selection Task Who Crowd Large group that performs a task Expert Someone with specialized knowledge of the domain Hierarchy Someone in authority Why Money Financial gain as motivation for doing work Love Enjoyment, fun, competition (from doing the task) Glory Reputation and sense of community How - Create Collection Independent tasks Contest One entry in the collection considered best and rewarded Collaboration Tasks are not independent (interdependent tasks), No way to divide an activity into smaller ones How - Decide Group Decision Includes voting, averaging, consensus, prediction Individual Decision Includes Markets and social networks edge. For the crowd gene to work, the task must be [98]. Reciprocity may also be said to fall into this cat- reasonably decomposable, and presented to the crowd egory, whereby the contributors receive an immediate in a feasible manner. At the same time, results must or long term benefit on performing a certain task. ensure feasible aggregability. In addition, mechanisms Glory: This includes reputation and sense of com- to prevent spam and sabotage are also indispensable. munity. Recognition and acknowledgement are pri- For those tasks where either the task is not decompos- mary contributors. able enough or the reliability of the task performance There may be several ways how the genes influence becomes questionable, experts are usually relied upon. contributions. Love and glory when put into play as Traditionally in the semantic web research much of se- motivators are known to reduce costs but not always. mantic content creation tasks have been performed by However, money and glory together can bring results experts. Only recently the researchers have started ex- faster. The right combination especially within seman- perimenting with and reported findings about the tasks tic web research remains to be experiemented upon; that are amenable to crowdsourcing and human com- however, this consideration is crucial. putation. 4.2.4. How the task is being performed? 4.2.3. Why the task is being performed? This is an important question that is a key determi- The question of who is incomplete without the ques- nant in driving collective intelligence in any system: tion of why? Why would someone perform a given How is it being done? A key deciding factor when task? What motivates them to participate? What in- ansering this question depends on the nature of deci- centives are provided to them? In a simplified sense sion making that follows crowd participation. Are the the three broad level motivations are Money, Love and contributions and decisions independent? Or are there Glory. Siorpaes and Simperl proposed some [98] in- strong inter-dependencies between the contributions. centive based measures for sparking motivation for hu- The four types of How genes are: Collection, Collabo- man contribution in the semantic web research. Their ration, Individual Decision, and Group Decision. breakdown of incentive scheme is more or less encom- For Create tasks, Collection and Collaboration are passed in the Money, Love and Glory model. relevant whereas for Decide tasks Individual or Group Money: It is long known that financial gain is one of Decisions are relevant. the primary motivations for most players in the mar- Collection: The task performed is independent of kets and traditional organizations. each other. Such gene is only useful when the over- Love: This is manifested in various forms. Users all activity can be divided into tasks independent of enjoy engaging in certain tasks, especially feeling of each other. If this is not the case then collaboration is community and socialization gives them a sense of be- needed. longing and purpose. This may include fun and compe- Contest: This is a subtype of collection gene. One tition incentives as suggested by Siorpaes and Simperl of the entries is declared best. Or a participant with 18 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

Table 4 Application of Collective Intelligence Genome to Semantic Web Research Source What Create/Decide Who Why How? C-Create D-Decide GuessWhat [62] Domain Ontology Cre- Evaluate if the class name fits the GWAP Love/Glory C-Collection ation from Linked Data description D-Consensus D-Majority Vot- ing Verbosity [111] Not pure Ontology Engi- Select an appropriate relation from GWAP Love/Glory C-Collection neering or Semantic Web the set of given relation types D-Consensus GWAP OntoPronto [96] Conceptual Modelling, Decide if given entity forms a class GWAP Love/Glory C-Collection Concept and Instance Col- or an instance D-Consensus lection Virtual Pet and Concept Collection Concept collection GWAP Love/Glory C-Collaboration Rapport [51] SpotTheLink Ontology Alignment, Select from set of terms, GWAP Love/Glory C-Collection [101,102] Entity Linking specify relation D-Consensus VeriLinks [52] Entity Linking Validation of linked data Crowd (MTurk) Money C-Collection D-Consensus UrbanMatch [17, Alignment and Interlink- Link photographs with LOD con- Mobile Crowd Love/Glory C-Collection 18,66] ing cepts D-Consensus CrowdSourced Conceptual Modelling: Select relations , select from term Crowd (MTurk) Money C-Collection InPhO [26] Conceptual Hierarchy pairs D-Consensus Formulation Noy [76] Ontology Verification Hierarchy verification Crowd (MTurk) Money C-Collection Mortensen et al., D-Consensus [69–73] CrowdMap[85] Ontology Alignment Mappings between Ontologies Crowd (Crowd- Money C-Collection Equivalence Flower) D-Consensus Subsumptions Validation: Validate a given mapping Identification: Select between different types of given relations Conference v2.0 Ontology Alignment Validation: Given two concept de- Crowd (MTurk) Money C-Collection [19] (Benchmarks) scriptions, validate if they are simi- D-Consensus lar CrowdSPARQL Coneptual Modelling Use SPARQL Query processing Crowd Money C-Collection [2,92] Ontology Classification tasks as means of crowdsourcing D-NA Entity Resolution ZenCrowd Entity Linking Picking links in order to match sim- Crowd (MTurk) Money C-Collection [21,22] ilar entities D-Prediction CrowdLink [8,9] Linked Data Management Missing LOD Knowledge Crowd (MTurk) Money/Glory C-Collection Tasks Validate LOD Links D-Consensus Dr. Detective[25] Extraction of Text Annota- Medical text annotation (to obtain GWAP Love/Glory C-Collection tion groud Truth) D-Consensus CrowdTruth [40] Semantic Annotation Text annotation, Crowd-Flower, Money C-Collection Tasks Image annotation Crowd (MTurk) D-CrowdTruth Video annotation Metrics Acosta et al. [1] Quality Assessment of Incorrect or incomplete object Contest (TCM Money C-Collection Linked Data value Tool) & Crowd Love D-Consensus Incorrect data and links (MTurk) Glory uComp [114] Ontology Engineering A whole range of ontology engi- Crowd (MTurk) Money C-Collection Tasks neering tasks via Protege D-Consensus, Authority Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 19 best performance may also be rewarded with a prize or specialized for semantic web research or for GWAP. some other recognition. Nevertheless, the threads of the collective intelligence Collaboration: This occurs when members of a genome provide useful means to cross analyze the re- crowd work together and interdepencies exist between search in the domain. their work. This is usually needed when there is no way of dividing a large activity into smaller independent subsets. 5. Crowdsourcing based Workflow Systems for the Group Decision: This is useful when each partici- Semantic Web pant has to make the same decision. Variants of this mode include voting, consensus, averaging and predic- In this section, we review and compare some of tion. the most notable studies which adopt a crowdsourcing Individual Decision: This is employed when wide- based model or mechanized labor for solving seman- spread agreement is not needed. The decisions may not tic web tasks. In particular, we include those studies in be needed for all. Variants of individual decisions in- this analysis that adopt some form of workflow based clude markets and aocial networks. Markets may not approach in their design. We provide an overview of be relevant for semantic web research however, the do- the different threads we use for comparative analysis main of social semantic web has emerged as a result and then provide a detailed analysis using the dimen- of social networks contributing towards semantic web sions highlighted in these threads. research. 5.1. Threads for Comparative Analysis 4.3. Application of Collective Intelligence Genome to Semantic Web Research The parameters used for the comparison are the key threads of analysis, which provide essential insights Table 4 shows the application of collective intelli- into the design and configuration of the key state of gence genome to selective works from the ones pre- the art studies. The parameters are explained in Table sented in Table 1 and 2. When compared with the 5 and the associated dimensions used later for anal- genes of the original genome, to the actual mapping ysis are also enlisted. Primarily, the key parameters conducted on actual case studies from semantic web we consider when comparing these systems are nature research, it is obvious that there is good spread of re- of task support, task design, workflow and data repre- search available in both genres, i.e., GWAP and Mech- sentation, worker interaction and engagement strategy, anised Labor. The GWAP seems to dominate more. worker performance, data aggregation, and data qual- The mapping of collective intelligence is useful in pro- ity and reliability. viding us with some interesting insights. Although, the coverage of studies presented in Table 4 is in no way 5.2. Comparative Analysis using Ontology exhaustive, it gives reasonable insights into the cur- Engineering Tasks rent trends to date. While a range of semantic web tasks have been experimented with, the creation and Table 6 presents a comparative analysis of re- the decide modes are fairly consistent between collec- search studies primariy focusing on some core task tion and consensus. This reflects that most tasks are in ontology engineering or linked data management of simple and independent nature. This leaves room using a micro-task based crowdsourcing genre. We for more studies to experiment and dwell further with focus our analysis on five primary studies that in- more genes as presented in the genome. clude CrowdMap [85], Noy and Colleagues [76] & One limitation that is felt for the collective intelli- Mortensen et al.[69–73], ZenCrowd [22], CrowdLink gence when applied in this context to semantic web [8,9], and CrowdTruth[40]. We chose these systems research and tasks especially when carried out using since they present the state of the art when it comes GWAP genre, oftentimes the task as presented to the to the confluence of semantic web and human compu- user is different from what is intended or how the tation research. The essential elements in the design users responses will actually contribute towards the ul- of these systems provide the key ingredients for the timate objective of semantic content creation. This is design of similar applications in the near future. not reflected in a comprehensive sense, as of now, how- CrowdMap[85] presents a workflow model for ever, the genome may be revised to include this thread, crowdsourcing mappings between ontologies. Works 20 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

Table 5 Threads and Dimensions of Analysis Parameter Dimensions of Analysis Nature of Task Support a. What specific Semantic Web based tasks have been experimented with Task Design (Task Composition, a. What methods of task representation are used Representation and Reusability) b. What methods of task decomposition are used c. What kind of task templates and UIs have been used d. The extent of reusability of task design and templates Workflow and Data Representation a. How human and machine computation is combined into a workflow b. What are common data formats in the semantic web research when applied to human computation Worker Interaction & Engagement a. What strategies have been used for presenting semantic web knowl- Strategy edge to users in an easy manner b. What strategies have been used to engage workers c. How these strategies may be classified d. Which have been used in semantic web research and which have been not Worker Performance a. How workers have performed b. What are the factors the worker performance depends on c. What performance measures are used Data Aggregation a. How the data and results are aggregated b. What methods are used for aggregation of data c. What challenges and prospects in this direction d. How these methods may be classified Data Quality and Reliability a. How data quality and reliability is ensured b. What methods are used c. What are the challenges faced so far d. How the data quality and reliability measures may be classified from Noy and Collegues also aim to solve ontology concept level alignments are undertaken. The instances verification tasks using crowdsourcing. CrowdLink[8, and property level alignments are ignored. 9] and ZenCrowd are similar efforts, that focus on Noy: The tasks undertaken in this study concerns linked data management tasks. However, much of the primarily validation or verification tasks. This includes emphasis of these efforts lies in automating tasks for e.g. primarily a hierarchy verification i.e. to verify a microtask crowdsourcing. There is considerable need superclass-subclass relationship. for combining human computation with machine com- ZenCrowd: The ZenCrowd focuses on extracting en- putation. CrowdTruth [40] present a framework for tities from HTML pages and linking them with ap- harnessing disagreement in gathering annotated data. propriate entities on the LOD cloud. Candidate links They achieve a human computation workflow through are generated using automated extractors and some au- a pipeline of four processes: 1) machine processing tomated decision making is applied using probabilis- of media, 2) reusable task templates to collect human tic models to reduce the candidate mappings that need input, 3) application of disagreement metrics, and 4) verification from the crowd. Therefore, primarily, in result presentation. Similar workflow would be bene- terms of crowdsourced tasks, only validation is ob- ficial to engineering semantic web tasks. We present tained. a comparative analysis of crowdsourcing based work- CrowdLink: The proposed CrowdLink architecture flow systems below using the threads and dimensions proposes to support a number of tasks in the linked mentioned in Table 5. data management from acquiring missing facts and in- 5.2.1. Nature of Task Support formation to verification and validation. Support for The nature of the semantic web tasks that have been structural and schema level tasks is also mentioned. undertaken are varied and many. Experimentation is primarily carried out for LOD ver- CrowdMap: CrowdMap primarily focuses on Ontol- ification, knowledge acquisition and entity or link dis- ogy Alignment tasks and in that too, only class level or ambiguation. Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 21

CrowdTruth: CrowdTruth provides a wide range of does not pose restrictions for the creation of new tem- annotation tasks for a range of input media such as plates. Some of the key tasks include Span Correc- text, multimedia and images. While the focus is on tion, Relation identification, Relation Direction Iden- obtaining annotations and pure ontology engineering tification, Event and Type Identification, Event Loca- methodology is not incorporated in the framework, tion, Participants and Time Identification. In addition however, the crowdsourcing framework is generic and for images and videos, the annotation includes object mature to be extended to incorporate an ontology en- identification or Event identification. gineering methodology. Some additional discussion points: It must be noted that most of these systems present support for atomic 5.2.2. Task Design tasks. Only CrowdTruth has some level of support for CrowdMap: The CrowdMap system defines two composite tasks. Most tasks are simple and do not re- types of micro-tasks namely: 1) Validation microtasks quire any iteration. and 2) Identification microtasks. The validation microtasks presents a complete relationship between two 5.2.3. Workflow and Data Representation classes and requires them to specify if they agree with CrowdMap: The CrowdMap workflow architecture the relationship or not. The identification task asks for includes components for generating candidate pairs the workers to identify a particular relationship be- from a source ontology (Pairs Generator), a Micro- tween the source and the target class. The task includes Task Generator and a MicroTask publisher for publish- the contextual information available for both classes. ing the tasks. Another end of the workflow deals with Noy: The study primarily focuses on the verification reading the results, processing, alignments and results task, specifically hierarchy verification is addressed. evaluation. The functionality is meant to be generic to Therefore, there is only a single task used. Questions be integrated into existing environments of ontology of the same type were placed on the same page for alignment. quality control purposes. Some experimental varia- Noy: While [76] does not mention the use of any for- tions included question with or without term defini- mal workflow model in their study, a conceptual work- tions as contextual information. flow model is proposed by the parallel study of the ZenCrowd: The study creates a single task for each authors in [69,73]. The workflow includes processes entity that needs to be evaluated by the crowd. Some for entity selection from a source ontology, task gen- textual content along with the entity is included as eration, optimization, spam and filtering and response means of contextual informations. The workers in turn aggregation. An additional process accounts for some have select the URIs that match the entity. Therefore form of context provisioning from external sources. the task at hand is primarily a selection task. ZenCrowd: ZenCrowd leverages algorithmic and CrowdLink: CrowdLink provides a number of task human workers working together to produce high qual- templates e.g. new or missing triple information on the ity results. The architecture of ZenCrowd takes as in- LOD may be sought from the crowd. Similarly, ver- put HTML pages and enriches them using Entity Ex- ification or validation tasks may be created using the tractors, that extract textual entities, and linking them available templates. A unique feature of the task design to the Linked Open Data cloud. For doing this it uses is that the input is dynamically obtained from the LOD Algorithmic matchers for generating candidate map- sources using SPARQL Query and may be optionally pings between the entities and the LOD links. It then attached with the task input parameters. Most task tem- sends these to a decision engine that sends selective plates include a ’confidence level’ question that asks candidates to a microtask manager, which creates tasks the workers to provide their confidence level for their and publishes to the crowdsourcing platform. The de- responses. This is done as means of quality control. cision engine evaluates the crowd responses using a CrowdTruth: The CrowdTruth provides use cases probabilistic network and results are used to decide that illustrate some 14 distinct annotation templates most appropriate links for the entities. The output are across three content modalities (text, image, video) and pages with entities linked to the respective URIs on the three domains (medical, news, culture). The frame- LOD cloud. work design claims to provide its template collection CrowdLink: The CrowdLink architecture is similar as a continuously extendible library of annotation task to CrowdMap and ZenCrowd with regards to task pub- templates, which can be reused and adapted for new lishing and results retrieval. CrowdLink, uses reusable data and use cases. The implementation of CrowdTruth task profiles and task specifications for a wider range 22 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

Table 6 Comparative Analysis of Crowdsourcing Research in the Semantic web Parameter CrowdMap [85] Noy [76] Mortensen et ZenCrowd [21,22] CrowdLink [8,9] CrowdTruth [40] al. [69–73] Task Category Ontology Engineering Ontology Engineering Linked Data Management Linked Data Management Neither pure Ontology Engineering nor pure Linked Management Tasks Ontology Alignment Ontology Verification Entity Linking Schema and Instance (Semantic) Annotation (Support for Level Tasks Text, Image and Multi-media) Task Granularity Validation & Identifica- Hierarchy Verification Verify Entity Links Create New Knowledge Text Annotation (Span and Relation tion (Concepts only) Verify a Superclass- Review Knowledge Extraction) No Properties or instances subclass relationship Entity Disambiguation Event Extraction Video Annotation Domain of Study Conference Ontology Biomedical Ontologies News Articles Domain Independent Domain Independent (OAEI) Comparison of Worker Evaluations done for Evaluations done for Medical Re- performance Upper vs. DBLP, GeoCities and US lation Extraction, Newspaper Event Application Ontology Census LOD datasets Extraction BWW, SUMO, WordNet, CARO Task Representation Simple NA Static Dynamic Dynamic Task Template Simple (Limited to pairs NA NA Task Templates: Task Templates: generator) – Entity Disambiguation – Factor Span Correction – Create New Knowl- – Relation Identification edge – Relation and Direction Identifi- – Review Knowledge cation – Event Identification – Event Loc, Participants, and Time Identification

Reusable Task Tem- Yes NA Yes Yes Yes plate Use of Task Composi- NA NA NA NA Composite Tasks, made up of tion/Decomposition atomic tasks Workflow Represen- Automated NA Static Automated Automated tation (Automated vs. Static) Workflow Human and Yes No Yes Yes Yes machine computation Candidate mappings are Conceptual Workflow de- Uses SPARQL Queries combined automatically generated scribed in [73] to generate input to the Microtasks are automated Micro-tasks by Querying LOD sources Any particular data NA NA NA SPARQL Queries as part Customizable Job Configuration format of the input workflow On- and Settings API tology Driven task profiles Worker Engage- CrowdFlower Crowd (MTurk) Crowd (MTurk) Crowd (MTurk) CrowdFlower Crowd (MTurk) ment/Interaction 7 Alignment Questions Qualification Test (8 out Presenting Knowl- into one HIT to facil- of 12 to Qualify) edge to Users itate worker assignment $50 award to student with Engage Workers and resource optimization the best result Question formulation with positive and negative polarities Additional cognitive overload with negative polarity questions reduces performance Worker Performance Workers Only Turkers vs. Students Workers Workers Workers Benchmark Turkers vs. Domain Ex- perts Data Aggregation CrowdFlower Aggrega- Baysian Inference Model Agreement Vote Majority worker Agree- Sophisticated Disagreement Met- Measures tion [70] Precision and Recall mea- ment Worker Confidence rics Precision and Recall sures for evaluation based measure Utilization of Annotation Vectors measures for evaluation Precision and Recall mea- Worker and Unit Level Metrics sures for evaluation Annotation Metrics

Data Visualization NA NA NA NA Interface to support data and results Visualization Data Quality and Golden unit question Redundancy in responses: Spam Detection using 3 Simple Aggregation Harnesses worker disagreement Reliability Measures, included in each HIT 32 consecutive random an- based on majority votes. using Annotation Vectors and Spam Prevention (of 7 questions each) Random responses e.g. 23 swers. No spam prevention Media Unit Vectors for evaluating worker identical answers out of Controlled test settings Specialized Worker, Annotation performance, detecting 28 Disqualified and Unit Metrics Used spam and validation 90% Approval rating from other requestors Making question answering time consuming by placing all 28 questions in the same page Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 23 of tasks, and optionally uses SPARQL queries for the out compromising quality. They compare performance purpose of obtaining input from LOD sources. The ar- of turkers vs. students and also turkers vs. domain ex- chitecture is equally capable of using any SPARQL perts. They primarily rely on MTurk for engaging the compatible ontology source. The workflow manage- workers. The specific research questions that the au- ment engine in CrowdLink architecture includes a task thors attempt to address are crucial in particular the publishing workflow manager and a task review work- aspect of determining the suitability of using crowd- flow manager, responsible for publishing and review- sourcing for verification of ontologies in specialized ing tasks respectively. A specific feature of the task re- domains such as biomedicine. In addition, the authors view workflow is the update mechanism that directly also attempt to determine whether the workers perfor- updates the source ontology based on the evaluated re- mance change depending on how specific or generic sults and specified acceptance thresholds. This is done the ontology is. Their results suggest that crowdsourc- using the SPARQL update protocol, and makes it more ing is indeed feasible for specialized domains. convenient to incorporate into ontology engineering ZenCrowd: MTurk is used as the platform for en- methodology. gaging the workers. CrowdTruth: The CrowdTruth workflow framework CrowdLink: MTurk is used as the platform for en- is most sophisticated and extensive of all systems pre- gaging the workers. Only sandbox testing is con- sented thus far. The CrowdTruth architecture utilizes ducted. No payments are used. a range of pre-processing techniques for processing a CrowdTruth: CrowdTruth uses the CrowdFlower wide variety of input media such as text and multi- and the MTurk platforms for engaging the workers. media collections. A number of customizable job templates are support for a range of tasks. The data post- 5.2.5. Data Aggregation, Quality and Reliability processing applies a number of metrics for the analy- CrowdMap:CrowdMap relies on CrowdFlower ag- sis of the obtained results and extensive data analyt- gregation methods and uses Precision and Recall mea- ics are applied. The very emphasis of the CrowdTruth sures for evaluating the results. methodology is to harness disagreements in the crowd Noy: Primary aggregation measure used is a major- responses. The CrowdTruth framework is perhaps the ity consensus. In one version of the experiments, the only one of ones presented that allows for composite author utilize a Bayesian inference model for verifica- tasks such that a complex tasks are broken into smaller tion [70]. A number of experimental settings are tried ones. by the authors for quality control. Some of the con- Discussion Points: Most of these studies combine trol parameters include, qualification questions, ques- human and machine computation at some level. How- tion design, redundancy and disqualification, and spam ever, if we strictly restrict ourselves to to combine hu- identification. Qualification questions were employed man and machine computation in cooperative and iter- however according to [70] these did not affect the re- ative workflows, such that human tasks are adaptively sults much. They introduce redundancy in responses. generated on the fly, then this is still something that is They also disqualify the responses of the worker if 23 yet to be seen however, presents a promising direction out of 28 answers are identical. In addition, 90% ap- for research. proval rating is also enforced. The question answering 5.2.4. Worker Interaction, Engagement Strategy and process is also created to be time consuming to ensure Performance only serious participants record their responses. These CrowdMap: CrowdMap uses the CrowdFlower sys- are some of the ways that spam has been attempted tem to engage workers. Researchers agree that worker to be prevented. However, as authors note these tests responses can depend on the nature of the task design alone are not sufficient. and optimal task design depending on the nature of the ZenCrowd: The Agreement voting technique is em- problem to be solved is always crucial. CrowdMap at- ployed in this study. 5 different workers are required tempts the strategy of packing 7 Alignment questions to select from amongst the proposed mappings and the into one HIT or job to facilitate worker assignment and URIs with at least 2 votes are selected as valid links. resource optimization. A technique of blacklisting bad workers (or spammer) Noy: Noy and collegue experiment with a number is used. If a worker randomly and rapidly selects the of experimental setups to establish the effectiveness of links, its considered noise in the system and the worker crowdsourcing methods for ontology verification with- is blacklisted. Consecutive 3 bad answers in the train- 24 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey ing phase are considered enough to identify the worker representations have been used or there is no evidence as a spammer. of any specific task representation being devised. Simi- CrowdLink: The experiments are done under con- larly, it is not evident if any well defined task templates trolled settings, therefore the spam element is not taken have been developed for ontology engineering tasks. into consideration. The task design includes a confi- Another significant limitation that can be seen is the dence measure which is obtained from the workers lack of formal workflow engineering. The workflows to indicate the worker confidence into consideration. so far have minimal formal representation and are sim- Simple agreement based aggregation is done. plistic in nature. There is considerable need for more CrowdTruth: CrowdTruth has an extensive Data An- efforts directed in this direction. alytics component. There are several annotation (disagreement) metrics used [5,6,100]. The CrowdTruth Framework aggregates the annotations of multiple 6. Application of Semantic Web Techniques to workers across different media units (e.g. text, image Facilitate Human Computation and video) using Annotation Vectors, and MediaUnit Vectors. The Annotation vector records the responses Research towards this direction, whereby there is of the workers and the similarity is computed using evidence of semantic web techniques applied to im- cosine similarity measure. The MediaUnit vector ac- prove the state of human computation systems are counts for all worker submissions on a unit for a given fewer. Most notable work in this domain is in the task. Analysis of ambiguity to prevent spam is done direction of generating human-computer micro-task across each system component i.e. worker, annotation workflows using domain ontologies Luz et al. [58]. and unit, resulting in ’Worker Metrics’, ’Annotation There is considerable interest in resolution of com- Metrics’ and ’Unit Metrics’. plex tasks that require the cooperation of human and Worker metrics determine the measure of disagree- machine participants has emerged. The approach pro- ment at the level of the worker in order to distinguish posed by Luz et al. [58], consists of a semi-automatic spam from a high quality response. Various measures generation environment for human-computer micro- such as worker-unit disagreement, worker-worker dis- task workflows from domain ontologies. The inher- agreement, and average annotations per unit are used. ent process relies in the domain expertise of the re- The unit metrics are used to determine the clarity of quester to supervise the automatic interpretation of the the input unit given to the crowd. To identify ambigu- domain ontology. According to the authors, the un- ous units, measures such as unit annotation score, and structured nature of micro-tasks in terms of domain unit clarity are used. The Annotation metrics aim to representation makes it difficult (i) for task requesters measure the quality of pre-defined annotation types, not familiar with the crowdsourcing platform to build in attempt to distinguish between disagreement result- complex micro-task workflows and (ii) to include ma- ing from either low quality workers or disagreement chine workers in the workflow execution process. As resulting from badly designed task. The specific mea- claimed by the authors, the structure and semantics sures used in this category are annotation similarity, of the domain ontology provides a common ground annotation ambiguity, annotation clarity and annota- for understanding and enhances human-computer co- tion frequency. operation. The study provides an interesting dimension Together these metrics provide an in-depth perspec- about the use of ontologies for the purpose of workflow tive to improve the crowdsourcing task design and generation and may inspire future relevant studies. analysis of results. The CrowdTruth is the only frame- One evidence of utilization of ontologies for facil- work which also makes use of Data Visualization of itating human computation systems is for provenance the annotation metrics once aggregated. management. The vocabulary SC-PROV, a provenance vocabulary for social computation [63] is an impor- 5.3. Discussion tant step in managing the provenance information and thus contributing to enhancing the transparency in hu- As illustrated in the Table 6, the granularity of tasks man computation systems. This is crucial for enabling that has been subject to crowdsourcing in the recent decision making about the reliability of participants studies are fairly at low level of granularity. There is and quality of the generated solutions. Baillie et al., little evidence of task composition or decomposition, [7], while attempting to solve the issue of quality rea- since most tasks are fairly simple. Either simple task soning for the semantic web also highlight the im- Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 25 portance of obtaining provenance information in case fostering user participation, deriving semantic data, ef- of crowdsourced knowledge followed by enhancement ficient distribution of labor and scalability and perfor- measures to ensure quality. mance to name a few. Some of these overlap with the other genres in human computation, however, semantic web games have their own particular challenges such 7. Discussion and Analysis as game design, game scenarios and user engagement. 7.1.2. Methodological Considerations In this paper we set out to examine the application While on one hand the domain of crowdsourcing has of human computation methods and crowdsourcing to matured over the years, however, its intersection with the domain of semantic web. In particular, we first ana- ontology engineering has yet to see a formal methodol- lyzed and presented the need for human contribution in ogy taking shape. [114] highlight the need for method- the semantic content creation. We dealt with this in two ology and best practices, while moving away from iso- specific categories i.e. ontology engineering and linked lated approaches. open data management. We also examined the key human computation genres that have been applied to the 7.1.3. Integrated Tool Support for Human semantic web tasks. We also provided an in-depth clas- Computation Workflows sification of these tasks according the the examined It has to be recognized that a significant limitation genres. We also provided brief insights into the auto- that can be felt is integrated tool support for semantic mated tool support for human computation tasks in the web specific human computation and crowdsourcing. semantic web. Another key contribution of the paper Recent work by Hanika, Wohlgenannt and collegues was the collective intelligence genome applied to the have addressed this limitation [33,34,114] by provid- semantic web case studies. The genome helps analyzing a plugin for the popular ontology development tool ing the essential threads of composition that require Protégé. This is a crucial advancement towards mea- thought and effort in conceiving a human computation sures that aim to integrate crowdsourcing into ontol- driven application. We also carried out a dedicated dis- ogy engineering practices. However, this is only the cussion focusing on those state of the art systems that beginning and there is much room for further research have attempted to design and implement workflow sys- including human computation workflows in ontology tems for driving the semantic web tasks using the hu- engineering, quality of crowdsourcing results and the man computation methods, since we strongly feel, the large scale application and usability of such plugins. future will see more of such systems, with more ma- The results of these tools may vary greatly between ture and well grounded design principles. Before con- different tasks (depending on the type of task and the cluding the paper, we highlight some of the key out- difficulty of the domain). They also show sensitivity standing challenges which present areas for further re- to the timing when the task is crowdsourced and the search. responses obtained from the available workers [114]. Most applications themselves are single purpose and 7.1. Key Outstanding Challenges standalone. The further success demands integrated tool support in particular with existing semantic web Different genres of human computation present dif- tools to realize the complete potential of crowdsourc- ferent challenges. We briefly discuss the challenges ing in particular and human computation in general. faced by semantic GWAPs and focus more on the chal- Kondreddi et al, [47] have proposed methods to lenges faced in the domain of mechanized labor or hu- combine information extraction approaches with human computation systems in general. man computation for knowledge acquisition, and claim to reduce to cost of human computation effort. Such 7.1.1. Challenges in Semantic GWAPs efforts are claimed to be ’game changers’ in the new Siorpaes and Hepp [95] provide some key chal- generation of systems that combine power of humans lenges that are faced when creating semantic GWAPs, and machines. including, task identification in the semantic content creation, designing game scenarios, designing a us- 7.1.4. Aggregating Answers able and attractive interface, identifying reusable bod- One of the key issues faced when inputs from a large ies of knowledge, preventing cheating, avoiding pit- crowd is obtained is that of aggregating answers in a falls, e.g. unintentional agreement on wrong choices, quality manner. Lopez et al., [57] argue that merging 26 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey and ranking of answers in the semantic web domain ’Global Brain’ is deeply different from programming that are obtained from different sources, similar to wis- traditional computers [10]. dom of the crowds tend to produce higher precision. There is need for more well defined quality measures 7.1.9. Harnessing Crowds vs. Experts: Choosing the especially in specialized domains. Right Crowd Crowdsourcing simple (independent) vs. complex 7.1.5. Adequately Utilizing Crowdsourced (interdependent tasks) presents a challenge especially Knowledge and Annotations to semantic web researchers. Research evidence sug- Often times it becomes a challenge in itself to utilize gests that there is strong correlation between the na- the crowdsourced knowledge especially in those tasks ture of task, its difficulty, skill and knowledge required where domain specific annotations are needed. Thus, to the accuracy obtained from crowdworkers [77,78]. workflows and techniques emphasizing such adequate Although, crowdsourcing is promising, however for utilization need to be thought of. As an example, 3DSA domain specific tasks, it needs to be establish which [116] is a system that utilizes rule based reasoning to task is suitable for which kind of crowd. The idea of classify crowdsourced annotations. nichesourcing [20] or expert sourcing [114] and expert 7.1.6. Design of (Optimal) Semantically Engineered finding [14] is being proposed for knowledge intensive Workflows tasks. These ideas bring their own challenges such as Whereas a combination of human and computa- adequate task distribution and quality assurance. Some tional intelligence is often likely to yield superior re- efforts have attempted to tackle skill based task match- sults [89] , the design of an optimal workflow depends ing or task recommendation such as [30], [118] and on various dimensions, such as the type of the task and [119] may serve to provide useful insights in this re- the degree to which it can be automated, the amount of gard. data for which the task is expected to be (repeatedly) executed, and the (estimated) availability of capable 7.1.10. Compromise between Quality and Complexity workforce potentially interested in en-gaging with the of Tasks task [43]. Luz et. al [59] highlights the need for a struc- Simperl and Wolger et al., [93] highlight that the tured and semantically enriched representation of the area of human computation in semantic web engineer- micro-task workflow and data to allow for a better in- ing and especially linked data management is in its tegration of human computation and machine compu- early stages and more research remains to be done tation efforts. The use of ontologies as means of rep- in order to achieve the overall vision. They highlight resenting task profiles is also suggested by [8,9]. One the constrains that result in the complexity of the such effort of representing micro-tasks using ontolo- tasks that can be feasibly undertaken and the domains gies and their dependencies through the specification for which knowledge can be reliably collected from of their domain (input, output and context) has been a mass audience, when a mass userbase is targeted presented by [58]. More efforts to this end are expected against tasks and tools designed for experts. This in- to emerge. Combining human and algorithmic compu- troduces the need for specific evaluation and quality tations needs more work. assurance mechanisms. 7.1.7. Motivating Users and the Role of Incentives Want et al., [113] also highlight the importance of It has yet to be seen what type of incentives, plat- Data complexity and specificity of the tasks in appro- form, games, rules, systems and architectures will priate task design and management. work best for human computation on the semantic web 7.1.11. Quality Measurement [95]. In order to benefit from network effects and pos- Quality management for crowdsourcing is critical. itive externalities, end-user semantic content author- Techniques for accurately estimating the quality of ing technology needs to offer an appropriate set of in- workers is pivotal for the success of crowdsourcing re- centives in order to stimulate and reward users partici- search [41]. Developing common grounds for quality pation, generating in massive production of useful se- assessment is crucial when obtaining semantic annota- mantic content [97] . tions from the crowd [39] and result in annotator di- 7.1.8. Dealing with Diversity versity that can be harnessed. Aroyo and Welty present Dealing with Motivational, Cognitive and Error Di- a series of works in order to harness disagreement in versity: Because people are involved, programming the crowdsourcing gold standards [4–6]. Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 27

7.1.12. Constraints to Microtask Approach 7.2.4. New Programming Models and Frameworks Decomposability, verifiability and expertise are Similar to the ideas of programming the ’Global three crucial factors that constrain the microtask de- Brain’, programming the ’Global Brain Semantic sign [91]. Techniques that resolve to achieve a suit- Web’ [10] demands the need for programming lan- able compromise are need to see the research through guages and frameworks such as the one presented in a state of maturity. Several parameters can have an im- [68]. Exploring iterative and parallel human compu- pact the on the approach used and the resulting quality tation processes [55] is another prospective idea for such as effect of context on worker performance, effect further research. of more domain specific tasks on the accuracy of the 7.2.5. Design of Better User Interfaces crowdsourcing model, improving worker questions by New visualizations and usable design is needed for fine tuning the questions, and optimizing the number better user engagement. of questions issued to the user [85]. 7.2.6. State Space in Human Computation Systems 7.1.13. Scalability According to Difranzo and Hendler [23], ’state Scalability in both reading and writing semantic space’ in a human computation system is the collec- data. It is yet to be seen what a truly web scale seman- tion of knowledge, artifacts and skills of both the hu- tic human computation system will look like [23]. man users and the computer system that help define the state, or current stage at a given time, of the hu- 7.2. Open Research Issues for Semantics Driven man computation system. This state space can often Human Computation Systems be very messy, disjointed, incomplete or inconsistent. The semantic web could provide this common plat- While the view of challenges presented in the pre- form and medium for representing knowledge that per- ceding section itself sheds light on the current and sists despite the asynchronous behaviors of the human open research directions towards incorporating human participants. More research is needed to explore how computation methods into semantic web processes, on this could work, how best to represent this knowledge, a parallel note, we also enlist a few directions to be and what advantages this could bring to future human possibly taken up for further research, towards aug- computation systems. menting semantic content to better design and facilitate human computation systems: 7.3. Some Reflections 7.2.1. Design of New Vocabularies Most of the current semantic web research has New ontologies and vocabularies will need to be de- primarily focused on studying the feasibility and veloped to help manage and link these human compu- amenability of applying crowdsourcing and human tation systems together[23]. computation techniques to the semantic web tasks. 7.2.2. Semantic User Management There is much work that remains to be done. Carletti Users can easily sign on into new systems, and have et al., [16] highlight an essential challenge about the their points and reputation follow them [23]. separation that occurs between crowdsourced activities and organizational workflows. Although, the dis- 7.2.3. Meta-data and Provenance Management cussion is in the context of digital humanities, the se- The idea of trust, reliability and ensuring quality mantic web landscape appears to be facing a similar of crowdsourced contributions demand that prove- challenge. While there has been significant interest and nance information be essentially embedded within contribution on these fronts, it still remains to be es- workflows and responses and be included for relevant tablished how much of the crowd-sourced contribu- analysis[23]. tions have really made a real impact. Crowdsourcing Recent proposition by Markovic and collegues high- has also shown to contribute to innovation [3]. How- light the prevenance perspective [63–67] when deal- ever the key consideration for crowdsourcing has to be ing with the crowd and linked data. Provenance infor- the goals for the crowdsourcing activity [3]. The inter- mation is critical to be maintained and therefore ade- leaving of human, machine, and semantics even have quate measures for managing provenance becomes a the potential to overcome some of the issues currently key challenge. surrounding Big Data [74]. 28 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

8. Conclusions [9] Amna Basharat, I. Budak Arpinar, Shima Dastgheib, Ugur Kursuncu, Krys Kochut, and Erdogan Dogdu. Semantically In this paper we presented a review of the chal- enriched task and workflow automation in crowdsourcing for linked data management. International Journal of Semantic lenges that the semantic web domain has faced es- Computing, 8(4), 2014. pecially in terms of the need for human intervention, [10] Abraham Bernstein. The global brain semantic web - in- and in this light analyzed the intersection of semantic terleaving human-machine knowledge and computation. In web and the human computation paradigm. Two fold ISWC2012 Workshop on What will the Semantic Web Look Like 10 Years From Now?, pages 1–6, Boston, MA, 2012. approach towards understanding this intersection pro- [11] Abraham Bernstein, Mark Klein, and Thomas W Malone. vided interesting insights. Both semantic web and hu- Programming the global brain. Communications of the ACM, man computation have potential to benefit from each 55(5):41–43, 2012. ISSN 0001-0782. other. While there seems to be more interest by the [12] Christian Bizer, Tom Heath, and Tim Berners-Lee. Linked semantic web community to benefit from the human data-the story so far, 2009. [13] Kalina Bontcheva, Ian Roberts, Leon Derczynski, and Do- computation paradigm, there is evidence of the latter minic Rout. The gate crowdsourcing plugin: Crowdsourcing being enhanced using the former as well. The compre- annotated corpora made easy. In Proc. EACL, 2014. hensive view of the promises and challenges offered [14] Alessandro Bozzon, Marco Brambilla, Stefano Ceri, Matteo by the successful synergy of semantic web and human Silvestri, Giuliano Vesci, and Politecnico Milano. Choosing the right crowd : Expert finding in social networks. pages computation as discussed in this paper indicates that 637–648. ISSN 9781450315975. this is a thriving research direction and is expected to [15] Paul Buitelaar, Philipp Cimiano, and Bernardo Magnini. On- grow further. We hope that this review will serve as tology learning from text: An overview, volume 123. 2005. basis for exploring newer threads of synergy between [16] L. Carletti, G. Giannachi, and D. McAuley. Digital human- the semantic web and human computation research, re- ities and crowdsourcing: An exploration. Museums and the Web, Portland, Oregon., 2013. sulting in the creation of better applications and ap- [17] Irene Celino, Simone Contessa, Marta Corubolo, Daniele proaches that advance both domains. Dell’Aglio, Emanuele Della Valle, Stefano Fumeo, Thorsten KrÃijger, and Thorsten KrÃijger. Urbanmatch-linking and improving smart cities data. In LDOW, 2012. [18] Irene Celino, Simone Contessa, Marta Corubolo, Daniele References DellâA˘ ZAglio,´ Emanuele Della Valle, Stefano Fumeo, and Thorsten KrÃijger. Linking smart cities datasets with hu- [1] M. Acosta, A. Zaveri, E. Simperl, D. Kontokostas, S. Auer, man computationâA¸Sthecase˘ of urbanmatch. In The Seman- and J. Lehmann. Crowdsourcing linked data quality assess- tic WebâA¸SISWC2012˘ , pages 34–49. Springer, 2012. ISBN ment. Semantic Web - Iswc 2013, Part Ii, 8219:260–276, 3642351727. 2013. ISSN 0302-9743. [19] Michelle Cheatham and Pascal Hitzler. Conference v2. 0: An [2] Maribel Acosta, Elena Simperl, Fabian FlÃ˝uck,and Barry uncertain version of the oaei conference benchmark. In The Norton. A sparql engine for crowdsourcing query process- Semantic WebâA¸SISWC2014˘ , pages 33–48. Springer, 2014. ing using microtasksâA¸Stechnicalreport.˘ Institute AIFB, KIT, [20] Victor De Boer, Michiel Hildebrand, Lora Aroyo, Pieter Karlsruhe, 2012. De Leenheer, Chris Dijkshoorn, Binyam Tesfa, and Guus [3] Tanja Aitamurto, Aija Leiponen, and Richard Tee. The Schreiber. Nichesourcing: Harnessing the power of crowds of promise of idea crowdsourcingâA¸Sbenefits,contexts,˘ limita- experts. In Knowledge Engineering and Knowledge Manage- tions. Whitepaper for Nokia Ideas Project, 2011. ment, pages 16–20. Springer, 2012. [4] Lora Aroyo and Chris Welty. Harnessing disagreement in [21] G. Demartini, D. E. Difallah, and P. Cudre-Mauroux. Large- crowdsourcing a relation extraction gold standard. Technical scale linked data integration using probabilistic reasoning report, Tech. Rep. RC25371 (WAT1304-058), IBM Research, and crowdsourcing. Vldb Journal, 22(5):665–687, Oct 2013. 2013. ISSN 1066-8888. [5] Lora Aroyo and Chris Welty. Measuring crowd truth for med- [22] Gianluca Demartini. Zencrowd: leveraging probabilistic rea- ical relation extraction, 2013. soning and crowdsourcing techniques for large-scale entity [6] Lora Aroyo and Chris Welty. Crowd truth: Harnessing dis- linking. Proceedings of the 21st âA˛e˘ , pages 469–478, 2012. agreement in crowdsourcing a relation extraction gold stan- ISSN 9781450312295. dard. WebSci2013. ACM, 2013 2013. [23] Dominic DiFranzo and James Hendler. The semantic web and [7] Chris Baillie, Peter Edwards, and Edoardo Pignotti. Qual- the next generation of human computation. In Handbook of ity reasoning in the semantic web. In The Semantic We- Human Computation, pages 523–530. Springer, 2013. ISBN bâA¸SISWC2012˘ , pages 383–390. Springer, 2012. 1461488052. [8] Amna Basharat, I. Budak Arpinar, Shima Dastgheib, Ugur [24] Anhai Doan, Raghu Ramakrishnan, and Alon Y Halevy. Kursuncu, Krys Kochut, and Erdogan Dogdu. Crowdlink: Crowdsourcing systems on the world-wide web. Communi- Crowdsourcing for large-scale linked data management. cations of the ACM, 54(4):86–96, 2011. ISSN 0001-0782. pages 227–234. IEEE, 2014. Semantic Computing (ICSC), [25] Anca Dumitrache, Lora Aroyo, Chris Welty, Robert-Jan Sips, 2014 IEEE International Conference on. and Anthony Levas. âAIJdr.˘ detectiveâA˘ I:˙ combining gamifi- Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 29

cation techniques and crowdsourcing to create a gold standard 64–67. ACM, 2010. ISBN 145030222X. in medical text. Crowdsourcing, the, Semantic, Web, 2013. [42] Prateek Jain, Pascal Hitzler, Peter Z Yeh, Kunal Verma, and [26] Kai Eckert, Mathias Niepert, Christof Niemann, Cameron Amit P Sheth. Linked data is merely more data. In AAAI Buckner, Colin Allen, and Heiner Stuckenschmidt. Crowd- Spring Symposium: linked data meets artificial intelligence, sourcing the assembly of concept hierarchies. Proceedings of volume 11, 2010. the 10th annual joint conference on Digital libraries - JCDL [43] A. Kittur, E. H. Chi, and B. Suh. Crowdsourcing user stud- ’10, page 139, 2010. ISSN 9781450300858. ies with mechanical turk. Chi 2008: 26th Annual Chi Con- [27] E. Estelles-Arolas and F. Gonzalez-Ladron-de Guevara. To- ference on Human Factors in Computing Systems Vols 1 and wards an integrated crowdsourcing definition. Journal of In- 2, Conference Proceedings, pages 453–456, 2008. ISSN formation Science, 38(2):189–200, Apr 2012. ISSN 0165- 9781605580111. 5515. [44] Shirlee-ann Knight and Janice M. Burn. Developing a frame- [28] Dieter Fensel and Mark A Musen. The semantic web: a brain work for assessing information quality on the world wide for humankind. IEEE Intelligent systems, 16(2):24–25, 2001. web. Informing Science: International Journal of an Emerg- ISSN 1094-7167. ing Transdiscipline, 8(5):159–172, 2005. Informing Science [29] Michael J Franklin, Donald Kossmann, Tim Kraska, Sukriti Institute. Ramesh, and Reynold Xin. Crowddb: answering queries with [45] Magnus Knuth and Harald Sack. Data cleansing consolida- crowdsourcing. In Proceedings of the 2011 ACM SIGMOD tion with patchr. The Semantic Web: ESWC 2014 Satellite International Conference on Management of data, pages 61– Events, pages 231–235. 72. ACM, 2011. [46] Magnus Knuth, Johannes Hercher, and Harald Sack. [30] David Geiger and Martin Schader. Personalized task recom- Collaboratively patching linked data. arXiv preprint mendation in crowdsourcing information systemsâA˘Tcurrentˇ arXiv:1204.2715, 2012. state of the art. Decision Support Systems, 65:3–16, 2014. [47] Sarath Kumar Kondreddi, Peter Triantafillou, and Gerhard [31] Thomas R Gruber. A translation approach to portable on- Weikum. Combining information extraction and human com- tology specifications. Knowledge acquisition, 5(2):199–220, puting for crowdsourced knowledge acquisition. In Data En- 1993. gineering (ICDE), 2014 IEEE 30th International Conference [32] Thomas R Gruber. Toward principles for the design of on- on, pages 988–999. IEEE, 2014. tologies used for knowledge sharing? International journal [48] D. Kontokostas, A. Zaveri, S. Auer, and J. Lehmann. of human-computer studies, 43(5):907–928, 1995. Triplecheckmate: A tool for crowdsourcing the quality assess- [33] Florian Hanika, Gerhard Wohlgenannt, and Marta Sabou. The ment of linked data. Knowledge Engineering and the Seman- ucomp protÃl’gÃl’ plugin for crowdsourcing ontology vali- tic Web (Kesw 2013), 394:265–272, 2013. ISSN 1865-0929. dation. [49] Markus Krause, Aneta Takhtamysheva, Marion Wittstock, [34] Florian Hanika, Gerhard Wohlgenannt, and Marta Sabou. The and Rainer Malaka. Frontiers of a paradigm: exploring hu- ucomp protÃl’gÃl’ plugin: Crowdsourcing enabled ontology man computation with digital games. pages 22–25. ACM, engineering. In Knowledge Engineering and Knowledge 2010. Proceedings of the acm sigkdd workshop on human Management, pages 181–196. Springer International Publish- computation. ing, 2014. ISBN 3319137034. [50] Anand P Kulkarni, Matthew Can, and Bjoern Hartmann. [35] Tom Heath and Christian Bizer. Linked data: Evolving the Turkomatic: automatic recursive task and workflow design web into a global data space. Synthesis lectures on the seman- for mechanical turk. In CHI’11 Extended Abstracts on Hu- tic web: theory and technology, 1(1):1–136, 2011. man Factors in Computing Systems, pages 2053–2058. ACM, [36] Francis Heylighen. Conceptions of a global brain: an histor- 2011. ical review. Evolution: Cosmic, Biological, and Social, eds. [51] Yen-ling Kuo, Jong-Chuan Lee, Kai-yang Chiang, Rex Wang, Grinin, LE, Carneiro, RL, Korotayev AV, Spier F, pages 274– Edward Shen, Cheng-wei Chan, and Jane Yung-jen Hsu. 289, 2011. Community-based game design: experiments on social games [37] J Howe. The rise of crowdsourcing. Wired magazine, pages for commonsense data collection. In Proceedings of the ACM 1–5, 2006. SIGKDD Workshop on Human Computation, pages 15–22. [38] Jeff Howe. Crowdsourcing: How the power of the crowd is ACM, 2009. ISBN 1605586722. driving the future of business. Random House, 2008. ISBN [52] Jens Lehmann, Tri Quan Nguyen, and Timofey Ermilov. Can 1905211112. we create better links by playing games? In Semantic Com- [39] Oana Inel, Lora Aroyo, Chris Welty, and Robert-Jan Sips. puting (ICSC), 2013 IEEE Seventh International Conference Domain-independent quality measures for crowd truth dis- on, pages 322–329. IEEE, 2013. agreement. Detection, Representation, and Exploitation of [53] Henry Lieberman, Dustin Smith, and Alea Teeters. Com- Events in the Semantic Web, page 2, 2013. mon consensus: a web-based game for collecting common- [40] Oana Inel, Khalid Khamkham, Tatiana Cristea, Anca Dumi- sense goals. 2007. ACM Workshop on Common Sense for trache, Arne Rutjes, Jelle van der Ploeg, Lukasz Romaszko, Intelligent Interfaces. Lora Aroyo, and Robert-Jan Sips. Crowdtruth: Machine- [54] Greg Little, Lydia B Chilton, Max Goldman, and Robert C human computation framework for harnessing disagreement Miller. Turkit: tools for iterative tasks on mechanical turk. In in gathering annotated data. In The Semantic WebâA¸SISWC˘ Proceedings of the ACM SIGKDD workshop on human com- 2014, pages 486–504. Springer, 2014. putation, pages 29–30. ACM, 2009. [41] Panagiotis G Ipeirotis, Foster Provost, and Jing Wang. Quality [55] Greg Little, LB Chilton, Max Goldman, and RC Miller. Ex- management on amazon mechanical turk. In Proceedings of ploring iterative and parallel human computation processes. the ACM SIGKDD workshop on human computation, pages âA˛e˘ on human computation, pages 68–76, 2010. ISSN 30 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

9781450302227. [74] Tom Narock and Pascal Hitzler. Crowdsourcing semantics [56] Greg Little, Lydia B Chilton, Max Goldman, and Robert C for big data in geoscience applications. In 2013 AAAI Fall Miller. Turkit: human computation algorithms on mechanical Symposium Series, 2013. turk. In Proceedings of the 23nd annual ACM symposium on [75] Mathias Niepert, Cameron Buckner, and Colin Allen. Work- User interface software and technology, pages 57–66. ACM, ing the crowd: Design principles and early lessons from the 2010. social-semantic web. 2009. Proceedings of the Workshop [57] Vanessa Lopez, Andriy Nikolov, Miriam Fernandez, Marta on Web 3.0: Merging Semantic Web and Social Web at ACM Sabou, Victoria Uren, and Enrico Motta. Merging and rank- Hypertext. ing answers in the semantic web: The wisdom of crowds. In [76] Natalya F Noy, Jonathan Mortensen, Mark A Musen, and The semantic web, pages 135–152. Springer, 2009. Paul R Alexander. Mechanical turk as an ontology engineer? [58] Nuno Luz, Nuno Silva, and Paulo Novais. Generating -using microtasks as a component of an ontology-engineering human-computer micro-task workflows from domain ontolo- workflow, 2013. gies. Springer, 2014. [77] Jasper Oosterman, Alessandro Bozzon, Geert-Jan Houben, [59] Nuno Luz, Nuno Silva, and Paulo Novais. A survey of task- Archana Nottamkandath, Chris Dijkshoorn, Lora Aroyo, oriented crowdsourcing. Artificial Intelligence Review, pages Mieke H. R. Leyssen, and Myriam C. Traub. Crowd vs. ex- 1–27, 2014. perts: nichesourcing for knowledge intensive tasks in cultural [60] Thomas W Malone, Robert Laubacher, and Chrysanthos Del- heritage. pages 567–568. International World Wide Web Con- larocas. Harnessing crowds: Mapping the genome of collec- ferences Steering Committee, 2014. Proceedings of the com- tive intelligence. 2009. panion publication of the 23rd international conference on [61] Thomas W Malone, Robert Laubacher, and Chrysanthos Del- World wide web companion. larocas. The collective intelligence genome. IEEE Engineer- [78] Jasper Oosterman, Archana Nottamkandath, Chris Dijk- ing Management Review, 38(3):38, 2010. ISSN 0360-8581. shoorn, Alessandro Bozzon, Geert-Jan Houben, and Lora [62] Thomas Markotschi and Johanna VÃ˝ulker. Guess- Aroyo. Crowdsourcing knowledge-intensive tasks in cultural what?!âA¸Shumanintelligence˘ for mining linked data. 2010. heritage. pages 267–268. ACM, 2014. Proceedings of the [63] Milan Markovic, Peter Edwards, and David Corsar. Sc-prov: 2014 ACM conference on Web science. A provenance vocabulary for social computationâN´ ˛E. [79] Ei Pa Pa Pe-Than, Dion Hoe-Lian Goh, and Chei Sian Lee. A [64] Milan Markovic, Peter Edwards, David Corsar, and Jeff Z typology of human computation games: an analysis and a re- Pan. The crowd and the web of linked data: A provenance per- view of current games. Behaviour & Information Technology, spective. In AAAI Spring Symposium: Wisdom of the Crowd, (ahead-of-print):1–16, 2013. ISSN 0144-929X. 2012. [80] A Peres, M Lopes, and O Corcho. Ontological Engineering: [65] Milan Markovic, Peter Edwards, David Corsar, and Jeff Z with Examples from the Areas of knowledge Management, e- Pan. Demo: Managing the provenance of crowdsourced Commerce and Semantic Web. Springer„ 2004. disruption reports. In Provenance and Annotation of Data [81] AJ Quinn and BB Bederson. Human computation: a survey and Processes, pages 209–213. Springer, 2012. ISBN and taxonomy of a growing field. âA˛eConference˘ on Human 3642342213. Factors in Computing âA˛e˘ , pages 1403–1412, 2011. ISSN [66] Milan Markovic, Peter Edwards, and David Corsar. Utilising 9781450302678. provenance to enhance social computation. In The Semantic [82] Antonio J. Roa-Valverde. Combining gamification, crowd- WebâA¸SISWC2013˘ , pages 440–447. Springer, 2013. ISBN sourcing and semantics for leveraging linguistic open data. 3642413374. [83] M. Sabou, A. Scharl, and M. Fols. Crowdsourced knowledge [67] Milan Markovic, Peter Edwards, and David Corsar. A role acquisition: Towards hybrid-genre workflows. International for provenance in social computation. CrowdSem, pages 1–3, Journal on Semantic Web and Information Systems, 9(3):14– 2013. 41, 2013. ISSN 1552-6283. [68] Patrick Minder and Abraham Bernstein. Crowdlang: A pro- [84] Marta Sabou, Kalina Bontcheva, and Arno Scharl. Crowd- gramming language for the systematic exploration of human sourcing research opportunities: lessons from natural lan- computation systems. In Social Informatics, pages 124–137. guage processing. In Proceedings of the 12th Interna- Springer, 2012. ISBN 3642353851. tional Conference on Knowledge Management and Knowl- [69] J. M. Mortensen. Crowdsourcing ontology verification. Se- edge Technologies, page 17. ACM, 2012. mantic Web - Iswc 2013, Part Ii, 8219:448–455, 2013. ISSN [85] Cristina Sarasua, Elena Simperl, and Natalya F Noy. 0302-9743. Crowdmap: Crowdsourcing ontology alignment with micro- [70] J. M. Mortensen, M. A. Musen, and N. F. Noy. Crowdsourc- tasks. Springer, 2012. ing the verification of relationships in biomedical ontologies. [86] Arno Scharl, Marta Sabou, and Michael FÃ˝uls.Climate quiz: AMIA Annu Symp Proc, 2013:1020–9, 2013. ISSN 1942- a web application for eliciting and validating knowledge from 597X (Electronic) 1559-4076 (Linking). social networks. pages 189–192. ACM, 2012. Proceedings of [71] JM Mortensen, MA Musen, and NF Noy. Developing crowd- the 18th Brazilian symposium on Multimedia and the web. sourced ontology engineering tasks: An iterative process. [87] Jakub Simko, Michal Tvarozek, and Maria Bielikova. Little Crowdsourcing, the, Semantic, Web, pages 1–10, 2013. search game: term network acquisition via a human compu- [72] Jonathan M Mortensen, Paul R Alexander, Mark A Musen, tation game. pages 57–62. ACM, 2011. Proceedings of the and Natalya F Noy. Crowdsourcing ontology verification. 22nd ACM conference on Hypertext and hypermedia. [73] Jonathan M Mortensen, Mark A Musen, and Natalya F Noy. [88] Elana Simperl, Stephan Wölger, Stefan Thaler, and Katha- Ontology quality assurance with the crowd, 2013. rina Siorpaes. Learning ontologies via games with a purpose, 2014. Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey 31

[89] Elena Simperl. Crowdsourcing semantic data management: word games for semantic data collection. pages 533–542. As- challenges and opportunities. page 1. ACM, 2012. Proceed- sociation for Computational Linguistics, 2008. Proceedings ings of the 2nd International Conference on Web Intelligence, of the Conference on Empirical Methods in Natural Language Mining and Semantics. Processing. [90] Elena Simperl and Markus Luczak-RÃ˝usch. Collaborative [107] Luis Von Ahn. Games with a purpose. Computer, 39(6):92– ontology engineering: a survey. The Knowledge Engineering 94, 2006. ISSN 0018-9162. Review, 29(01):101–131, 2014. Cambridge Univ Press. [108] Luis Von Ahn. Human computation. In Design Automation [91] Elena Simperl, Barry Norton, and Denny Vrandecic. Crowd- Conference, 2009. DAC’09. 46th ACM/IEEE, pages 418–419. sourcing tasks in linked data management. IEEE, 2009. ISBN 1605584975. [92] Elena Simperl, Maribel Acosta, and Barry Norton. A semanti- [109] Luis Von Ahn and Laura Dabbish. Labeling images with a cally enabled architecture for crowdsourced linked data man- computer game. In Proceedings of the SIGCHI conference on agement. In CrowdSearch, pages 9–14. Citeseer, 2012. Human factors in computing systems, pages 319–326. ACM, [93] Elena Simperl, Stephan WÃ˝ulger, Stefan Thaler, Barry Nor- 2004. ISBN 1581137028. ton, and Tobias BÃijrger. Combining human and computation [110] Luis Von Ahn and Laura Dabbish. Designing games with a intelligence: the case of data interlinking tools. International purpose. Communications of the ACM, 51(8):58–67, 2008. Journal of Metadata, Semantics and Ontologies, 7(2):77–92, ISSN 0001-0782. 2012. ISSN 1744-2621. [111] Luis Von Ahn, Mihir Kedia, and Manuel Blum. Verbosity: a [94] Elena Simperl, Maribel Acosta, and Fabian FlÃ˝uck.Knowl- game for collecting common-sense facts. pages 75–78. ACM, edge engineering via human computation. Springer, 2013. 2006. Proceedings of the SIGCHI conference on Human Fac- [95] Katharina Siorpaes and Martin Hepp. Games with a purpose tors in computing systems. for the semantic web. IEEE Intelligent Systems, 23(3):50–60, [112] JÃ˝urg Waitelonis, Nadine Ludwig, Magnus Knuth, and Har- 2008. ISSN 1541-1672. ald Sack. Whoknows? evaluating linked data heuristics with a [96] Katharina Siorpaes and Martin Hepp. Ontogame: Weaving quiz that cleans up dbpedia. Interactive Technology and Smart the semantic web by online games. Springer, 2008. ISBN Education, 8(4):236–248, 2011. Emerald Group Publishing 3540682333. Limited. [97] Katharina Siorpaes and Elena Simperl. Incentives, moti- [113] Aobo Wang, Cong Duy Vu Hoang, and Min-Yen Kan. Per- vation, participation, games: Human computation for linked spectives on crowdsourcing annotations for natural language data. 2010. CEUR Proceedings of the Workshop on Linked processing. Language resources and evaluation, 47(1):9–31, Data in the Future Internet at the Future Internet Assembly, 2013. Ghent, Belgium. [114] Gerhard Wohlgenannt, Marta Sabou, and Florian Hanika. [98] Katharina Siorpaes and Elena Simperl. Human intelligence Crowd-based ontology engineering with the ucomp pro- in the process of semantic content creation. World Wide Web, tÃl’gÃl’ plugin. 13(1-2):33–59, 2010. ISSN 1386-145X. [115] Lina Wolf, Magnus Knuth, Johannes Osterhoff, and Harald [99] Katharina Siorpaes, Martin Hepp, Andreas Klotz, and Sack. Risq! renowned individuals semantic quiz: a jeopardy Michael Hackl. myontology: tapping the wisdom of crowds like quiz game for ranking facts. pages 71–78. ACM, 2011. for building ontologies. Citeseer, 2008. Proceedings of the 7th International Conference on Semantic [100] Guillermo Soberón, Lora Aroyo, Chris Welty, Oana Inel, Hui Systems. Lin, and Manfred Overmeen. Measuring crowd truth: Dis- [116] Chih-Hao Yu, Tudor Groza, and Jane Hunter. Reasoning on agreement metrics combined with worker behavior ﬁlters. In crowd-sourced semantic annotations to facilitate cataloguing Proc. of CrowdSem 2013 Workshop, ISWC, 2013. of 3d artefacts in the cultural heritage domain. In The Se- [101] Stefan Thaler, Elena Simperl, and Katharina Siorpaes. Spot- mantic WebâA¸SISWC2013˘ , pages 228–243. Springer, 2013. thelink: playful alignment of ontologies. In Proceedings ISBN 3642413374. of the 2011 ACM Symposium on Applied Computing, pages [117] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. A sur- 1711–1712. ACM, 2011. ISBN 1450301134. vey of crowdsourcing systems. In Privacy, Security, Risk and [102] Stefan Thaler, Elena Paslaru Bontas Simperl, and Katharina Trust (PASSAT) and 2011 IEEE Third Inernational Confer- Siorpaes. Spotthelink: A game for ontology alignment. Wis- ence on Social Computing (SocialCom), 2011 IEEE Third In- sensmanagement, 182:246–253, 2011. ternational Conference on, pages 766–773. IEEE, 2011. [103] Stefan Thaler, Katharina Siorpaes, David Mear, Elena Sim- [118] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. perl, and Carl Goodman. Seaﬁsh: a game for collaborative Task matching in crowdsourcing. In Internet of Things and visual image annotation and interlinking. In The Semanic (iThings/CPSCom), 2011 International Conference on and Web: Research and Applications, pages 466–470. Springer, 4th International Conference on Cyber, Physical and Social 2011. Computing, pages 409–412. IEEE, 2011. [104] Stefan Thaler, Elena Simperl, and Stephan Wolger. An exper- [119] Man-Ching Yuen, Irwin King, and Kwong-Sak Leung. iment in comparing human-computation techniques. Internet Taskrec: A task recommendation framework in crowdsourc- Computing, IEEE, 16(5):52–58, 2012. ISSN 1089-7801. ing systems. Neural Processing Letters, pages 1–16, 2014. [105] Andreas Thalhammer, Magnus Knuth, and Harald Sack. [120] Amrapali Zaveri, Anisa Rula, Andrea Maurino, Ricardo Evaluating entity summarization using a game-based ground Pietrobon, Jens Lehmann, SÃ˝urenAuer, and Pascal Hitzler. truth. In The Semantic WebâA¸SISWC2012˘ , pages 350–361. Quality assessment methodologies for linked open data. Sub- Springer, 2012. ISBN 3642351727. mitted to Semantic Web Journal, 2013. [106] David Vickrey, Aaron Bronzan, William Choi, Aman Kumar, [121] Jakub Åaimko.˘ Harnessing manpower for creating semantics. Jason Turner-Maier, Arthur Wang, and Daphne Koller. Online Information Sciences & Technologies: Bulletin of the ACM 32 Basharat et al. / Human Computation and Crowdsourcing meet the Semantic Web: A Survey

Slovakia, 5(3), 2013. Citeseer. [124] Jakub Åaimko,˘ Michal TvaroÅ¿ek, and MÃ ˛ariaBielikovÃ ˛a. [122] Jakub Åaimko˘ and MÃ ˛ariaBielikovÃ ˛a.Little search game: Semantics discovery via human computation games. Interna- Lightweight domain modeling. In Semantic Acquisition tional Journal on Semantic Web and Information Systems, 7: Games, pages 51–65. Springer, 2014. 23–45, 2011. [123] Jakub Åaimko˘ and MÃ ˛ariaBielikovÃ ˛a.Semantic acquisition games. pages 35–50, 2014. ISSN 978-3-319-06114-6.