Researcher or Crowd Member? Why not both! The Open Research Knowledge Graph for Applying and Communicating CrowdRE Research

Oliver Karras∗, Eduard C. Groen†‡, Javed Ali Khan§, Sören Auer∗ ∗Leibniz Information Centre for Science and Technology, Germany, {oliver.karras, soeren.auer}@tib.eu †Fraunhofer IESE, Germany, [email protected] ‡Department of Information and Computing Sciences, Utrecht University, Netherlands §Department of Software Engineering, University of Science and Technology Bannu, Pakistan, [email protected]

Abstract—In recent decades, there has been a major shift while this representation is easy for humans to process, it is towards improved digital access to scholarly works. However, poorly interlinked and not machine-actionable [3]. The next step even now that these works are available in digital form, they in the digital transformation of scholarly knowledge requires remain document-based, making it difficult to communicate the knowledge they contain. The next logical step is to extend a more flexible, fine-grained, semantic, and context-sensitive these works with more flexible, fine-grained, semantic, and representation that allows humans to quickly compare research context-sensitive representations of scholarly knowledge. The results, and machines to process them. This representation can Open Research Knowledge Graph (ORKG) is a platform that hardly be created automatically, but requires domain experts [1] structures and interlinks scholarly knowledge, relying on crowd- and an infrastructure for acquiring, curating, publishing, and sourced contributions from researchers (as a crowd) to acquire, curate, publish, and process this knowledge. In this experience processing scholarly knowledge [3]. report, we consider the ORKG in the context of Crowd-based Numerous projects [6]–[13] provide corresponding solutions Requirements Engineering (CrowdRE) from two perspectives: using knowledge graphs (see Section II) as structured, inter- (1) As CrowdRE researchers, we investigate how the ORKG linked, and semantically rich representations of knowledge. practically applies CrowdRE techniques to involve scholars in While established knowledge graphs exist for representing its development to make it align better with their academic work. We determined that the ORKG readily provides social and encyclopedic and factual knowledge, e.g., in DBpedia [14] and financial incentives, feedback elicitation channels, and support WikiData [15], the use of knowledge graphs for scholarly for context and usage monitoring, but that there is improvement knowledge is a rather new approach [2]. One of these potential regarding automated user feedback analyses and a projects is developing the Open Research Knowledge Graph1 holistic CrowdRE approach. (2) As crowd members, we explore (ORKG) [13]. The ORKG is a platform that uses crowdsourcing how the ORKG can be used to communicate scholarly knowl- edge about CrowdRE research. For this purpose, we curated to acquire and curate scholarly knowledge. The project explores qualitative and quantitative scholarly knowledge in the ORKG how scholarly knowledge can be acquired along the research based on papers contained in two previously published systematic lifecycle, relying on the manual acquisition and curation of literature reviews (SLRs) on CrowdRE. This knowledge can scholarly knowledge through crowdsourced contributions from be explored and compared interactively, and with more data experts [1], [3], [13]. Researchers from various research fields than what the SLRs originally contained. Therefore, the ORKG improves access and communication of the scholarly knowledge form a crowd that acquires, curates, publishes, and processes about CrowdRE research. For both perspectives, we found the scholarly knowledge. Based on the publicly available beta ORKG to be a useful multi-tool for CrowdRE research. version of the ORKG1, the ORKG project team has two long- arXiv:2108.05085v1 [cs.DL] 11 Aug 2021 Index Terms—Crowd, crowd-based requirements engineering, term goals. First, they aim to integrate more strategies for crowdsourcing, knowledge graph, open research crowdsourcing to enable crowd members to contribute their research results to the ORKG in a more flexible and lightweight I.INTRODUCTION manner [13, p. 9]. Second, the team aims to tailor the platform Historically, research results were published in collections of to the needs and requirements of the expert crowd by involving printed works, such as journals [1]. Studying the literature on the crowd members in the development and soliciting feedback a particular topic required hours or days of browsing through a from them on problems and features [13, p. 5]. Although library’s collection. In recent decades, the research community these goals align with CrowdRE, the ORKG project team has has made great efforts to improve access to scholarly knowledge not yet consciously applied CrowdRE. This fact aroused our as part of the digital transformation [2], [3]. Digitizing interest to research the implementation of CrowdRE in the real articles was the first step in this transformation. However, development setting of the ORKG platform. In this experience even digitized articles are just digital representatives of their report, we address two research questions: printed counterparts [4]. For this reason, digital articles impede scholarly communication by still being document-based [5]; 1http://orkg.org/ RQ1: What potential does the ORKG have as a platform The organization of scholarly communication and knowledge for applying CrowdRE research in a real development based on the structured, standardized, and semantic represen- setting? tation form of a knowledge graph offers several benefits [20]. It increases the unique identification of the relevant artifacts, RQ2: What potential does the ORKG have as a platform concepts, attributes, and relationships of research results. All for communicating scholarly knowledge about CrowdRE these elements can be linked with each other, considering the research? constraints imposed by the underlying ontology. It increases Regarding RQ1, we take the perspective of CrowdRE traceability through improved and explicit linking of the researchers. We describe the current state and features of artifacts and information sources. In this way, graph-based the ORKG as a crowdsourcing platform that involves its crowd knowledge sharing also helps to reduce redundancy and members in the development along the four key activities duplication because repetitive content, such as related work on a of CrowdRE (motivating crowd members, eliciting feedback, topic, can be continuously described, stored, and communicated analyzing feedback, monitoring context & usage data) [16]. over time. This type of scholarly communication curbs the This overview shows to what extent and how the ORKG already proliferation of scientific publications and leads to an increase addresses CrowdRE, and highlights improvement potential. in efficiency by avoiding media discontinuities in the various CrowdRE researchers benefit from this overview as it enables phases of scientific work. It reduces ambiguity through more them to assess the suitability of the ORKG as a basis for their fu- terminological and conceptual clarity because the concepts and ture work and studies on CrowdRE research, while the efficacy relationships can be reused across disciplinary boundaries. It of CrowdRE in a real development setting can be demonstrated increases machine actionability on the content of scientific to practitioners. Regarding RQ2, we take the perspective of publications, enabling machines to understand the structure crowd members. We provide insights into our experiences with and semantics of the content of scientific publications. Finally, acquiring, curating, and publishing qualitative and quantitative because the content is machine-actionable, it increases develop- scholarly knowledge about CrowdRE research in the ORKG. ment opportunities for applications that provide search, retrieval, Using two examples from CrowdRE research [17], [18], we mining, and assistance features for scholarly knowledge. These illustrate how (CrowdRE) researchers can use the ORKG to applications could support open science by making knowledge improve access and communication of scholarly knowledge more accessible, e.g., to early career scientists, lay people, or about their research. Based on the two perspectives, we assess researchers with a visual impairment. the potential of the ORKG as a platform for applying and communicating CrowdRE research. We gained the following III.RELATED WORK insights: In recent years, the role of a crowd has attracted much attention across many disciplines through means such as The ORKG has a two-fold potential for CrowdRE: crowdsourcing. Numerous projects have emerged offering (1) The platform and its features provide a solid basis for platforms that involve a crowd as an inherent part of their applying CrowdRE research in a real development setting systems [21]–[23]. The digital transformation has compelled and in close collaboration with the ORKG project team. the domain of requirements engineering (RE) to respond in (2) The platform enables the communication of CrowdRE various ways, including placing greater emphasis on automated research by allowing more comprehensive curation of RE [24]. A particular kind of data-driven RE [25], [26] relies on scholarly knowledge than document-based works do. the involvement of a crowd as a source of data, and is typically This two-fold potential makes the ORKG a useful multi- referred to as CrowdRE [16]. CrowdRE describes an iterative tool for applying and communicating CrowdRE research. cycle of eliciting feedback from the crowd and monitoring context and usage data to derive the needs and requirements II.BACKGROUND of the crowd, which, once validated, are implemented into the In this section, we briefly introduce knowledge graphs and product [16]. Current CrowdRE research focuses on the use their contribution to the digital transformation of scholarly com- of various existing channels with a general purpose, aimed munication towards graph-based knowledge sharing. According at accessing the crowd, such as social networks and mobile to Brack et al. [19, p. 1], “A knowledge graph (KG) consists application marketplaces [18]. These independent channels are of (1) an ontology describing a conceptual model (e.g., with separated from the actual product but allow crowd members to classes, relation types, and axioms), and (2) the corresponding provide feedback on the product. However, they are not enough instance data (e.g., objects, literals, and hsubject, predicate, for successful CrowdRE [18], [27]; a combined feedback and objecti-triplets) following the constraints posed by the ontology monitoring solution needs to be integrated into the actual (e.g., instance-of relations, axioms, etc.). The construction of a product to optimally support CrowdRE. This integration can KG involves ontology design and population with instances.” simplify crowd involvement by making it easier for crowd In the context of scholarly communication, a knowledge members to provide feedback and thus actively participate in graph represents original research results semantically, i.e., the development. It also helps developers better understand explicitly and formally, and comprehensively links existing the feedback given because of the details obtained through the data, metadata, knowledge, and information resources [20]. monitored context and usage data [18], [27]. Approaches such as FAME [28] propose a possible integration of multiple data Artifact-Based Analysis sources. The crowd can also be involved through crowdsourced Project Page Publications Wiki GitLab tasks, e.g., requirements classification [29]. In the long term, however, we need systematic and holistic approaches covering Groen Review-Meeting the entire software development process to create information ORKG Features systems that involve a crowd operationally, e.g., to provide Karras content or form a social network, and to support development Paper Contributions ORKG Platform Comparisons with user feedback. As this topic has only emerged in recent Khan years and is still being researched, such approaches are still Project Team lacking. To address this deficit, platforms and projects must Usage-based Analysis work closely with researchers to implement CrowdRE in real Fig. 1: Overview of the procedure for analyzing the ORKG. development settings [18], [27], [30]. B. Identified Features of ORKG as a Crowdsourcing Platform IV. THE ORKG ASA PLATFORM FOR CROWDRE Described along the four pillars of crowdsourcing [23], the In his CrowdRE’19 keynote, Glinz [31] emphasized the ORKG is a crowdsourcing platform with a crowd, mainly need for CrowdRE to venture out into open source and open consisting of researchers, that is diverse in terms of spatial research settings because its characteristics make it highly distribution, gender, age, and expertise. As of August 2021, suitable for their respective crowds. Based on this notion, the crowd consists of 530 members, 307 of whom actively the ORKG aroused our interest for several reasons. (1) The contribute scholarly knowledge to the platform. The ORKG ORKG is an open source platform for open research. (2) The project team as the sole crowdsourcer focuses on expert- goals of the ORKG project team are aligned with CrowdRE. based crowdsourcing. Nevertheless, the platform is open (3) The project team seeks to collaborate with others on use to anyone willing to give it a try. The crowdsourced task cases and new features for the ORKG2. (4) The project offers is the acquisition and curation of scholarly knowledge by two perspectives on the ORKG: As researchers for applying publishing and processing state-of-the-art comparisons and CrowdRE research in a real development setting, and as crowd corresponding articles—so-called smart reviews—in arbitrary members for communicating this research. We first describe research fields. To better reflect the platform’s use of CrowdRE our analysis procedure in Section IV-A. We then present our from a researcher’s perspective, we organized the identified assessment of the features of the ORKG as a crowdsourcing features of the platform along the four key activities of platform that involves its crowd members in the development CrowdRE: motivating crowd members, eliciting feedback, in Section IV-B, and we report our experiences in using the analyzing feedback, and monitoring context & usage data [16]. ORKG in Section IV-C. 1) Motivating Crowd Members: Crowd members must be A. Procedure motivated to become and remain active participants. Their We performed an artifact- and usage-based analysis of motivation can be intrinsic and/or extrinsic. Intrinsic motivation the ORKG (see Fig. 1). On the one hand, we analyzed the is mainly rooted in knowledge sharing and love of the commu- documentation from the project page3, the wiki4, GitLab5, and nity. This motivates crowd members to share their research by related publications [1]–[3], [19], [32]–[34]. On the other hand, acquiring and curating scholarly knowledge on the platform we used the ORKG as crowd members to acquire, curate, and with others, while valuing the contributions of the other crowd publish scholarly knowledge about CrowdRE research [35], members. In this way, crowd members also achieve mental [36]. From the analysis, we identified features that typify satisfaction, boosted self-esteem, and personal skill development the ORKG as a crowdsourcing platform. We structured these since they support the research community with open data, features according to Hosseini et al.’s [23] reference model for open research, and open knowledge by creating state-of-the-art crowdsourcing—the so-called four pillars of crowdsourcing. comparisons and writing smart reviews. Extrinsic motivation This reference model provides a taxonomy of crowdsourcing is mainly achieved through social incentives in the ORKG. to describe the individual features of the four pillars—the The project team essentially uses public acknowledgments of crowd, the crowdsourcer, the crowdsourced task, and the crowd members to motivate them, through visible rankings crowdsourcing platform—in a hierarchical manner. When we and citable contributions, to contribute scholarly knowledge presented our results to the ORKG project team during a review to the ORKG. Among other things, a unique Digital Object meeting for verification, they provided suggestions to extend Identifier (DOI) can be assigned to a comparison or a smart some descriptions. TABLE I shows an excerpt6 of the features review. In May 2021, the project team also introduced financial of the ORKG as a crowdsourcing platform. incentives through the ORKG Curation Grant. Crowd members can apply for this grant based upon demonstrable contributions 2 https://projects.tib.eu/orkg/get-involved/ to the ORKG. If accepted, they receive C400 per month and 3https://projects.tib.eu/orkg/ 4https://gitlab.com/TIBHannover/orkg/orkg-frontend/-/wikis/home commit to making regular contributions to the ORKG for a 5https://gitlab.com/groups/TIBHannover/orkg/-/issues period of six months. At the moment, the ORKG does not 6The full overview is available online in a supplement to this paper [37]. provide entertainment incentives, such as gamification. TABLE I: Excerpt of identified features of the ORKG as a crowdsourcing platform (cf. Karras et al. [37]) Feature Description Pillar 1: The crowd 5.4 Motivation 5.4.1 Mental satisfaction The crowd members support open data, open research, and open knowledge for all. 5.4.2 Self-esteem The crowd members know that they support the research community. 5.4.3 Personal skill development The crowd members can develop their research skills by creating state-of-the-art comparisons and smart reviews. 5.4.4 Knowledge sharing The crowd members share their research by acquiring and curating their scholarly knowledge with others. 5.4.5 Love of community The crowd members value each other’s results since the platform addresses an open research community. Pillar 2: The crowdsourcer 1. Incentives provision 1.1 Financial incentives The project team launched the ORKG Curation Grant Competition in May 2021, paying C400 per month for regular contributions to the ORKG (initially limited to six months). 1.2 Social incentives The crowdsourcer uses public acknowledgments of contributors and curators of scholarly knowledge on the platform with prominently visible rankings and mentions. 1.3 Entertainment incentives This feature is not currently supported. Pillar 3: The crowdsourced task 7.1 Problem solving The task of acquiring, curating, publishing, and processing scholarly knowledge can consider a specific research problem that can be answered with an analysis of a state-of-the-art comparison in the respective research field. 7.2 Innovation The task of acquiring, curating, publishing, and processing scholarly knowledge can lead to new ideas. 7.3 Co-creation The task of acquiring, curating, publishing, and processing scholarly knowledge requires collaboration with crowd members to communicate and maintain scholarly knowledge in the long term. Pillar 4: The crowdsourcing platform 1. Crowd-related interactions 1.9 Provide feedback loops The platform uses several options to provide feedback to the crowd as a whole: mailing list2, Twitter account7, project page3, and the ORKG website1. They are used to communicate regularly about the current status and changes to the platform, including technical improvements and content development achieved through crowdsourcing. However, there are currently no mechanisms to provide feedback to individual crowd members. 2. Crowdsourcer-related interactions 2.8 Provide feedback loops The platform gives the crowd several options to provide feedback to the project team as the crowdsourcer: 1) Different communication mechanisms for contacting the project team: Chatwoot8, email contacts2, Skype group2, Twitter account7, and a GitLab issue tracker5. 2) Surveys integrated into the platform after specific processes, e.g., adding a publication to the platform. 3. Task-related facilities 3.3 Store history of completed tasks The platform stores all tasks and changes for each individual crowd member. In addition, the platform supports versioning for created comparisons and smart reviews. 4. Platform-related facilities 4.3 Provide ease of use The platform has its own front-end development team for continuously improving the interface for the crowd. 4.4 Provide attraction The platform has its own front-end development team for continuously improving the interface for the crowd.

Finding 1: The ORKG strongly relies on the intrinsic a GitLab issue tracker, both of which are common data sources motivation of researchers to share and communicate for eliciting and analyzing user feedback. (4) The project team research contributions. The platform does support extrinsic also started to integrate feedback loops into the ORKG through motivation by providing social incentives through public surveys conducted after certain activities in order to understand acknowledgments of contributions, which are also citable how well crowd members got along with the system. through DOIs. Recently, the project team added a financial Besides feedback loops from the crowd to the project incentive. The ORKG currently does not employ entertain- team, the ORKG also offers mechanisms in the opposite ment incentives such as enjoyment, fun, and gamification. direction. These mechanisms include a mailing list, the Twitter 2) Eliciting Feedback: The foundation of CrowdRE is account, the project page, and the ORKG platform itself. These the ability to elicit feedback from the crowd and derive mechanisms are used to communicate regularly about the requirements in return. The ORKG already provides several current status of and changes to the ORKG, including its feedback loops through which the crowd can provide feedback technical and content development that was achieved with the to the project team, including means of contacting the project help of the crowd. However, the ORKG lacks mechanisms for team. (1) The ORKG uses Chatwoot; a support communication providing targeted feedback to individual crowd members. system that can be integrated into platforms such as websites to enable direct communication between crowd members and Finding 2: The ORKG project team uses various mecha- administrators. (2) The project team provides email contacts nisms to communicate bilaterally with the crowd. Chan- and a Skype group to offer help and support. Both channels nels external to the ORKG include Twitter and a GitLab are less typical for CrowdRE, but can certainly be used as data issue tracker, while integrated channels include Chatwoot, sources for analysis. (3) The ORKG has a Twitter account and surveys, Skype, and email. In this way, the ORKG uses both familiar and less typical feedback channels for 7https://twitter.com/orkg_org CrowdRE, and is in the process of integrating feedback 8https://www.chatwoot.com/ mechanisms into the platform itself. 3) Analyzing Feedback: A central concern in CrowdRE us to ensure that the contributions from the SLRs’ papers were is the derivation of requirements from the elicited feedback curated according to the interpretation of its authors. (2) Both through analysis. The ORKG project team currently relies on SLRs represent milestones regarding the current status of two direct communication with the crowd, and thus on immediate important topics in CrowdRE research. (3) Due to their different analysis and processing of the feedback, which is reflected nature, the SLRs allow us to determine the potential of the in the extensive use of direct communication channels such ORKG for qualitative and quantitative data, respectively. as Chatwoot and a Skype group. Accordingly, the analysis of feedback is done manually, without any (semi-)automated 1) Case I – Crowd Intelligence in Requirements Engineering: analyses. This was a conscious design decision due to the The SLR conducted by Khan et al. [18] marks one of the most smaller size of the project team and the expected limited—thus, comprehensive overviews of the literature on CrowdRE to manageable—crowd size in the early stages of the project. date, encompassing 77 papers. A noteworthy contribution of In the long term, however, more (semi-)automated CrowdRE this SLR is that it organizes these papers according to five feedback analysis measures are to be put in place if the platform phases of RE, along with the CrowdRE utilities applied in each is to cater for the anticipated thousands of researchers from paper, such as the crowd, the crowdsourced task, incentives various research fields, where direct contact between crowd for motivation, and channels for feedback elicitation. This members and the project team will be limited by necessity. contribution is based on a qualitative expert analysis of the papers. Although the classification provides a strong overview Finding 3: The ORKG project team analyzes the elicited of the works published until 2019, it is nearly impossible to feedback manually and immediately. It deliberately has no dynamically keep this overview up-to-date in the long term as (semi-)automated feedback analysis approaches in place. a document-based publication. Once the crowd’s size is no longer manageable, keeping 2) Case II – User Feedback Classification Approaches: The the crowd involved in the development inevitably demands SLR conducted by Santos et al. [17] provides a quantitative other feedback analysis paradigms. comparison of 43 papers on how well Machine Learning (ML) 4) Monitoring Context & Usage Data: The ORKG project algorithms perform in classifying elicited user feedback. Out team provides several options to assist the crowd members of a total of 78 classification categories in the field of feedback in using the ORKG. These options include tutorial videos, a analysis for RE [38], their SLR made a quantitative comparison guided tour of and tooltips on the user interface, templates, for only the single most frequently found classification category, comprehensive documentation, and different communication “Feature Request”, because the large tables needed to present mechanisms for support. The project’s dedicated front-end all results conflicted with space constraints. Realistically, a development team continuously improves the user interface publication that presents such a comparison even for a subset for the crowd based on the provided feedback, which can of the 78 classification categories could potentially become long be enriched with monitored context and usage data to better and repetitive, or require complex groupings of classification understand the feedback of the crowd members. For usage categories. The presentation would then likely overshoot its monitoring, the ORKG stores the history of all changes and goal of conveying comparable scholarly knowledge. tasks completed by the crowd members, and the web analytics 3) State-of-the-Art Comparisons with the ORKG: Cases I tool Matomo9 is used to evaluate the crowd members’ journeys and II provide valid motivations for creating state-of-the-art on the platform. In addition, administrators and curators can comparisons with the ORKG, given the need for continuous supervise the activities of the crowd in the background. curations of scholarly knowledge as well as flexible, fine- Finding 4: The ORKG has basic approaches for monitor- grained, semantic, and context-sensitive representations. ing context and usage data in place. It was inherently de- The ORKG organizes the acquired and curated scholarly signed to store any history, which fundamentally facilitates knowledge by paper as a collection of so-called contributions, tracking usage behavior in compliance with data privacy which address a research problem and consist of scholarly regulations. This historical data can be supplemented with knowledge. This knowledge is stored in a knowledge graph, web analytics data collected through Matomo. from which the crowd members distill the contributions they are looking for. The selected contributions are compared in C. Experiences with Using ORKG as Crowd Members state-of-the-art comparisons. Fig. 2 shows an excerpt from Besides reflecting on the ORKG as CrowdRE researchers, our comparison for Case II [35]. The columns denote the we also experienced the platform and its features as crowd contributions by paper, and the rows denote the scholarly members. We created two state-of-the-art comparisons in knowledge. The comparison of Case II currently describes the ORKG [35], [36] based on papers contained in two the ML classifiers, ML features, and the quantitative classifier previously published systematic literature reviews (SLRs) on performance values (Precision, Recall, the F1 and Fβ measures, CrowdRE [17], [18]. We selected these for three reasons. and Berry’s [39] task-based βT value) from the 19 papers (1) One author of each SLR co-authored this paper, enabling described in the SLR by Santos et al. [17] that used the classification category “Feature Request”. The comparison 9https://matomo.org/ of Case I so far consists of contributions from 27 of the Fig. 2: Excerpt from our comparison for Case II [35]. 77 papers from the SLR by Khan et al. [18], describing the example, a development succeeding the SLR by Santos et relation of the papers to five phases of RE and the CrowdRE al. [17] are reports of Deep Learning algorithms showing utilities applied [36]. We are still in the process of adding the promising results in classifying user feedback [41], [42], which contributions from the remaining 50 papers, which is more should be successively added to the comparison. time-consuming than for the quantitative data from Case I Despite all these advantages, the ORKG also has limitations. because of the expert judgments needed for classifying the Most of the limitations we experienced can be attributed to papers’ contributions. The comparison of the 27 papers makes the development status of the platform, which is currently it easy to identify, for example, the four papers that address in beta. Further development of the ORKG must improve the runtime purpose of monitoring for requirements evolution. interactions for the expert crowd by enabling better workflows With the created comparisons [35], [36], we achieved our for entering data and creating visualizations. Nevertheless, we goal of acquiring and curating the detailed results of both also experienced that the project team has always responded SLRs. The knowledge-based representation in the form of directly to our reported issues, which we could see getting comparisons has several advantages over a purely document- added to the GitLab issue tracker11 and addressed shortly based representation. The comparisons are interactive and allow thereafter. filtering of views by different scholarly knowledge contained V. DISCUSSION in each row, even by specific value ranges of qualitative and quantitative content. The ORKG also provides a service for The ORKG aroused our interest as a crowdsourcing platform generating several graphical visualizations based on data in the for applying and communicating CrowdRE research. In this comparisons, helping the reader understand information faster experience report, we explored whether the ORKG can promote than through the large comparison table. The most important the potential of CrowdRE in open source and open research feature of the ORKG is that the added contributions and settings, taking two perspectives: that of CrowdRE researchers created comparisons are available to anyone. In this way, every and that of crowd members. crowd member can use the curated scholarly knowledge and Our first contribution is that we provide a comprehensive created comparisons as a basis for new comparisons. Moreover, overview of the ORKG’s features as a crowdsourcing platform the existing comparisons can be expanded with additional for acquiring and curating scholarly knowledge [37], mapped scholarly knowledge from papers already included, and even to the four key activities of CrowdRE. Our findings show with new contributions from papers added later to the ORKG. that the ORKG is a crowdsourcing platform offering several We already expanded several contributions, e.g., the results of features that can facilitate successful CrowdRE. Although the other classifications reported in Dhinakaran et al.’s paper [40]10. ORKG project team has not yet consciously applied CrowdRE, For Case I, we added the details of the three crowd properties they already address crucial parts of the CrowdRE cycle by scale; level of knowledge, skills & expertise; and roles, which motivating crowd members to participate, eliciting feedback, are only briefly and superficially described in the SLR [18]. For and monitoring context & usage data, which they analyze to Case II, we added links to the datasets used and performance derive and implement the needs and requirements of the crowd. values to classification categories other than “Feature Request”. To motivate crowd members, the project team uses es- This expansion is relevant to enable long-term curation. For tablished mechanisms and incentives to boost intrinsic and extrinsic motivation (see Finding 1). Feedback is elicited

10https://www.orkg.org/orkg/paper/R76818/R76825 11https://gitlab.com/TIBHannover/orkg/orkg-frontend/-/issues/634 through channels integrated into the ORKG, i.a., Chatwoot, and potential of the ORKG, and we propose its use as a platform standalone channels, i.a., a GitLab issue tracker (see Finding 2). for applying and communicating CrowdRE research to further However, analysis of this feedback is currently a weakness advance this research while keeping its exponentially growing of the ORKG since the project team still does this manually body of knowledge manageable [43]. (see Finding 3). While there are also basic mechanisms for monitoring context and usage data in place (see Finding 4), Answer to RQ2: The ORKG enables a new way the feedback and the monitored data are currently not analyzed of communicating (CrowdRE) research through more together. The overview and the findings show to what extent comprehensive acquisition, curation, publication, and and how the ORKG already addresses CrowdRE and highlights processing of scholarly knowledge than document-based gaps, which helps CrowdRE researchers make an informed works. For CrowdRE researchers, the ORKG does not decision on how suitable the ORKG is as a basis for future work only provide improved access to and communication of and studies. Leveraging the inherent features of the ORKG scholarly knowledge about their research, but also the for successful CrowdRE provides a solid basis for applying opportunity to experience the ORKG as crowd members, CrowdRE research in a real development setting, which in offering a new perspective on their research. turn can help the project team further improve and enrich the VI.CONCLUSION platform and involve its crowd members in the development. The ORKG is a crowdsourcing platform that already ad- Answer to RQ1: Although the ORKG project team has dresses crucial aspects of the CrowdRE cycle by offering not yet consciously applied CrowdRE, the ORKG is a several features that facilitate successful CrowdRE. However, crowdsourcing platform that already has several impor- these features must be further expanded to develop a systematic tant features for successful CrowdRE in place. Despite and holistic approach to achieve the goals of the ORKG project improvement potential—e.g., adding (semi-)automated team, i.e., to involve researchers from various research fields feedback analyses—the ORKG provides a solid basis as crowd members who both use the ORKG and participate for researchers to apply and study CrowdRE in a real in its development. This calls for collaborations between the development setting and in close collaboration with the ORKG project team and CrowdRE researchers to foster mutual ORKG project team. benefits. On the one hand, the ORKG project team will benefit from new crowd members and partners who can help to develop In addition to reflecting on the ORKG’s features, we also a corresponding systematic and holistic CrowdRE approach provided insights into our experiences using the ORKG as to continuously adapt the ORKG to the evolving needs and crowd members to communicate scholarly knowledge about requirements of the crowd in the long term. On the other hand, CrowdRE research. Despite its limitations, especially regarding CrowdRE researchers will benefit in two ways. First, they usability, the use of the ORKG and the exchanges with the get a stable platform to apply their research that already has project team were positive experiences. We observed that our several features for successful CrowdRE, whose continuous feedback was received and addressed directly, highlighting development is guaranteed, and whose project team is interested the team’s efforts to involve and acknowledge the crowd in in collaborating to implement CrowdRE in a real development the development. Overall, we achieved our goal of acquiring setting. In this way, the ORKG can be an interesting research and curating the detailed results of two SLRs with a very object in the future in terms of how CrowdRE techniques are different nature—Case I focuses on qualitative and Case II incrementally added to the platform. Thus, the ORKG provides on quantitative scholarly knowledge. The ORKG has shown a basis for case studies in open source and open research that it supports the acquisition and curation of both kinds settings (cf. Glinz [31]). Second, we laid the foundation of knowledge. Even though the input and representation of for communicating CrowdRE research with the ORKG by this information may initially appear to be human-readable acquiring, curating, and publishing scholarly knowledge about only, the way the data is entered and stored in the underly- CrowdRE. CrowdRE researchers can build on this foundation ing data structure makes the scholarly knowledge machine- and expand the ORKG by adding more CrowdRE papers with actionable. As a result, scholarly knowledge, e.g., concepts their contributions, as well as corresponding comparisons and and relationships, can be identified more easily due to greater smart reviews. In this way, they will also gain a fresh new terminological and conceptual precision and sharpness [20]. perspective on their research as members of the ORKG’s target In this way, the ORKG does not only provide researchers crowd. For this reason, the ORKG is a crowdsourcing platform with another opportunity to publish and process literature, but that can act as a useful multi-tool for CrowdRE research. also to develop novel services that make scholarly knowledge accessible with new search, retrieval, mining, and assistance ACKNOWLEDGMENT applications [20]. For example, the ORKG recently served as a data source for a dashboard that searches and visualizes This work was co-funded by the European Research Council academic literature on students’ attitudes towards ICT in the for the project ScienceGRAPH (Grant agreement ID: 819536) PISA program12. Our experiences have convinced us of the and by the TIB – Leibniz Information Centre for Science and Technology. We thank Sonnhild Namingha for proofreading 12https://www.orkg.org/orkg/usecases/pisa-dashboard/ this paper. REFERENCES [23] M. Hosseini, K. Phalp, J. Taylor, and R. Ali, “The Four Pillars of Crowdsourcing: A Reference Model,” in 2014 IEEE 8th International [1] S. Auer, A. Oelen, M. Haris, M. Stocker, J. D’Souza, K. E. Farfar, Conference on Research Challenges in Information Science (RCIS). L. Vogt, M. Prinz, V. Wiens, and M. Y. Jaradeh, “Improving Access to IEEE, 2014, pp. 1–12. Scientific Literature with Knowledge Graphs,” Bibliothek Forschung und [24] K. Villela, A. Hess, M. Koch, R. Falçao, E. C. Groen, J. Dörr, C. N. Valero, Praxis, vol. 44, no. 3, pp. 516–529, 2020. and A. Ebert, “Towards Ubiquitous RE: A Perspective on Requirements [2] S. Auer, V. Kovtun, M. Prinz, A. Kasprzik, M. Stocker, and M. E. Vidal, Engineering in the Era of Digital Transformation,” in 2018 IEEE 26th “Towards a Knowledge Graph for Science,” in Proceedings of the 8th International Requirements Engineering Conference (RE), 2018. International Conference on Web Intelligence, Mining and Semantics, [25] W. Maalej, M. Nayebi, T. Johann, and G. Ruhe, “Toward Data-Driven 2018, pp. 1–6. Requirements Engineering,” IEEE Software, vol. 33, no. 1, 2015. [3] M. Y. Jaradeh, A. Oelen, K. E. Farfar, M. Prinz, J. D’Souza, G. Kismihók, [26] W. Maalej, M. Nayebi, and G. Ruhe, “Data-Driven Requirements M. Stocker, and S. Auer, “Open Research Knowledge Graph: Next Gener- Engineering – An Update,” in 2019 IEEE/ACM 41st International ation Infrastructure for Semantic Scholarly Knowledge,” in Proceedings Conference on Software Engineering: Software Engineering in Practice of the 10th International Conference on Knowledge Capture, 2019. (ICSE-SEIP). IEEE, 2019, pp. 289–290. [4] H. van de Sompel and C. Lagoze, “All Aboard: Toward a Machine- [27] M. Stade, M. Oriol, O. Cabrera, F. Fotrousi, R. Schaniel, N. Seyff, and Friendly Scholarly Communication System,” in The Fourth Paradigm, O. Schmidt, “Providing a User Forum is not Enough: First Experiences of 2009. a Software Company with CrowdRE,” in 2017 IEEE 25th International [5] J. Bosman, I. Bruno, C. Chapman, B. G. Tzovaras, N. Jacobs, B. Kramer, Requirements Engineering Conference Workshops (REW). IEEE, 2017. M. E. Martone, F. Murphy, D. P. O’Donnell, M. Bar-Sinai et al., [28] M. Oriol, M. Stade, F. Fotrousi, S. Nadal, J. Varga, N. Seyff, A. Abello, “The Scholarly Commons – Principles and Practices to Guide Research X. Franch, J. Marco, and O. Schmidt, “FAME: Supporting Continuous Communication,” 2017. Requirements Elicitation by Combining User Feedback and Monitoring,” [6] A. Aryani and J. Wang, “Research Graph: Building a Distributed Graph in 2018 IEEE 26th International Requirements Engineering Conference of Scholarly Works Using Research Data Switchboard,” 2017. (RE). IEEE, 2018, pp. 217–227. [7] A. Burton, H. Koers, P. Manghi, M. Stocker, M. Fenner, A. Aryani, [29] M. van Vliet, E. C. Groen, F. Dalpiaz, and S. Brinkkemper, “Identifying S. La Bruzzo, M. Diepenbroek, and U. Schindler, “The Scholix and Classifying User Requirements in Online Feedback via Crowdsourc- Framework for Interoperability in Data-Literature Information Exchange,” ing,” in Proceedings of Requirements Engineering – Foundation for D-Lib Magazine, vol. 23, no. 1/2, 2017. Software Quality (REFSQ). LNCS 12045, 2020, pp. 143–159. [8] S. Bechhofer, D. De Roure, M. Gamble, C. Goble, and I. Buchan, [30] X. Franch, N. Seyff, M. Oriol, S. Fricker, I. Groher, M. Vierhauser, “Research Objects: Towards Exchange and Reuse of Digital Knowledge,” and M. Wimmer, “Towards Integrating Data-Driven Requirements Nature Precedings, pp. 1–1, 2010. Engineering into the Software Development Process: A Vision Paper,” [9] K. L. Hanson, T. DiLauro, and M. Donoghue, “The RMap Project: in International Working Conference on Requirements Engineering: Capturing and Preserving Associations Amongst Multi-Part Distributed Foundation for Software Quality. Springer, 2020, pp. 135–142. Publications,” in Proceedings of the 15th ACM/IEEE-CS Joint Conference [31] M. Glinz, “CrowdRE: Achievements, Opportunities and Pitfalls,” in Pro- on Digital Libraries, 2015, pp. 281–282. ceedings of the 27th International Requirements Engineering Conference [10] A. Sinha, Z. Shen, Y. Song, H. Ma, D. Eide, B.-J. P. Hsu, and K. Wang, Workshops (REW). IEEE, 2019, pp. 172–173. “An Overview of Microsoft Academic Service (MAS) and Applications,” [32] M. Y. Jaradeh, A. Oelen, M. Prinz, M. Stocker, and S. Auer, “Open in Proceedings of the 24th International Conference on World Wide Web. Research Knowledge Graph: A System Walkthrough,” in International Association for Computing Machinery, 2015, p. 243–246. Conference on Theory and Practice of Digital Libraries. Springer, 2019. [11] T. Hammond, M. Pasin, and E. Theodoridis, “Data Integration and [33] A. Oelen, M. Y. Jaradeh, M. Stocker, and S. Auer, “Generate FAIR Disintegration: Managing Springer Nature SciGraph with SHACL and Literature Surveys with Scholarly Knowledge Graphs,” in Proceedings OWL,” in International Semantic Web Conference, 2017. of the ACM/IEEE Joint Conference on Digital Libraries in 2020, 2020. [12] V. G. Meister, “Towards a Knowledge Graph for a Research Group with [34] V. Wiens, M. Stocker, and S. Auer, “Towards Customizable Chart Focus on Qualitative Analysis of Scholarly Papers,” in Proceedings of the Visualizations of Tabular Data Using Knowledge Graphs,” in International First Workshop on Enabling Open Semantic Science, 2017, pp. 71–76. Conference on Asian Digital Libraries. Springer, 2020, pp. 71–80. [13] M. Y. Jaradeh, S. Auer, M. Prinz, V. Kovtun, G. Kismihók, and M. Stocker, [35] O. Karras and E. C. Groen, “Overview of Approaches that Classify “Open Research knowledge Graph: Towards Machine Actionability in User Feedback as Feature Request,” 2021. [Online]. Available: Scholarly Communication,” arXiv preprint arXiv:1901.10816, 2019. https://www.orkg.org/orkg/comparison/R112387 [14] S. Auer, C. Bizer, G. Kobilarov, J. Lehmann, R. Cyganiak, and Z. Ives, [36] O. Karras and J. A. Khan, “Overview of Crowd Intelligence “DBpedia: A Nucleus for a Web of Open Data,” in The Semantic Web. in Requirements Engineering,” 2021. [Online]. Available: https: Springer, 2007, pp. 722–735. //www.orkg.org/orkg/comparison/R114155 [15] D. Vrandeciˇ c´ and M. Krötzsch, “Wikidata: A Free Collaborative [37] O. Karras, E. C. Groen, J. A. Khan, and S. Auer, “Description of the Knowledge Base,” Communications of the ACM, vol. 57, no. 10, 2014. Features of the Open Research Knowledge Graph as a Crowdsourcing [16] E. C. Groen, N. Seyff, R. Ali, F. Dalpiaz, J. Doerr, E. Guzman, Platform based on the 4 Pillars of Crowdsourcing.” [Online]. Available: M. Hosseini, J. Marco, M. Oriol, A. Perini et al., “The Crowd in https://doi.org/10.5281/zenodo.5172132 Requirements Engineering: The Landscape and Challenges,” IEEE [38] R. Santos, E. C. Groen, and K. Villela, “A Taxonomy for User Feedback software, vol. 34, no. 2, pp. 44–52, 2017. Classifications,” in REFSQ Workshops, 2019. [17] R. Santos, E. C. Groen, and K. Villela, “An Overview of User Feedback [39] D. M. Berry, “Evaluation of Tools for Hairy Requirements Engineering Classification Approaches,” in REFSQ Workshops, 2019. Tasks,” Empirical Software Engineering, in press. [Online]. Available: [18] J. A. Khan, L. Liu, L. Wen, and R. Ali, “Crowd Intelligence in http://dx.doi.org/10.1007/s10664-021-09986-0 Requirements Engineering: Current Status and Future Directions,” in [40] V. T. Dhinakaran, R. Pulle, N. Ajmeri, and P. K. Murukannaiah, “App International Working Conference on Requirements Engineering: Foun- Review Analysis Via Active Learning: Reducing Supervision Effort dation for Software Quality (REFSQ). Springer, 2019, pp. 245–261. without Compromising Classification Accuracy,” in 2018 IEEE 26th [19] A. Brack, A. Hoppe, M. Stocker, S. Auer, and R. Ewerth, “Analysing International Requirements Engineering Conference (RE), 2018. the Requirements for an Open Research Knowledge Graph: Use Cases, [41] R. Reddy Mekala, A. Irfan, E. C. Groen, A. Porter, and M. Lindvall, Quality Requirements, and Construction Strategies,” International Journal “Classifying User Requirements from Online Feedback in Small Dataset on Digital Libraries, pp. 1–23, 2021. Environments Using Deep Learning,” in 2021 IEEE 29th International [20] S. Auer, “Towards an Open Research Knowledge Graph,” 2018. [Online]. Requirements Engineering Conference. IEEE, 2021. Available: https://doi.org/10.5281/zenodo.1157185 [42] C. Stanik, T. Pietz, and W. Maalej, “Unsupervised Topic Discovery [21] U. Huws, N. Spencer, and S. Joyce, “Crowd Work in Europe: Preliminary in User Comments,” in 2021 IEEE 29th International Requirements Results from a Survey in the UK, Sweden, Germany, Austria and the Engineering Conference. IEEE, 2021. Netherlands,” 2016. [43] L. Bornmann and R. Mutz, “Growth Rates of Modern Science: A [22] K. Mao, L. Capra, M. Harman, and Y. Jia, “A Survey of the Use Bibliometric Analysis Based on the Number of Publications and Cited of Crowdsourcing in Software Engineering,” Journal of Systems and References,” Journal of the Association for Information Science and Software, vol. 126, pp. 57–84, 2017. Technology, vol. 66, no. 11, pp. 2215–2222, 2015.