RMLEditor User Evaluation UC2

1. Introduction

Thank you for participating in our research.

This survey supports the user evaluations of the RMLEditor, a user-oriented visual notation for mapping rules which define how is generated from raw data. To be able to sensefully complete the survey a basic understanding of the RDF Mapping Language (RML) is a prerequisite.

The results will be processed anonymously and will only be used for non-commercial, scientific purposes.

The survey has been set up as part of a research project of IDLab (http://idlab.technology/). For more information concerning this research or for other questions, you can contact Pieter Heyvaert ([email protected]). 2. Socio-Demographics and Skill Level

The questions in this section intent to get a quick overview of the socio-demographics of the participants and their experience with Linked Data.

4.1. What is your year of birth?

4.2. What is your gender? . Male . Female

4.3. What is the highest level of education that you completed? . Did not complete high school . High school . Bachelor’s degree . Master’s degree . Advanced graduate work or Ph.D . Not sure

4.4. What is your employment status? . Employed for wages . Self-employed . Out of work . A homemaker . A Student . Retired

4.5. What is your experience with Linked Data? (Multiple options possible) . I generate Linked Data . I check the quality of Linked Data . I use Linked Data . I publish Linked Data . I perform reasoning on Linked Data . I have a basic understanding of Linked Data . I have no knowledge about Linked Data

4.6. Please indicate where you assess yourself on the following scale: . Where do you assess yourself in relation to the topic of Linked Data? . (1: Novice / 2: Emerging / 3: Developing / 4: Proficient / 5: Expert)

4.7. Do you already create custom tools / software / scripts to generate Linked Data? (Yes/No)

4.8. Please specify which custom tools / software / scripts you have created (open question) 4.9. Please ndicate if you have heard of or used any of these tools:

I have used this tool I have heard of this I haven’t used nor tool heard of this tool Juma FluidOps editor Map-On RMLx Visual Editor Karma Other

4.10. If you have a research/professional position in the , what are your main topics? (open question)

4.11. Have you ever head of the RMLEditor? . I have never heard about it . I have heard about it, but I am not quite sure what it does . I have heard about it and I know what it does . I have heard about it and I have already used the RMLEditor 3. Use Case 1: RMLx Visual Editor (1/2)

This the central part of the test. A use case will be presented, which you can run through, making use of the RMLx Visual Editor.

4.1. Use Case 1: RMLx Visual Editor (1/2) The goal of this use case is to generate Linked Data, making use of the RMLx Visual Editor, based on the existing data about employees and the projects they are working on. We provide you with two data sources: employees.csv and projects.csv. These can be found here (rml.io/data/rmleditor-evaluation/input-files.zip), in the folder "use case 1".

Each column contains information about a specific employee or project. The project_id column of employees.csv depicts on which project an employee works by stating the project’s id. This id is the same as the value for the id column of projects.csv.

Use a unique URI for the employees: http://www.example.com/employee/ concatenated with its ID. Use a unique URI for the projects: http://www.example.com/project/ concatenated with its ID. Templating can be done by using http://www.example.com/employee/{ID}. Mind the { and } around the column name.

For this use case, generate mappings by starting with the data. Semantically annotating the data happens afterwards.

For annotating the data, we expect you to reuse concepts from the ontologies FOAF, schema.org and XSD, but only when possible. You can consult the Linked Open Vocabularies (LOV). If you are using the RMLEditor you can do this via the GUI, if not you can go to http://lov.okfn.org/dataset/lov/.

The title of the projects all have to be in uppercase in the triples.

When you are finished, you will be asked to provide us with the mapping document.

Remarks As the tools do not have an undo button, we advise you to export your mapping from time to time (by exporting the graph). Refreshing the page will undo all your actions. When you encounter a problem that blocks your progress, do not hesitate to ask us for help. .2. Post-Assessment RMLx Visual Editor In this section, we want to gather some insights on your opinion about working with the RMLx Visual Editor

3.1. How difficult was it to perform the tasks? . Extremely easy / moderatly easy / slightly easy / neither easy nor difficult / slightly difficult / moderately difficult / extremely difficult

3.2. I am confident that I executed the tasks successfully . Strongly agree / agree/ somewhat agree / neither agree nor disagree / somewhat disagree / disagree / strongly disagree

3.3. What features of the environment did you find the most interesting? (open question)

3.4. Please indicate below how much you agree with the following statements?

Strongly Somewhat Neither Somewhat Strongly agree agree agree nor disagree disagree disagree I think I would like to use the RMLx Visual Editor frequently I found the RMLx Visual Editor unnecessaril y complex I thought the RMLx Visual Editor was easy to use I think that I would need the support of a technical person to be able to use the RMLx Visual Editor I found the various functions in the RMLx Visual Editor were well integrated I thought there was too much inconsistency in the RMLx Visual Editor system I would imagine that most people would learn to use the RMLx Visual Editor very quickly I found the RMLx Visual Editor very cumbersome to use I felt very confident using the RMLx Visual Editor I needed to learn a lot of things before I could get going with the RMLx Visual Editor

3.5. Overall, I would rate the user-friendliness of the RMLx Visual Editor as … . Worst imaginable / awful / poor / ok / good / excellent / best imaginable

3.6. How would you assess the difficulty of the RMLx Visual Editor? . Extremely easy / Moderately easy / Slightly easy / Neither easy nor difficult / slightly difficult / moderately difficult / Extremely difficult

Do you have any other feedback regarding your experience with the RMLx Visual Editor? (open question) 3. Use Case 2: RMLEditor (2/2) A use case will be presented, which you can run through, making use of the RML Editor.

In order to be able to work with the tool, you will need some introduction to the visualisation used, this can be found here (https://docs.google.com/document/d/1- 116zmhUxj6zavwDaLUT9xRa5g5REGZc31snA1cn9YQ/edit) . Please take some time to read through it.

An introduction to the RML Editor, can be found here (https://docs.google.com/document/d/1y3_rj5GT3SCEiPRGURyyZ5eGHK59cD- LgxQN18fNjrs/edit).

.2. Use Case 2: RMLEditor (2/2)

The goal of this use case is to generate Linked Data, making use of the RMLEditor, based on the existing data about movies and the directors that created them. We provide you with two data sources: movies.csv and directors.json. These can be found here (rml.io/data/rmleditor-evaluation/input-files.zip), in the folder "use case 2".

The director_id column of movies.csv depicts which movie a director created by stating the director’s id. This id is the same as the value for the ID attribute of directors.json.

Use a unique URI for the movies: http://www.example.com/movie/ concatenated with its ID. Use a unique URI for the directors: http://www.example.com/director/ concatenated with its ID. Templating can be done by using http://www.example.com/movie/{ID}. Mind the { and } around the column/attribute name.

For this use case, generate mappings by starting with the ontologies. Selecting the correct data from the data sources is done afterwards. For annotating the data, we expect you to use the following concepts:

Concepts/Types Link (to ontology description) Shortcut film, movie http://dbpedia.org/ontology/Film :Film person http://xmlns.com/foaf/0.1/Person :Person title http://purl.org/dc/terms/title dcterms:title release date http://dbpedia.org/ontology/releaseDate dbpedia:releaseDate data is of the type http://www.w3.org/2001/XMLSchema#date xsd:date 'date' is director of h ttp://dbpedia.org/ontology/director dbpedia:director first name, given http://xmlns.com/foaf/0.1/givenname foaf:givenname name family name, last http://xmlns.com/foaf/0.1/familyName foaf:familyName name birth date http://dbpedia.org/ontology/birthDate dbpedia:birthDate birth place http://dbpedia.org/ontology/birthPlace dbpedia:birthPlace

The title of the movies all have to be in uppercase in the triples.

When you are finished, you will be asked to provide us with the mapping document.

Remarks  As the tools do not have an undo button, we advise you to export your mapping from time to time (by exporting the graph).  Refreshing the page will undo all your actions.  When you encounter a problem that blocks your progress, do not hesitate to ask us for help. 4.2. Extra questions RMLEditor In this part you will get presented with a number of images. Please answer the corresponding questions. If something is unclear, please do not hesitate to contact the researcher present.

A. Different Panels (1/4)

4.1. In which panel can you find a sample of the input data? . Input panel . Modelling panel . Results panel

.2. In which panel do you define the data transformations? . Input panel . Modelling panel . Results panel

4.3. Which panel do you use to determine whether datasets are linked? . Input panel . Modelling panel . Results panel B. Heterogeneous Data Sources (2/4)

4.4. How many data sources do you see in the following picture?

4.5. How many data fractions do you see in the following picture? 4.6. How many data fractions do you see in the following picture?

4.7. How difficult did you find it to associate the data fractions with the corresponding data values of the input data sample? . Extremely easy / somewhat easy / neither easy or difficult / somewhat difficult / extremely difficult C. Heterogeneous data values (3/4)

4.8. What data transformation was applied to this term?

4.9. To which data fractions was this data transformation applied? (open question)

4.10. How many transformed literals can you perceive? 4.11. How easy it to understand and apply the transformations? . Extremely easy / somewhat easy / neither easy or difficult / somewhat difficult / extremely difficult D. Zooming (4/4)

4.12. Which of the following figures do you prefer for editing a resource’s attributes? 4.12.1. Dense without zooming

4.12.2. Less dense with zooming

4.12.3. Least dense with more zooming . Dense without zooming . Less dense with zooming . Lease dense with more zooming . No preference

4.13. Which of the following figures do you prefer for adding and editing links between resources? 4.13.1. Dense without zooming

4.13.2. Less dense with zooming

4.13.3. Rare with zooming . Dense with zooming . Less dense with zooming . Rare with zooming . No preference

4.14. How many Linked Data set islands can you identify? An island is a set of nodes which are not connected to others.

4.15. How many Linked Data set islands can you identify? An island is a set of nodes which are not connected to others. 4.16. How many Linked Data set islands can you identify? An island is a set of nodes which are not connected to others.

4.17. How many entities are completely unlinked to other entities? 4.3. Post-Assessment RMLEditor In this section, we want to gather some insights on your opinion about working with the RMLEditor

4.18. How difficult was it to perform the tasks? 4.18.1.1. Extremely easy / moderatly easy / slightly easy / neither easy nor difficult / slightly difficult / moderately difficult / extremely difficult

4.19. I am confident that I executed the tasks successfully 4.19.1. Strongly agree / agree/ somewhat agree / neither agree nor disagree / somewhat disagree / disagree / strongly disagree

4.20. What features of the environment did you find the most interesting? (open question)

4.21. Please indicate below how much you agree with the following statements?

Strongly Somewhat Neither Somewhat Strongly agree agree agree nor disagree disagree disagree I think I would like to use the RMLEditor frequently I found the RMLEditor unnecessaril y complex I thought the RMLEditor was easy to use I think that I would need the support of a technical person to be able to use the RMLEditor I found the various functions in the RMLEditor were well integrated I thought there was too much inconsistency in the RMLEditor system I would imagine that most people would learn to use the RMLEditor very quickly I found the RMLEditor very cumbersome to use I felt very confident using the RMLEditor I needed to learn a lot of things before I could get going with the RMLEditor

4.22. Overall, I would rate the user-friendliness of the RMLEditor as … 4.22.1. Worst imaginable / awful / poor / ok / good / excellent / best imaginable

4.23. How would you assess the difficulty of the RMLEditor? 4.23.1. Extremely easy / Moderately easy / Slightly easy / Neither easy nor difficult / slightly difficult / moderately difficult / Extremely difficult

4.24. Do you have any other feedback regarding your experience with the RMLEditor? (open question)