ENVRIplus DELIVERABLE D8.5 Data provenance and tracing for environmental sciences: system design WORK PACKAGE 8 – Data Curation and Cataloging LEADING BENEFICIARY: Umweltbundesamt GmbH (Environment Agency Austria) Author(s): Beneficiary/Institution Barbara Magagna Umweltbundesamt GmbH (EAA) Doron Goldfarb Umweltbundesamt GmbH (EAA) Paul Martin UvA Frank Toussaint, Stephan Kindermann DKRZ Malcolm Atkinson University of Edinburgh Keith Jeffery NERC Margareta Hellström Lund University Markus Fiebig NILU Abraham Nieva de la Hidalga University of Cardiff Alessandro Spinuso KNMI Accepted by: Keith Jeffery (WP 8 leader) Deliverable type: [REPORT] Dissemination level: PUBLIC A document of ENVRIplus project - www.envri.eu/envriplus 1 This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 654182 Deliverable due date: 30.04.2018/M36 Actual Date of Submission: 30.04.2018/M36 2 ABSTRACT This deliverable reports the group efforts of Working Package 8 Task T8.3 on Inter RI data provenance and trace services during M24-36. Project internal reviewer(s): Project internal reviewer(s): Beneficiary/Institution Markus Stocker Technische Informationsbibliothek (TIB) Robert Huber UniHB Document history: Date Version 16th April 2018 Draft for comments 26th April 2018 Corrected version 28th April 2018 Accepted by Keith Jeffery DOCUMENT AMENDMENT PROCEDURE Amendments, comments and suggestions should be sent to the author (Barbara Magagna [email protected]) TERMINOLOGY A complete project glossary is provided online here: https://envriplus.manageprojects.com/s/text-documents/LFCMXHHCwS5hh PROJECT SUMMARY ENVRIplus is a Horizon 2020 project bringing together Environmental and Earth System Research Infrastructures, projects and networks together with technical specialist partners to create a more coherent, interdisciplinary and interoperable cluster of Environmental Research Infrastructures across Europe. It is driven by three overarching goals: 1) promoting cross- fertilization between infrastructures, 2) implementing innovative concepts and devices across RIs, and 3) facilitating research and innovation in the field of environment for an increasing number of users outside the RIs. ENVRIplus aligns its activities to a core strategic plan where sharing multi-disciplinary expertise will be most effective. The project aims to improve Earth observation monitoring systems and strategies, including actions to improve harmonization and innovation, and generate common solutions to many shared information technology and data related challenges. It also seeks to 3 harmonize policies for access and provide strategies for knowledge transfer amongst RIs. ENVRIplus develops guidelines to enhance transdisciplinary use of data and data-products supported by applied use-cases involving RIs from different domains. The project coordinates actions to improve communication and cooperation, addressing Environmental RIs at all levels, from management to end-users, implementing RI-staff exchange programs, generating material for RI personnel, and proposing common strategic developments and actions for enhancing services to users and evaluating the socio-economic impacts. ENVRIplus is expected to facilitate structuration and improve quality of services offered both within single RIs and at the pan-RI level. It promotes efficient and multi-disciplinary research offering new opportunities to users, new tools to RI managers and new communication strategies for environmental RI communities. The resulting solutions, services and other project outcomes are made available to all environmental RI initiatives, thus contributing to the development of a coherent European RI ecosystem. 4 TABLE OF CONTENTS 1 Introduction 10 1.1 Motivation 10 1.2 Task description 10 1.3 Activities in the task 11 1.4 Layout 11 2 Background 12 2.1 Definition and demarcation 12 2.1.1 Historic provenance principles 12 2.1.2 Definition 12 2.1.3 Provenance and metadata 13 2.1.4 Provenance and logs 13 2.2 Basic aspects on provenance 13 2.2.1 Referable object 13 2.2.2 Uses of provenance 14 2.2.3 Provenance life cycle 15 2.2.4 Types of provenance information 15 2.2.5 Challenges 16 2.3 Technology review 17 2.3.1 Provenance for manual processes 18 2.3.2 Provenance for automated processes 19 2.3.3 Provenance data management, discovery and retrieval 23 2.3.4 Provenance visualization 24 2.4 Provenance models 27 2.4.1 Open Provenance Model (OPM) 27 2.4.2 PROV 28 2.4.3 Provenir 33 2.4.4 P-PLAN 33 2.4.5 OPMW 34 2.4.6 ProvONE 35 2.4.7 PROV-Wf 35 2.4.8 S-PROV 36 2.4.9 W4Ever models 37 2.4.10 CERIF 41 2.4.11 Blockchain 42 5 2.4.12 CKAN 42 2.5 Provenance related initiatives 42 2.5.1 The DataONE perspective 42 2.5.2 The ESIP perspective 44 2.5.3 The RDA perspective 45 3 ENVRIplus provenance requirements 46 3.1 Methodology 46 3.2 Requirements and use cases background 47 3.2.1 Definitions 47 3.2.2 RDA Provenance Use Cases 48 3.2.3 Provenance Incubator Group requirements 48 3.2.4 Process for requirements and use case collection 49 3.2.4.1 First requirements gathering (2015/2016) 49 3.2.4.2 Second requirements and use case gathering (2018) 50 3.3 Synopsis on ENVRIplus provenance needs 51 3.3.1 Provenance needs specified for each individual RI 51 3.3.2 Synopsis on use cases 59 3.3.3 ENVRIplus use cases to be included as new Use Cases in the PPD 59 3.3.4 Synopsis on requirements 60 4 Best practices and implementations 60 4.1 RDA Provenance Patterns 60 4.2 EPOS 61 4.2.1 EPOS data lifecycle 61 4.2.2 VERCE 62 4.2.3 S-ProvFlow 64 4.3 IS-ENES 66 4.3.1 Scientific background and application domain 66 4.3.2 ENES data life cycle 67 4.3.3 Use case: data typing in CMIP6 69 4.3.4 IS-ENES provenance support roadmap 71 5 Provenance related ENVRIPLUS Implementation Cases 71 5.1 Provenance-related issues in (dynamic) data identification 71 5.1.1 Dynamic data 71 5.1.2 Subsets and collections 72 5.2 Connecting the particle formation research community to research infrastructure 73 6 5.2.1 Short description 73 5.2.2 Problems 75 5.2.3 Implementation 75 5.2.4 Provenance 75 5.2.5 Discussion and Conclusion 76 5.3 Quantitative Accounting of Open Data Use 76 5.3.1 Objective 77 5.3.2 Achievements 77 5.3.3 Next Steps 78 5.4 Data acquisition aspects of LTER data life cycle expressed in PROV-O 78 5.4.1 Data collection 79 5.4.1.1 Sensor based data collection 79 5.4.1.2 Observation based data collection 80 5.4.1.3 Sample based data collection 81 5.4.2 Data transfer, storage and query/file export 84 5.4.2.1 Import of datasets into central DB 84 5.4.2.2 Creating datasets for export 85 5.4.3 Data upload/export to EUDAT B2Share 85 5.4.4 Lessons learned from modeling LTER Data Life Cycle with PROV 86 6 Recommendations 87 6.1 Provenance in the ENVRI RM 87 6.2 Provenance and semantic linking 89 6.3 Provenance and the ENVRI architecture 90 6.4 Provenance in the longer term 92 6.4.1 A fundamental transition 92 6.4.2 Enabling professional judgement and responsibility 94 6.4.3 Incrementally introducing pervasive provenance 95 6.4.4 Architectural interdependencies 96 6.4.5 Professional groups and modes of working 98 6.4.6 Pervasive Persistent Provenance (P3) 100 6.4.7 Functions 102 6.4.8 Uses 102 6.4.9 Summary and implications 106 7 REFERENCES 107 8 APPENDIX A. PROVENANCE REQUIREMENTS TEMPLATE 114 7 8.1 Template for use cases and requirements 114 8 EXECUTIVE SUMMARY D8.5 Data provenance and tracing for environmental sciences: system design CONTENT OF DOCUMENT What the reader of this report can expect is listed below: Definition and demarcation A description of relevant aspects of provenance management Challenges related to provenance A review of the latest technologies for data provenance An overview on existing standards and provenance models A description of international initiatives on provenance A collection of requirements and use cases of ENVRI Research Infrastructures Bringing this collection in relation to requirements and use cases of other initiatives A description of provenance practices of two ENVRI Research Infrastructures A description of ENVRI provenance related implementation cases How provenance relates to the ENVRI Reference Model and the OIL-E ontology How provenance relates to the ENVRI architecture A long term perspective on provenance 9 1 INTRODUCTION The main objective of the T8.3 “Inter RI data provenance and trace services” is to provide an overview of common standards and best practices related to data provenance management and to increase the scientists’ awareness of the benefit of proper provenance information. 1.1 Motivation Provenance has – over the last decade – gained some prominence in academic discussions surrounding curation, cataloguing and system architecture. There is a broad consensus that provenance is central to the requirement that scientific research should be reproducible. Yet only a few Research Infrastructures (RIs) have implemented provenance services or embedded tools in their system architecture. Data (but not only, also software versions, workflows, etc,) are useful if accompanied by context on how they are captured, processed, analyzed, and validated and other relevant information that enables interpretation and use. This is what provenance is about. For users of data, the scientific basis of their analysis relies to a great extent on the credibility and trustworthiness of their input data. For publishers of data, the provision of provenance as part of their cited data is crucial for scholarship and reproducibility.