PaN-data ODI Deliverable D5.3 Report on the implementation of the three vir- tual laboratories Grant Agreement Number RI-283556 Project Title PaN-data Open Data Infrastructure Title of Deliverable Report on the implementation of the three virtual laboratories Deliverable Number D5.3 Lead Beneficiary STFC Deliverable Public Dissemination Level Deliverable Nature Report Contractual Delivery Date 31 Aug. 2014 Actual Delivery Date XX Oct. 2014 The PaN-data ODI project is partly funded by the European Commission under the 7th Framework Programme, Information Society Technologies, Research Infrastructures. PaN-data ODI Deliverable: D5.3 Abstract This report demonstrates and evaluates the services being developed within the PaN-data ODI project. A survey of the implementation status of these services at all participating institutes is in- cluded. The metadata catalogue plays a central role in the PaN-data project. Therefore it is dis- cussed how scientists benefit from this service. The status of the tomography database is re- viewed. Details of the NeXus deployment at various partners are presented. Eventually, three data processing frameworks, which are developed within the PaN-data community, are compared by implementing two scientific use cases. Keyword list PaN-data ODI, virtual laboratories, metadata, data catalogue, standard data format Document approval Approved for submission to EC by all partners on XX.10.2014 Revision history Issue Author(s) Date Description 1.0 Thorsten Kracht Aug 2014 First version 1.1 Jan Kotanski Aug 2014 NeXus deployment at DESY 1.2 Gero Flucke Aug 2014 Comparison of data processing frameworks 1.3 Eugen Wintersberger Aug 2014 Status of the NeXus library 1.4 Mark Koennecke Jul 2014 NeXus deployment at PSI-SINQ 1.5 Jens-Uwe Hoffmann Aug 2014 NeXus deployment at HZB 1.6 Guifre Cuni Aug 2014 NeXus deployment at ALBA 1.7 Frank Schluenzen Aug 2014 Science3D section 1.8 Tobias Richter Sep 2014 NeXus deployment at DLS 1.9 Jean-Francois Perrin Sep 2014 NeXus deployment at ILL 1.10 Tom Griffin Sep 2014 NeXus deployment at ISIS Page 2 of 107 PaN-data ODI Deliverable: D5.3 Table of contents Page 1 INTRODUCTION ..................................................................................................................... 6 1.1 THE AMENDMENT OF THE DESCRIPTION OF WORK .................................................................... 6 1.2 OVERVIEW OF THE DELIVERABLE ............................................................................................ 7 2 SURVEY OF THE IMPLEMENTATION OF THE PAN-DATA ODI SERVICES ....................... 9 2.1 SCIENTIFIC DATA POLICY ....................................................................................................... 9 2.2 COMMON IDENTITY MANAGEMENT SYSTEM UMBRELLA ........................................................... 10 2.3 COMMON DATA FORMAT NEXUS/HDF5 ................................................................................. 10 2.4 METADATA CATALOGUE ICAT-4 ........................................................................................... 12 3 DEMONSTRATION AND EVALUATION OF THE PAN-DATA SERVICES .......................... 15 3.1 INTRODUCTION.................................................................................................................... 15 3.2 VL1: POWDER DIFFRACTION ................................................................................................ 15 3.3 VL2: SMALL ANGLE SCATTERING .......................................................................................... 18 3.4 ISIS ICAT .......................................................................................................................... 19 3.5 SUMMARY ........................................................................................................................... 24 4 SCIENCE 3D – THE TOMOGRAPHY DATABASE .............................................................. 25 4.1 THE SCIENCE 3D KICK-OFF WORKSHOP ............................................................................... 26 4.2 CONCEPT OF THE SCIENCE 3D OPEN ACCESS CATALOGUE .................................................... 26 4.3 CURRENT STATUS OF THE SCIENCE 3D OPEN ACCESS CATALOGUE ....................................... 29 4.3.1 Schema ...................................................................................................................................... 30 4.3.2 Catalog ....................................................................................................................................... 30 4.3.3 Documentation & Dissemination ................................................................................................ 31 4.3.4 Summary .................................................................................................................................... 31 5 NEXUS DEPLOYMENT ........................................................................................................ 33 5.1 INTRODUCTION.................................................................................................................... 33 5.2 STATUS OF THE NEXUS LIBRARY .......................................................................................... 33 5.2.1 Why implementing a new Nexus library? ................................................................................... 33 5.2.2 Project status ............................................................................................................................. 34 5.2.3 Performance markers for libpnicore ........................................................................................... 35 5.2.4 Performance markers for libpniio ............................................................................................... 36 5.3 EXTERNAL FILTERS.............................................................................................................. 37 5.3.1 First implementation of an external filters infrastructure ............................................................ 38 5.3.2 External filters and user applications ......................................................................................... 39 5.3.3 External filters with Python ......................................................................................................... 40 5.4 NEXUS INTEGRATION AT DESY ........................................................................................... 41 5.4.1 Overview .................................................................................................................................... 41 5.4.2 The configuration procedure ...................................................................................................... 42 Page 3 of 107 PaN-data ODI Deliverable: D5.3 5.4.3 Component selection ................................................................................................................. 45 5.4.4 Mandatory metadata .................................................................................................................. 47 5.4.5 The data acquisition process ..................................................................................................... 48 5.4.6 Generalizing the DESY NeXus implementation......................................................................... 49 5.5 NEXUS INTEGRATION AT ALBA ............................................................................................ 49 5.5.1 Overview .................................................................................................................................... 49 5.5.2 Application definitions ................................................................................................................ 51 5.5.3 Data and metadata sources selection ....................................................................................... 51 5.6 NEXUS INTEGRATION AT DLS .............................................................................................. 53 5.6.1 Overview .................................................................................................................................... 53 5.6.2 Device selection ......................................................................................................................... 53 5.6.3 Configuration .............................................................................................................................. 54 5.6.4 Default metadata ........................................................................................................................ 54 5.6.6 Application definitions ................................................................................................................ 55 5.7 NEXUS INTEGRATION AT HZB .............................................................................................. 55 5.7.1 Overview .................................................................................................................................... 55 5.7.2 Mandatory metadata .................................................................................................................. 56 5.7.3 Generalizing the HZB NeXus implementation ........................................................................... 57 5.8 NEXUS INTEGRATION AT ILL ................................................................................................ 57 5.8.1 Configuration .............................................................................................................................