Updated Guidelines for Seadatanet ODV Production
Total Page:16
File Type:pdf, Size:1020Kb
EMODnet Thematic Lot n° 4 - Chemistry EMODnet Phase III Updated guidelines for SeaDataNet ODV production M. Lipizer, M. Vinci, A. Giorgetti, L. Buga, M. Fichault, J. Gatti, S. Iona, M. Larsen, R. Schlitzer, D. Schaap, M. Wenzer, E. Molina Date: 12/04/2018 EMODnet Thematic Lot n° 4 - Chemistry Updated guidelines for SDN ODV production Index EMODnet Phase III............................................................................................................................1 Updated guidelines for SeaDataNet ODV producton......................................................................1 Updated guidelines for SeaDataNet ODV producton..........................................................................1 Introducton......................................................................................................................................1 SeaDataNet ODV import format.......................................................................................................1 How to check your SeaDataNet ODV fle format?............................................................................6 Vocabulary........................................................................................................................................7 How to choose the correct P01?......................................................................................................8 Flagging of Data Below Detecton Limits and Data Below Limit of Quantfcatoni.......................13 Guidelines for SeaDataNet ODV producton for sediment data and contaminants in biota.............14 Contaminants in the Sediment.......................................................................................................14 Contaminants in Biota....................................................................................................................18 Partcular case in biota datai......................................................................................................18 Other general issues and inventory of common errors.................................................................21 Acknowledgements: We acknowledge the fundamental contribution of the British Oceanographic Data Centre (BODC) for the design, development and continuous management and update of the Vocabularies and for the provision of the webservices. How to cite this document: M. Lipizer, M. Vinci, A. Giorgetti, L. Buga, M. Fichault, J. Gatti, S. Iona, M. Larsen, R. Schlitzer, D. Shaap, M. Wenzer, E. Molina, 2018, EMODnet Phase III - Updated guidelines for SeaDataNet ODV production, 12/04/2018, 23 pp., DOI: 10.6092/259c43eb-4ba4- 419b-bb38-df00e189bd35 0 EMODnet Thematic Lot n° 4 - Chemistry Updated guidelines for SDN ODV production Updated guidelines for SeaDataNet ODV production Introduction EMODnet Chemistry uses SeaDataNet infrastructure (https://www.seadatanet.org/) for the technical set-up. In particular, it adopts: A set of Standards for metadata description and data formats: . SeaDataNet (SDN) Standards for metadata (https://www.seadatanet.org/Standards/Metadata-formats) . Common Vocabularies (i.e. standardised terms that cover a broad spectrum of disci- plines) to allow consistency and interoperability (https://www.seadatanet.org/Standards/Common-Vocabularies) . Common Data Index mechanism (CDI) to access data with data policy (https://www.seadatanet.org/Standards/Metadata-formats/CDI) . Ocean Data View format (ODV) for data exchange (https://www.seadatanet.org/Standards/Data-Transport-Formats) . Data Quality Control procedures (https://www.seadatanet.org/Standards/Data-Quality-Control) A set of software tools specifically developed for metadata and data formatting, data exchange and visualization: . MIKADO to prepare XML metadata files . NEMO to enable conversion from any type of ASCII format to the SeaDataNet ODV and Medatlas ASCII formats as well as the SeaDataNet NetCDF (CF) format . OCTOPUS to convert files in a given SeaDataNet format to another SeaDataNet format (e.g.: ODV to NetCDF, MedAtlas to NetCDF, MedAtlas to ODV) and to check the compli- ancy of SeaDataNet MedAtlas and ODV files format . ODV as the fundamental data analysis and visualisation software. DIVA to spatially interpolate (or analyse) observations on a regular grid in an optimal way SeaDataNet ODV import format All data entering EMODnet Chemistry must be converted to SeaDataNet ODV import format, which is the common standard format. Delivery of data to users requires common data trans- port formats, which interact with other SeaDataNet standards (Vocabularies, Quality Flag Scale) and SeaDataNet analysis & presentation tools (ODV, DIVA). Detailed guidelines on Data transport formats area for data in the water column and specific for biological data are avail- able on: https://www.seadatanet.org/Standards/Data-Transport-Formats (Fig. 1). All SDN ODV files are .txt files. 1 EMODnet Thematic Lot n° 4 - Chemistry Updated guidelines for SDN ODV production Fig. 1: SeaDataNet datafile formats documentation (https://www.seadatanet.org/Standards/Data-Transport-Formats) From: “SeaDatanet DATAFILE FORMATS”: The fundamental data model underlying the format is the ASCII spreadsheet (ODV spread- sheet files may contain comments, and the column separation character may be TAB or semi- colon, https://odv.awi.de/fileadmin/user_upload/odv/misc/odv4Guide.pdf ) : i.e. a collection of rows each having the same fixed number of columns. There are three different types of col- umn: . Metadata columns . Primary variable data columns (one column for the value plus one for the qualifying flag) . Data columns (one column for the value plus one for the qualifying flag) The metadata columns are stored at the left hand end of the row, followed by the primary variable columns and then the data columns. There are three different types of rows: . Comment rows 2 EMODnet Thematic Lot n° 4 - Chemistry Updated guidelines for SDN ODV production . Semantic header row . Column header row . Data row Comment lines start with two slashes //as first two characters of the line and may contain ar- bitrary text in free format. Comment lines may, in principle, appear anywhere in the file, most commonly, however, they are placed at the beginning of the file and contain descriptions of the data, information about the originator or definitions of the variables included in the file. The Semantic header consists in a set of ‘special’ mandatory comment lines that must appear before the column header row. Their function is to map the text strings used to label the meta- data and data columns to standardized SeaDataNet concepts, which is necessary if data files from different sources are to be combined in a meaningful way (Fig. 2, Fig. 3). Fig. 2: Example of SDN ODV file, with depth as primary variable (in column 10): Important: there MUST be exact correspondence between Semantic header and Header. The order of the first 9 metadata column MUST be respected (ie. Cruise; Station; Type; date-time; Long.; Lat.; Local_CDI_ID; EDMO_Code; Bot. Depth.). 3 EMODnet Thematic Lot n° 4 - Chemistry Updated guidelines for SDN ODV production The metadata values are followed by the primary variable value (on the 10th col- umn), its flag and then the data value plus qualifying flag pairs for each parameter. Flag values are taken from the SeaDataNet vocabulary for qualifying flags (L20 at http://vocab.ner- c.ac.uk/collection/L20/current). Fig. 3: Example of SDN ODV file showing correspondence between SDN_parameter_map- ping lines and measured parameters. The primary variable depends on the type of dataset: Time series: point time series have row_groups made up of measurements from a given in- strument at different times. The primary variable (column 10) is time. In case data come from samples taken in the same position and sampled at a fix frequency, the dataset type can be considered as “time series” and the SDN ODV file must provide time as primary variable in the 10th column. This is particularly suitable for regu- lar monitoring stations and is commonly used for mooring data (eg. oceanographic buoy). Vertical profile: profile data have row_groups made up of measurements at different depths. The primary variable (which is indicated in column 10) is the ‘z co-ordinate’, which is either depth in meters or pressure in decibars (in case of water column data), or depth below seabed (in case of sediment data). It is the choice for stations usually not repeated regularly over time, for which several depths are sampled. Typical ex- amples: CTD profiles, bottle stations (chemical data, chlorophyll, plankton,…) However, this distinction is not always clear and in cases of a same position, sampled over time at different depths, some data providers prefer to consider these as “repeated profiles” stations. 4 EMODnet Thematic Lot n° 4 - Chemistry Updated guidelines for SDN ODV production Lastly, when data are not available for the same position at more or less regular time periods, the dataset type is regarded as “vertical profiles” even if there is only 1 sampling depth and it is not a real profile. This can be the case of sediment samples taken during occasional surveys, for example. In order to produce correctly SDN ODV datasets for EMODnet Chemistry, the use NEMO software is STRONGLY RECCOMENDED. User Manual, NEMO presentation and NEMO examples are available on: https://www.seadatanet.org/Software/NEMO and lessons on NEMO given during EMODnet Chemistry 3 first training (18-19 May 2017, Trieste) are available on: http://www.emodnet-chemistry.eu/trieste_16-19_May_2017.html.