STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE IMPLEMENTOR’S GUIDE FOR SDMX FORMAT STANDARDS (VERSION 1.0) STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 Initial Release September 2004 26 First Revision December 2004 27 © SDMX 2004 28 http://www.sdmx.org/ 29 30 31 32 33 3 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 34 1 INTRODUCTION ............................................................................................................................ 5 35 2 SDMX INFORMATION MODEL FOR FORMAT IMPLEMENTORS .............................................. 5 36 2.1 Introduction...................................................................................................................................................5 37 2.2 Fundamental Parts of the Information Model .................................................................................6 38 2.3 Data Set.....................................................................................................................................................6 39 2.4 Attachment Levels and Data Formats....................................................................................................8 40 2.5 Concepts, Definitions, Properties and Rules.......................................................................................9 41 3 SDMX-ML AND SDMX-EDI: COMPARISON OF EXPRESSIVE CAPABILITIES AND 42 FUNCTION .......................................................................................................................................... 15 43 3.1 Format Optimizations and Differences ................................................................................................15 44 3.2 Data Types...................................................................................................................................................16 45 4 SDMX-ML AND SDMX-EDI BEST PRACTICES ........................................................................ 18 46 4.1 Reporting and Dissemination Guidelines......................................................................................18 47 4.1.1 Central Institutions and Their Role in Statistical Data Exchanges..........................................18 48 4.1.2 Defining Key Families ....................................................................................................................19 49 4.1.3 Time and Frequency ......................................................................................................................21 50 4.1.4 Exchanging Attributes....................................................................................................................23 51 4.2 Best Practices for Batch Data Exchange.......................................................................................24 52 4.2.1 More Than One Central Institutions Involved in a Data Exchange.........................................24 53 4.2.2 Positioning of the Dimension "Frequency" .................................................................................26 54 4.2.3 Identification of Key Families........................................................................................................26 55 4.2.4 Identification of the Data Sets.......................................................................................................26 56 4.2.5 Special Issues.................................................................................................................................26 57 4 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 58 1 INTRODUCTION 59 This guide exists to provide information to implementors of the SDMX format standards – SDMX- 60 ML and SDMX-EDI. This document is intended to provide information which will help users of 61 SDMX understand and implement the standards. It is not normative, and it does not provide any 62 rules for the use of the standards, such as those found in SDMX-ML: Schema and 63 Documentation and SDMX-EDI: Syntax and Documentation. 64 This document is organized into parts: 65 • A guide to the SDMX Information Model 66 • Statement of differences in functionality supported by the different formats and syntaxes 67 • Best practices for use of SDMX formats 68 2 SDMX INFORMATION MODEL FOR FORMAT IMPLEMENTORS 69 2.1 Introduction 70 The purpose of this section is to provide an introduction to the SDMX Information Model for those 71 whose primary interest is in the use of the XML or EDI formats. For those wishing to have a deeper 72 understanding of the Information Model, the full SDMX Information Model document provides a more 73 in-depth view, along with UML diagrams and supporting explanation. For those who are unfamiliar 74 with key families, an appendix to the SDMX Information Model provides a tutorial which may serve as 75 a useful introduction. 76 77 The SDMX Information Model is used to describe the basic data and metadata structures used in all 78 of the SDMX data formats. There is a primary division between time series and cross-sectional data 79 and the metadata which describes the structure of that data. The Information Model concerns itself 80 with statistical data and its structural metadata, and that is what is described here. Both structural 81 metadata and data have some additional metadata in common, related to their management and 82 administration. These aspects of the data model are not addressed here. 83 84 This information model is consistent with the GESMES/TS version 3.0 Data Model, with these 85 exceptions: 5 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 86 87 • the “sibling group” construct has been generalized to permit any dimension or dimensions to be 88 wildcarded, and not just frequency, as in GESMES/TS. It has been renamed a “group” to 89 distinguish it from the “sibling group” where only frequency is wildcarded. The set of allowable 90 partial “group” keys must be declared in the key family, and attributes may be attached to any 91 of these group keys; 92 • the section on data representation is now a convention, to support interoperability with 93 EDIFACT-syntax implementations; 94 • cross-sectional data formats are derived from the model, and some supporting features for 95 deriving cross-sectional and time-series views of a single data set structure have been added 96 to the structural metadata descriptions. 97 98 Clearly, this is not a coincidence - the intention is that the GESMES/TS Data Model becomes the 99 foundation for not only the EDIFACT messages, but also the XML used for web dissemination. 100 101 Note that in the descriptions below, text in courier and italicised are the names used in the information 102 model (e.g. DataSet). 103 2.2 Fundamental Parts of the Information Model 104 The statistical information in SDMX is broken down into two fundamental parts - structural metadata 105 (comprising the KeyFamily, and associated Concepts and Code Lists) – see Framework for 106 Standards -, and observational data (The DataSet). This is an important distinction, with specific 107 terminology associated with each part. Data - which is typically a set of numeric observations at 108 specific points in time - is organized into. data sets (DataSet) These data sets are structured 109 according to a specific key family (KeyFamily), and are described in the data flow definition 110 (DataFlowDefinition) The key family describes the metadata that allows an understanding of 111 what is expressed in the data set, whilst the data flow definition provides the identifier and other 112 important information (such as the periodicity or reporting) that is common to all of its component data 113 sets. 114 2.3 Data Set 115 Data sets are made up of a number of time series or sections (the cross-sectional organization of 116 observations at a single point in time). In addition to the numeric observation (Observation)and the 117 related date (TimePeriod), which are the core of the time series, there may be attributes 6 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 118 (AttributeValue) indicating the status of the observation, e.g. whether the value is a normal or 119 break value, etc. These attributes may be optional (or “conditional”), and may have coded or free text 120 values. They may pertain to any part of the data set - each observation might have a different value 121 for the attribute, or there might be only a single attribute value describing the entire data set, or for 122 each time series, etc. 123 124 Each time series can be identified by the values of its dimensions. Time series data can be seen as 125 n-dimensional. A given time series will have exactly one value (KeyValue), of the set of permissible 126 values, for each of its dimensions (Dimension), and a set of observations (Observation): one 127 value for each specific point in time (TimePeriod). A specific time series might have dimensions of 128 "frequency", "topic", "stock or flow", "reporting country", etc., with a single corresponding value for 129 each dimension. Taken together, this set of values uniquely identifies the time series within its data 130 set, and is called the time series key (TimeSeriesKey). 131 132 Cross-sectional representations of the data may be derived from the same key family from which 133 time-series representations are structured, so long as the needed additional structural metadata is 134 provided. This functionality allows multiple measures to be declared in the key family, associated with 135 the representational values of one dimension.