STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE SDMX-ML: SCHEMA AND DOCUMENTATION (VERSION 2.0) November 2005 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 © SDMX 2005 52 http://www.sdmx.org/ 2 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 53 Contents 54 55 1 BACKGROUND............................................................................................................ 5 56 1.1 History and Version 2.0 Developments ........................................................................ 5 57 1.2 The XML Design........................................................................................................... 5 58 1.3 Fostering the Use of a Standard SDMX-ML................................................................. 6 59 2 NORMATIVE REFERENCES....................................................................................... 6 60 3 CONFORMANCE ......................................................................................................... 6 61 4 DESIGN OVERVIEW ................................................................................................... 6 62 4.1 Scope and Requirements ............................................................................................. 6 63 4.2 Design Approach .......................................................................................................... 8 64 4.3 SDMX-ML Packaging: Namespace Modules ............................................................. 10 65 5 GENERIC (NON-STRUCTURE-DEFINITION-SPECIFIC) SCHEMAS...................... 12 66 5.1 SDMX Message Namespace Module ........................................................................ 12 67 5.2 SDMX Structure Namespace Module......................................................................... 18 68 5.3 SDMX Generic Data Namespace Module.................................................................. 65 69 5.4 SDMX Generic Metadata Namespace Module .......................................................... 69 70 5.5 SDMX Query Namespace Module ............................................................................. 73 71 5.6 SDMX Common Namespace Module ........................................................................ 82 72 5.7 SDMX Registry Interfaces Namespace Module......................................................... 86 73 5.8 Data Formatting and Character Encoding................................................................ 105 74 5.9 Missing Observation Values ..................................................................................... 105 75 6 KEY-FAMILY- AND METADATA-STRUCTURE-DEFINITION-SPECIFIC SCHEMAS: 76 CORE STRUCTURES & STANDARD MAPPINGS .............................................................. 105 77 6.1 Compact Data Message Core Structure .................................................................. 106 78 6.2 Utility Data Message Core Structure ........................................................................ 108 79 6.3 Cross-Sectional Data Message Core Structure ....................................................... 110 80 6.4 Metadata Report Core Structure .............................................................................. 111 81 6.5 Mappings to Key-Family-Specific Data Schemas .................................................... 113 3 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 82 6.6 Mappings to Metadata Structure Definition-Specific Metadata Schemas ................ 125 83 7 APPENDIX: SAMPLE SDMX-ML DATA MESSAGES ............................................. 127 84 7.1 CompactSample.xml ................................................................................................ 127 85 7.2 UtilitySample.xml ...................................................................................................... 129 86 7.3 GenericSample.xml .................................................................................................. 129 87 7.4 CrossSectionalSample.xml....................................................................................... 130 88 4 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 89 1 BACKGROUND 90 1.1 History and Version 2.0 Developments 91 The SDMX Technical Standards Version 1.0 established an information model which 92 described aggregated statistical data sets and the structural metadata needed to 93 exchange them in a standard fashion. This drew on the earlier example of the 94 GESMES/TS standard. Based on the SDMX information model, several formats were 95 developed: XML formats for exchange of structural metadata, data sets, and queries 96 for these (SDMX-ML), and EDIFACT formats for the structural metadata and data 97 sets (SDMX-EDI). These standards supported a number of exchange patterns, 98 characterized as "bilateral", "gateway", and "data-sharing" models, as described in 99 the Framework document in the Version 1.0 standards package. 100 101 Version 2.0 builds on this foundation to provide a higher degree of support for all of 102 these models, with an emphasis on data sharing in the form of a set of standard 103 registry services interfaces. It has also expanded to include support for new types of 104 metadata exchange and reporting, with a focus on "reference metadata" concerned 105 with quality, methodology, and other issues. Further, the ability to provide metadata 106 about the relationships between data sets and structures has been expanded, 107 providing more support for data cubes. Finally, experience has shown that some 108 minor additions to the existing structural metadata and dataset structures will allow 109 SDMX to support more different types of statistical information. 110 111 The scope of the Version 2.0 SDMX Technical Standards is thus much broader, and 112 is accompanied by a larger set of message types in the SDMX-ML formats. While the 113 XML formats described here have grown in number and scope, the EDI formats 114 remain relatively unchanged. 115 1.2 The XML Design 116 All of these document types will share a common "envelope" at the message level 117 ("SDMXMessage.xsd"), as well as a set of common low-level components 118 (“SDMXCommon.xsd”) so that header information and basic structure will always be 119 the same. 120 121 • Schema for describing all types of structural metadata – for data sets (key 122 families), for metadata sets (metadata structure definitions), for related groups 123 of metadata and data structures, and for all types of structural objects 124 involved in registry-based exchanges ("SDMXStructure.xsd") 125 • Generic data schema for data-sharing exchange ("SDMXGenericData.xsd") 126 • Generic query schema for invoking web services ("SDMXQuery.xsd") 127 • Key-family-specific schema for updates and revisions/bilateral exchange 128 ("SDMXCompactData.xsd") 129 • Key-family-specific schema for presentational processing and internal use 130 ("SDMXUtilityData.xsd") 5 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 131 • Key-family-specific schema for cross-sectional data 132 ("SDMXCrossSectionalData.xsd") 133 • Generic schema for registry interfaces (“SDMXRegistry.xsd”) 134 • Generic schema for reference metadata sets (“SDMXRefMetadata.xsd”) 135 • Metadata-structure-definition-specific schema for metadata sets 136 (“SDMXMetadataReport.xsd”) 137 138 1.3 Fostering the Use of a Standard SDMX-ML 139 In addition to these different formats, standard mappings and corresponding 140 transformation tools have been developed for the creation of key-family-specific 141 schemas from structure descriptions, to transform XML data instances from one XML 142 data description format to another, and from these formats into the corresponding 143 SDMX-ML messages. This level of free tools support will foster the early use of 144 SDMX and permit the data to be easily used across all processes, which is otherwise 145 a difficult requirement to meet. Ultimately, it is the fact that all formats share a 146 common information model that enables this approach to meet the wide set of SDMX 147 requirements. 148 149 2 NORMATIVE REFERENCES 150 W3C XML Schema Definition Language, version 1.0 (URL: 151 http://www.w3c.org/XML/Schema#dev), World Wide Web Consortium 152 W3C Extensible Markup Language, version 1.0, Third Edition (URL: 153 http://www.w3c.org/TR/2004/REC-xml-20040204/), World Wide Web Consortium 154 155 3 CONFORMANCE 156 Sections V and VI of this document are normative, providing rules for the creation of 157 conformant SDMX-ML XML instances and W3C XML Schemas. 158 159 4 DESIGN OVERVIEW 160 4.1 Scope and Requirements 161 To understand the relationships between the several document types, it is important 162 to have some familiarity with the requirements they are designed to fulfil. 163 164 • Large amounts of data must be captured in a reasonably compact format, 165 because of the potential size of databases being exchanged. 166 • It must be possible to send incremental updates, rather than entire, complete 167 databases. The validation of such exchanges demands not that an entire data 168 set be exchanged, but only that enough information be sent to ensure 169 accurate updating and revision processes. 6 STATISTICAL DATA AND METADATA EXCHANGE INITIATIVE 170 • Structural information as well as data will need to be transmitted. 171 • There must be a reliable transformation to and from the GESMES/TS 172 EDIFACT syntax. 173 • It should be possible to present natural-language