JOINT ADB – UNESCAP SDMX CAPACITY BUILDING INITIATIVE SDMX STARTER KIT FOR NATIONAL STATISTICAL AGENCIES Version: 5 October 2014 1 FOREWORD The aim of the Starter Kit is to provide a resource for national statistical agencies in countries in the Asia and Pacific region contemplating the implementation of the Statistical Data and Metadata Exchange (SDMX) technical standards and content-oriented guidelines for the exchange of aggregate data and their related methodological information (metadata). It outlines a structured process for the implementation of SDMX by agencies that have little knowledge about SDMX and how they would go about the implementation of the standards and guidelines. In order to avoid duplicating the work of the SDMX sponsoring agencies the Kit makes extensive use of links to existing SDMX background documents and artifacts that have been developed at the global level. The Starter Kit was developed under the auspices of the joint 2014 Asian Development Bank– United Nations Economic and Social Commission for Asia and the Pacific SDMX initiative to improve the efficiency of data and metadata exchange between national statistical agencies in the Asia and Pacific region and international organizations through the ongoing use of SDMX standards. The aim of the joint initiative is to build the capacity of countries in the region to apply SDMX standards through mapping national concepts to specific identified SDMX Data Structure Definitions (DSDs) and Metadata Structure Definitions (MSDs). The initiative also aims at enabling national agencies to determine which, of a range of available tools for SDMX implementation, best meets the needs of the organisation. The basic premise of the Kit is that SDMX implementation must be seen in the context of a wide range of corporate institutional, infrastructure and statistical initiatives currently underway in almost all statistical agencies around the globe to improve the quality and relevance of the service they provide to government and non-government users of their outputs. Such corporate initiatives are often undertaken within the framework of a national strategy for the development of statistics (NSDS), statistical master plans or similar strategic planning process. A number of corporate initiatives relevant to SDMX are outlined in the Implementation Plan for the Regional Programme for the Improvement of Economic Statistics in Asia and the Pacific endorsed by the ESCAP Committee on Statistics in December 2012. Planned improvements from the Implementation Plan covered areas such as advocacy, coordination, statistical infrastructures, and skills. SDMX implementation entails considerable use of resources, expertise and effort at the national level and these will only be forthcoming if a substantive business case of the potential benefits can be made for SDMX implementation, particularly in the context of both broader national strategic goals and alternative technical standards. The Starter Kit presents the business case for SDMX implementation from the perspectives of both international and national statistical agencies. The intended audience are senior management in national agencies as well as statistical subject matter experts. The business case outlined was derived from NSDSs that have been prepared by countries in the Asia and Pacific region which are available on the Paris21 website and websites of several other national statistical organisations in the region. The final key theme throughout the Starter Kit is that SDMX implementation by national statistical agencies is not just an IT issue but one that requires close collaboration between different parts of the organisation, in particular, management and statisticians working in the domains being implemented. The intended broad national audience for the kit therefore includes, national statistical agency: management; statistical domain specialists; coordination units; data dissemination units; and IT specialists. The current version of the Starter Kit is still very much work in progress and feedback and suggested improvements from other international organisations and national statistical agencies are welcomed. Some of the information on current and proposed SDMX implementations at both the global and national levels will be relocated onto the web where they can be updated more readily as new information becomes available. 2 CONTENTS A. OBJECTIVES OF STARTER KIT ................................................................................................. 5 B. BUSINESS CASE FOR IMPLEMENTING SDMX STANDARDS AND GUIDELINES ........... 6 1. The challenge ...................................................................................................................................... 7 2. Higher level rationale for utilising standards such as SDMX and DDI .............................................. 7 3. Specific cases for realignment of business objectives and processes: Analysis of existing national strategic plans.......................................................................................................................................... 8 4. Objectives of existing SDMX implementations ............................................................................... 10 C. STRUCTURED PROCESS FOR SDMX IMPLEMENTATION BY NATIONAL AGENCIES 11 STEP 1: ACQUIRE UNDERSTANDING OF KEY SDMX ARTIFACTS ......................................... 11 1. SDMX STANDARDS AND GUIDELINES ............................................................................ 12 2. DATA STRUCTURE DEFINITIONS AND METADATA STRUCTURE DEFINITIONS .. 14 a. Data Structure Definitions .................................................................................................... 14 b. Metadata Structure Definitions ............................................................................................. 18 i. Eurostat’s ESS Metadata Handler (ESS MH) ....................................................................... 19 ii. IMF’s Data Quality Assessment Framework (DQAF) / African Development Bank (AfDB) Open Data Platform (ODP) ........................................................................................................... 19 iii. Millennium Development Goal (MDG) metadata ............................................................ 20 3. SDMX IMPLEMENTATION TOOLS .................................................................................... 20 a. Eurostat’s SDMX Reference Infrastructure (SDMX-RI) ..................................................... 21 UNdata API ................................................................................................................................... 22 b. Eurostat’s SDMX Converter ................................................................................................. 22 c. IMF’s SDDS Plus ................................................................................................................. 22 IMF – African Development Bank (AfDB) Open Data Platform (ODP) ...................................... 23 d. DevInfo ................................................................................................................................. 25 UNSD’s use of DevInfo for the MDG and CountryData projects................................................. 25 ILO’s use of DevInfo ..................................................................................................................... 26 e. The SDMX Registry (local implementation or cloud solution) ............................................ 26 f. Concluding remarks on existing SDMX implementation tools ............................................ 26 g. Downloading and use of selected SDMX implementation tool ............................................ 27 STEP 2: ISSUES TO BE CONSIDERED BY COUNTRIES CONSIDER PRIOR TO EMBARKING ON SDMX IMPLEMENTATION........................................................................................................ 27 3 1. Institutional issues and objectives ............................................................................................. 28 2. IT-related issues ........................................................................................................................ 29 3. Statistical issues ........................................................................................................................ 29 4. Resource-related issues ............................................................................................................. 30 STEP 3: IDENTIFICATION OF SDMX IMPLEMENTATION BARRIERS AND CAPACITY DEVELOPMENT NEEDS ................................................................................................................... 30 STEP 4: DETERMINING WHICH SET OF SDMX IMPLEMENTATION TOOLS TO USE .......... 31 STEP 5: PARTICIPATION IN SDMX NETWORKS ......................................................................... 31 ANNEXES ............................................................................................................................................ 38 Annex 1: EURO-SDMX Metadata Structure Concepts .................................................................... 39 Annex 2: IMF Data Quality Assessment Framework Metadata Items ............................................. 41 Annex 3: NSDS Status of Countries in the Asia and Pacific Region ................................................... 43 4 A. OBJECTIVES OF STARTER KIT The Starter Kit outlined in the current document is a resource for