Analysis and Design of the Test Client

ESSNET on SDMX II

WP2: PC-Axis SDMX Integration

Tutorial of the process using PX-file to SDMX for CensusHub Version 0.1

2012-06-15 Type of Document Project deliverable WP 2; 2d Presentation and information for other countries Reference: Issue: Revision: Status: Draft 2 Created by: Lars Nordbäck Date: 2012-06-15 Updated by: Approved by:

Document Change Record

Issue/ Date Change Revision 2012-05-26 2012-06-15 Updated figures

Analysis and Design of the Test Client v 0.1 Date Updated: 6/3/2018 6:00:24 PM Page ii of 29

Page ii Prepared by: TC, IB Reviewed by: [Title]

Table of contents Page

1 Introduction...... 4 1.1 Purpose...... 4 1.2 Reference Documents and Standards...... 4 2 Description of process for CensusHub reporting from PC-Axis files...... 6 2.1 Workflow...... 6 2.2 Interaction with SDMX_RI modules...... 7 3 Creation of the example...... 8 4 Preparing the input for the Census HyperCube 6...... 9 5 Preparing to handle the output of HyperCube 6...... 14 6 Using Census HyperCube 6 in the PC-Axis file based process for Eurostat CensusHub. 16

List of figures Page

Figure 2-1: Process for CensusHub reporting from PC-Axis files...... 6

List of tables Page

Table 1-1: Terms and Abbreviations...... 5

Analysis and Design of the Test Client Version 0.1 Date Updated: 6/3/2018 6:00:24 PM Page 3 of 29

Page iii Prepared by: IB, TC, SL Reviewed by: SLI 1 Introduction

1.1 Purpose In this paragraph will be described the purpose of the document and how the document is structured

The purpose with this document is to give the participants in the Essnet for SDMX Workshop in Rome 25-27 a documentation of the processes involved when using PC-Axis files as the base for the use of Eurostat Reference Architecture in the CensusHub project.

1.2 Reference Documents and Standards Insert all references to documents or convention or standard used in the document. This part can be divided in:

Standards and Conventions

Documents or standard named in the document

Statistics Sweden documents

PC-Axis file format document: http://www.scb.se/Pages/List____314011.aspx

Attributes in the PX-file; New keywords for attributes on observation values, Petros Likidis

PX-Web Systems Diagram, Mikael Nordberg

PX-Web powerpoint presentation, Mikael Nordberg

Eurostat RI documents

On CIRCA

SDMX - training material

SDMX Self-learning tutorials

Mapping Assistant

Installation Guide of the Mapping Assistant.doc

Mapping Assistant Design and Implementation Guide for Programmers.doc

Mapping Assistant User Manual and Tutorial.doc

Test Client

No instructions found

NSI Web Service

Installing_NSI_Web_Service_(.NET).doc

Web Client Analysis of Web application for exposing browsing_dissemination environment of an NSI v0.7.doc

Installation Guide of the Web Application for exposing_browsing dissemination env of NSI (.NET) v0.8.doc

User Manual of Web application for exposing browsing_dissemination environment of an NSI v0.4.doc

Terms and abbreviations

Terms or abbreviations used in the document

Acronym Definition SCB Statistics Sweden RI Eurostat Reference Infrastructure CIRCA Eurostat website for access to documents and software SSD Statistics Sweden Statistical Database on the Website PX-file PC-Axis file PX-Edit Software for creating PC-Axis file from different input file formats Nordic Data Data model used by the countries using PC-Axis SQL based software Model PC-Axis file Open file format based on keywords for metadata and data

Table 1-1: Terms and Abbreviations

Definitions:

Definitions used in the document

Analysis and Design of the Test Client v 0.1 Date Updated: 6/3/2018 6:00:24 PM Page 5 of 29

Prepared by: TC, IB Reviewed by: 2 Description of process for CensusHub reporting from PC-Axis files In this chapter will be described the example providing a description of the whole workflow. This chapter should be a sort of short “User Manual” in which is described how to use the example.

2.1 Workflow In this paragraph should be described the workflow of the example. In a picture should be shown the modules used of the SMDX-RI and the new modules created for the purpose of the example.

Statistics from a material made of the PC-Axis file with the program PX-Edit version 3.

1. Statistical production produces statistics including attributes on cell level to be included in the PC- Axis file database.

2. Subsequently, the statistical product load statistics in PX-Edit and enter metadata including Metadata for Attributes on data cell level.

3. Check of the entered data in the database using PC-Axis SQL or the SSD Web interface.

4. Based on Eurostat’s description of the statistics, Data Structure Definition (DSD), which is stored in the local database Mapping Store, a mapping of the SCB material, can be made against what is in the DSD for the codes, etc. using the program Mapping Assistant. The resulting mapping is then stored in the Mapping Store.

5. To make sure the material is correct you can use the Test Client that creates an SDMX file of the statistics.

6. Publishing is executed based on normal copying to the external PC-Axis directory

This routine is not using RSS for notifying because Census data is only posted once.

7a and 7b. Eurostat software downloads then the statistics that dynamically is transformed into SDMX format.

In this case the Statistical database is outside the inner Firewall and security of the database has to be monitored.

Figure 2-1: Process for CensusHub reporting from PC-Axis files

Prepared by: TC, IB Reviewed by: Process for CensusHub reporting from PC-Axis files

6. Publishing Copying to external External PC-Axis Eurostat PC-Axis Directory directory 91.194.86.67 2. Loading data and entering metadata Mapping tables in PC-Axis directory PC-Axis With DSD Loading Directory

2. 5. Sending querry PX-Edit SDMX Publishing Check of entered data converter/Test specialist Web hotel Client For 7a. Pull 3. NSI Check Requestor PC-Axis Main SDMX- Module or ML 7b. PX- Web SDM Data Retriever X-ML m.fl. 4. Mapping Producer Assistant Fil http://censusweb. scb.se/NSIservice.asmx Mapping of PX-table 1. to DSD Create file for Statistical Listing of Mapping Store loading of product Mapping Data Structure Dissemination production information for Definitions, data base database check codes mappings etc Other International Organisations

Firewall Firewall

2012-05-26

Explanation of the staining of components in the image above: Yellow: Applications and databases developed by Eurostat called Eurostat Reference Infrastructure, Open Source software. Green: Program developed by Statistics Finland, freeware. Gray: Existing standard applications and databases for the dissemination of statistics by Statistics Sweden. Pink: Programs that are used by the product to produce the file for loading the statistics in the Statistical Database. Brown Pink: Product production database.

2.2 Interaction with SDMX_RI modules In this paragraph must be indicated the modules that interact with the SDMX-RI

No software module is interacting with the SDMX-RI. The interface is the PC-Axis file.

Prepared by: TC, IB Reviewed by: 3 Creation of the example In this chapter will be described the step done in order to realize the example. The example can use one or more modules belonging to the SDMX-RI, some of this modules can be modified and new code can be added in order to realize the example.

The example is the CensusHub Hypercube number 6.

Prepared by: TC, IB Reviewed by: 4 Preparing the input for the Census HyperCube 6 In this chapter a description of the input must be provided

The input in the process is a txt file for Hypercube 6 in the CensusHub project. A subset follows below:

The Attributes on Cell level are found after the pipe character | in the end of the lines that contains such an attribute. In this case P,P;””,”Source: SCB” on the fifth data line and NULL|CNI,P on the ninth data line. On this fifth line there are also marked two textual attributes expressed with the extra pair of “-characters.

This file is put into the PX-Edit software to make it a PC-Axis file.

In PX-Edit you start the process by setting the SDMX saving parameter.

Prepared by: TC, IB Reviewed by: Analysis and Design of the Test Client v 0.1 Date Updated: 6/3/2018 6:00:24 PM Page 10 of 29

Prepared by: TC, IB Reviewed by: In PX-Edit the table is exposed as above where you can find the red corners in data line 5 and 9 corresponding to the cell attributes in the txt file above.

Beneath the Table Metadata parts concerning the attributes are found on the three bottom lines.

Prepared by: TC, IB Reviewed by: Above is the given standard names of the attribute-id.

In the screen above the standard attribute-id are changed into the relevant names.

Prepared by: TC, IB Reviewed by: Above the default attribute are set to X,P

On the following cell note metadata screen you also can find the attributes as they are picked up in PX-Edit from the txt-file.

Prepared by: TC, IB Reviewed by: The resulting PC-Axis file looks like this:

Notice the attribute keywords ATTRIBUTE-ID and the cell related keywords ATTRIBUTES including the cell position for the expressed attributes.

Prepared by: TC, IB Reviewed by: 5 Preparing to handle the output of HyperCube 6 In this chapter a description on how to handle the output must be provided

In PX-Web it is possible to see the tables including the Attributes on cell level, this development is ongoing.

See http://groups.google.com/group/pcaxis/topics?hl=en

Attributes on cell level in PX - Solution proposal.pdf

98K View Download Attributes in the PX file format.pdf

216K View Download

Prepared by: TC, IB Reviewed by: 6 Using Census HyperCube 6 in the PC-Axis file based process for Eurostat CensusHub This chapter should be a sort of short “User Manual” in which is described how to use the example.

- Text-file, Census Hyper Cube 6, see chapter 4 above.

- PX-Edit, see chapter 4 above.

- PC-Axis file, see chapter 4 above.

- PX-Web including Attributes on cell level for visual checking, see chapter 4 above.

- Mapping Assistant mapping a PC-Axis file, see below.

- Test Client, see below.

- Web Client, see below.

Prepared by: TC, IB Reviewed by: Mapping Assistant mapping a PX-file

Data Flow

Prepared by: TC, IB Reviewed by: DDB Connections

Prepared by: TC, IB Reviewed by: Data Set

Prepared by: TC, IB Reviewed by: Adding table above.

Above selected dimensions in the dataset

Prepared by: TC, IB Reviewed by: Preview of dataset

Mapping Set

Mapping of variables between the local dataset and the DSD from Eurostat, above.

Prepared by: TC, IB Reviewed by: Mapping set completed above.

Header Info

Prepared by: TC, IB Reviewed by: View Mapped data

Mapping set view data, above.

Test Client

Prepared by: TC, IB Reviewed by: Web Client

Web Client main window

Prepared by: TC, IB Reviewed by: Selections

View result as a table

Prepared by: TC, IB Reviewed by: View results download in HTML

Prepared by: TC, IB Reviewed by: