ESSnet Project

“Automated and reporting in accommodation ”

Report on Work Package 6: “Evaluation of NSIs’ information systems”

Introduction

The objective of Work Package 6 is to analyze the different ways of introducing the data collected by XML files in the structure already existing in each country. The deliverable of Work Package 6 is a report describing the way each country introduces the data collected via XML in the NSI database.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 2 1. Information from Spain

Paper Q, MANUAL fax RECORDING IN NSI

DATATRANSFER TEMPORARY DATABASE e-Surveys SOFTWARE IN NSI (ARCE) (GRECO DB)

Prices & personnel variables DATA EDITING XML

Oracle DB Rest of variables Text files DATA TRANSFER (SIRENET) DATA CONTROL AND VALIDATION

PRODUCTION DATABASE (GENTUR DB)

Once it is checked that the schema of the XML file is correct and the validation rules are met, this file is uploaded (either manually or through the Web Service) in the Oracle DB and is divided into two parts: the mandatory one and the optional one. If the schema is wrong or the validation rules are not achieved, data are not uploaded in the NSI database.  The mandatory part is automatically transformed to a txt format, so it can be easily processed in the production database. This plain text file is downloaded using a web application in the INE intranet.  The optional part is also automatically transformed to be uploaded in the temporary database (where these data are merged with data from e-, paper and fax ). Prices and personnel data from XML files included in this database are edited. In some cases these variables are not included in the XML file and are collected in this phase of the process. After the editing, another plain text file is transferred to the production database using another web application (SIRENET) in the INE intranet.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 3 2. Information from Finland

1. Introduction

Statistics Finland’s data collection system in accommodation statistics consists of three different data collection methods. About 70 % of the reports are delivered using the electronic data collection which is composed of the automated data collection and Internet-based . Other options are more traditional such as answering by faxing reports generated from hotel management systems or filling in paper questionnaires.

The following figure depicts the overall design of the Finnish architecture in data collection

Figure 1. The Finnish architecture in accommodation statistics’ data collection

The process in both electronic collection modes is more or less similar. If the respondent delivers data by using automated data collection, the received data are transferred within a short delay to the Internet questionnaire. Otherwise, the respondent must manually key the data into Internet questionnaire. After this stage, the process is similar between two electronic alternatives. Data is transferred from the Internet questionnaire to the temporary database after data control and logical validations. The team responsible of the accommodation statistics does these procedures.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 4 Figure 2 shows the more detailed architecture of handling XML files in automated data reporting. The diagram consists of the data integration process before data is transferred to the Internet questionnaire. There are two types of XML files that can be sent to Statistics Finland. One is based on XML format and another in XBRL format. Nevertheless, the process is similar between both approaches.

First software suppliers implement the XML file into their hotel management software. After this, respondents send the reports automatically in every month. The files are handled in Cinereus server using Internet Information Services and ASP.NET Webservices. Authentication is done in Maggie server.

After basic handling and authentication, the files are delivered to temporary database situated in SQL server. The last step is to use StatTransCopy and StatTransCopyXBRL software. This is the first stage in which the team responsible of the accommodation statistics steps into picture. All the earlier steps are done automatically. The files are sent manually in this step back into SQL server and stored into “XCola database” which, in practice, the common Internet questionnaire.

Figure 2. The handling of XML files in Statistics Finland

Report on Work Package 6: “Evaluation of NSIs’ information systems” 5

Respondents can check their reports once the data is transferred to the Internet questionnaire. After this the team responsible of the accommodation statistics transfer reports to the temporary database. Data is logically checked and verified in “Accommodation statistics’ software” by the team both automatically and manually. After these procedures data can be transferred to the production database. In the end of the whole process data is once more checked before the publication.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 6 2. Delivery of the XML file

Respondents can transmit the XML file to Statistics Finland direct from the management system of the accommodation establishment simply “by pressing the button”.

This function is meant to be implemented utilising the Web service interface. When submitting accommodation statistics data the following parameters are entered in the Web service (WSDL):

- User ID of the establishment (provided by Statistics Finland) - Password (provided by Statistics Finland) - Collection identifier (provided by Statistics Finland) - XML file (character string input, which follows the XML Schema)

Once the software supplier has implemented the feature to their hotel management systems, data can be sent almost automatically by just choosing the right year and month and pressing the send button.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 7 3. Information from Poland

Introduction

Since 2009 the collection of in Poland has been carried out through the Reporting Portal which is an integrated communication platform for electronic reporting.

Reporting Portal provides an electronic form and ensures to deliver a data file in XML format at the time specifeid by the dates from the survey calendar.

Currently, the data from the KT-1 survey are submitted to the statistical office in the following ways: 1. electronically

a. by filling in the form directly on the page https://raport.stat.gov.pl

b. by providing a file in XML format

2. in the traditional manner, i.e. by providing a paper form.

The basic method of submitting forms is filling them in on the Reporting Portal.

Forms submitted in paper form are recorded by Statisticians in the database Reporting Portal through the same electronic form which is used by the Reporting Units.

The interval of data collection in the case of Reporting Unit is defined by the variables [dataOd] and [dataDoFormJS], in the case of Statisticians who enter paper forms using the interface of the electronic form the end date is different (variable [dataDoStatystyk]). The different end dates for the form being active, results from expected longer period for entering data from paper forms.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 8 Diagram showing the paths for providing reports to the statistical office sd

refill form using paper

Offline Application print

Statistical Office Consultant send validate

Form Application Manager

Paper Survey

fill form validateAndSaveSurvey fill form

Survey Site

saveAs

fill form

Form Form Management Database Reporter Process

upload generate saveAs

validateAndSaveSurvey Web Service

Tourism Establishment generate XML Software is used for validateAndSaveSurvey generation XML either saveAs Common Survey in Tourism Specific Format XML Format (XSD definition created in this project) or Common Survey Format (currently used XSL XSD definition). Legend transform Objects/Flows created in this project Tourism Survey Specyfic Format

TourismSurvey XML

1. Use of the Reporting Portal for sending KT-1 questionnaires.

In order to log in the Reporting Portal one should enter in a web browser the following address: https://raport.stat.gov.pl

On the login page, one should enter authorization data: ID and password obtained from the Statistical Office (or generated by the person in charge of reporting in the company/institution).

All the data in the KT-1 report are mandatory and must be completed. Additional data are required in the reports for July – in the remaining months the fields for these data are not active.

While working with the form the correctness of information entered is being controlled. Control procedures are activated automatically when going through pages, as well as when the user clicks on the Zapisz strone (Save page) or Zatwierd ź (Approve) buttons.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 9 Errors identified within the form data can be either absolute or discretionary .

The absolute error prevents the form from being approved, whereas the discretionary error serves only information purposes only and does not block such possibility.

In the absence of absolute errors one can save the form in a database system.

After the form has been successfully submitted there is no need to take any additional steps. The Reporting Portal sends a confirmation e-mail of submitting the report.

2. Saving the form on a user's computer

Apart from saving the form data automatically in the database the CSO, the user of the Portal application has the ability to save the data on his/her computer and import them later (paste) into the form of the same structure. This means that after entering data and approving the form the user can save the data so that in the next edition of the survey (e.g. in a month) he can paste these data and modify only these values that have changed since the previous reporting period.

The data are saved in XML format and are fully readable to the user only when imported to the structure of the form.

In order to save a form on local disk, one should use the function Export to XML .

The import of the form of the previously saved XML file is implemented by the XML import function.

3.Use of websites for sending questionnaires

The portal provides an interface (Web Service) through which one can directly upload the report and verify errors.

Interfaces (Web Service) provided by the portal

• AktualnaDefinicjaFormularzaService (current definition of the form) • InformacjaOEdycjiService (information on edition) • PoleSlownikoweService (glossary field) • WyslanieFormularzaService (sending the form)

Report on Work Package 6: “Evaluation of NSIs’ information systems” 10 For submitting forms walidujIZapiszSprawozdanie operation is used: Operation Description walidujIZapiszSprawozdanie (pXmlSprawozdania: Saves the main report after a successful validation. The String, pRegon: String, pZastap: boolean): first step in the validation report is to check its WalidacjaError[] compatibility with xsd report. Then, user’s permissions to write the report are checked. If there is no formal errors (as to the format of the report) the report is saved in the database. In the next stage validations of a given report are verified (the discretionary errors are also returned). Reporting Unit is identified by the REGON number. The table with validation errors is being returned, or null in case of no errors.

Below is the full definition of the service (WSDL):

The service is available at: https://raport.stat.gov.pl/gus.blc.loader/WyslanieFormularza

The service is accessible to authorized users. Authorization is performed at the level of HTTPS protocol. It uses the same login and password that is used by a reporting unit to log into the reporting portal.

Currently, this method of submitting forms is used by offline applications. It is a method to automatically send reports (KT-1) from the systems of their own tourist facilities.

4. Delivering questionnaires with the use of off-line applications

Reporting units may use off-line form available on the CSO website http://form.stat.gov.pl/formularze/2011/index.htm

Off-line application allows one to complete a report on own computer, without direct connection to the Internet, and then to send it to the Reporting Portal.

After entering the data on the reporting unit’s computer, one should run the validation of the data entered in the form. The correct form, i.e. the one that does not contain absolute errors can be sent to the Reporting Portal, whereas the one which contain such errors cannot be sent.

In order to send data to the Reporting Portal, Reporting Unit must fill in the fields ID and password to the Reporting Portal to be authenticated to the portal (logins and passwords to connect the user with the Portal).

After proper authentication (correct ID and password), accepting the certificate and establishing secure connection with the portal: the data is loaded into the data on Reporting Portal.

In case there is no Internet connection, it is allowed to submit to the Statistical Office a generated in the application XML file on magnetic medium or printed from the offline survey in paper form. An appointed statistician upload the data to the CSO

Report on Work Package 6: “Evaluation of NSIs’ information systems” 11 resources while maintaining the required level of authentication (identification of reporting unit) and security, e.g. as to the antivirus protection. This backup procedure is also possible for those reporting units who have not activated an account on the portal.

5. Delivering questionnaires with the use of systems of own accommodation establishments

In the case when the systems for managing accommodation establishments will generate a report in XML format compatible with the format specified by this project (MONTHLY_PL.XSD - see the report on the Polish XML format of Work Package 4), before sending it to the portal one should convert this form to the XML form common to all forms, which is used as an argument of the operation walidujIZapiszSprawozdanie of the service.

6.Export of data from database of reporting portal to logical and accounting control systems

Data provided by electronic means are collected centrally in the database of the Reporting Portal.

After the forms are no longer accessible on the Reporting Portal, the data obtained using the electronic form or through the transfer of XML files are exported to the logical and accounting control systems, where the final verification of their quality is performed, which is usually broader than in electronic form.

The data are exported as a direct transfer between the databases of the Reporting Portal and logic control system.

The result of the verification of data quality in the logic control systems is the identification of errors which require contact with the Reporting Unit in some cases.

Then the database containing the correct data is again sent to the CSO server.

The last stage is the preparation of tables for the needs of different users.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 12 4. Information from Latvia

WEB SERVICE IN NSI E-Surveys TEMPORARY DATA DATABASE CONTROL AND (GRECO DB) VALIDATION

*E-Surveys *XML upload

DATA *XML upload VALIDATION, *Webservice EDITING

MANUAL *Paper INPUT IN NSI PRODUCTION Questionnaire DATABASE *Fax

Figure 1.1.The schema of CentralStatisticalBureauofLatviadatacollection/transfer (CSB)

All process is driven by MD-ISDMS(Metadata Driven Integrated Statistical Data Management System)system, named ISDAVS. Microsoft SQL servers are used for data storage.

The way of dealing with XML file

Data from a hotel management system to CSB will be sent using our XML file transfer web service. The process of this mechanism can be described in a following way - data from the hotel management system are transformed to suit the web service requirements; then in a second phase the data are transferred to server – temporary database where before the entering the database they are fully checked and validated and after that a response message is delivered to the data provider (hotel) about successful upload in NSI.

After the XML schema is validated and met all the requirements it is deposited in productions database.

XML web service upload is not currently used for accommodation statistics in CSB.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 13 5. Information from Bulgaria

Data transfers E-Questionnaires software in NSI (SQL Server)

Data editing and validation.

Manual Paper Recording in Questionnaires Regional

Statistical Offices

Production Database

Accommodation statistics.

The data flow • The data filled in the e-questionnaires by respondents goes to the National Statistical Institute (in SQL Server 2008 R2 EE) .

• The paper questionnaires filled in by respondents go to the Regional Statistical Offices. The data are checked and entered to the Personal Computer, and then summarize it in file The National Statistical Institute receives files by email from all RSO.

Data validation and editing of data are done after the merge with the data from e- questionnaires. Finally the Production of Database is done.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 14 6. Information from Belgium

Transformation of XBRL instances.

This process transforms the collected data to a relational database table managed by the ICT service on the mainframe system. This process will be a scheduled process running daily. It retrieves data from the complete XBRL instances database provided by the establishments, transforms and stores the data within a table for further usage by DGSIE’s processes. At the end of the delay given to the establishments, the XBRL instances included in the incomplete database should be also transformed. A quality flag indicates if the instance was validated or not. Data control and validations are performed on the relational database. This relational database will be transferred each night to the DWH (Datawarehouse). The DWH will be used for e.g. the feed-back report.

The tables will look like this:

TXBRLO_DOCUMENT TXBRLO_VALUES TXBRLO_VARIABLES CODE_ENQUETE FK_VARIABLE_ID VARIABLE_ID YEAR FK_DOCUMENT_NUMBER DATE_START NUM_SEQ TUPLE_SEQ_NR DATE_END PERIODICITY ELEMENT_VALUE CODE_ENQUETE NUMBER_BCE NAME_DOCUMENT NAME_DOCUMENT CODE_SBS TRANSFERRED_FLAG CODE_ICT QUALITY_FLAG ELEMENT_PATH FK_DOCUMENT_NUMBER FORMAT_VARIABLE

The process is shown in the following figure:

Report on Work Package 6: “Evaluation of NSIs’ information systems” 15 7. Information from Lithuania

The figure describes the process of data collection at Statistics Lithuania.

E-Surveys system (e. statistika )

XML (offline data entering) Online data entering/ uploading/ TEMPORARY reviewing/ editing/ DATABASE XML approving (generated from BMS)

DATA TRANSFER

Data production system

Paper Q, MANUAL DATA fax ENTERING PRODUCTION DATABASE (Oracle DB)

DATA EDITING

Currently respondents can submit data to Statistics Lithuania by filling in a paper questionnaire (sent by post or fax) or an electronic questionnaire.

The e-surveys system ( e. statistika ) can accept two types of XML files:

1. A respondent can download an electronic questionnaire from the e-surveys system, fill it in manually in his own computer and then upload it to the e- surveys system (offline data entry) or send it by email. Free software ABBYY eFormFiller is used for filling in electronic questionnaires. The structure (schema) of such an XML file is adapted for a specific statistical questionnaire. Also, there is possibility to fill in the questionnaire online – directly in the e-surveys system.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 16 2. A respondent can transmit the XML file to the e-surveys system directly from the Business Management System (BMS). The structure of such an XML file is common, i.e. adapted for all statistical questionnaires at Statistics Lithuania. This new statistical data submit is currently under implementation.

If the schema of the XML file is correct and the validation rules are met, this file is uploaded (manually or through Web Service) to the e-surveys system’s temporary database. The respondent can review, edit and confirm (submit) these uploaded data. Confirmed data are automatically transferred to the Oracle production database, where they are finally edited and prepared for publication by statisticians.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 17 8. Information from Slovakia

There are 2 ways of sending the monthly questionnaire from accommodation establishment:

Paper questionnaire – filled in manually (according to establishments own records). Establishments send the questionnaire (via post, via fax, via e-mail) to NSI, where it is manually typed into the computer, checked and corrected when needed (possible mistakes/problems discussed with establishments) and saved for the database.

On-line questionnaire – questionnaire is filled in online by the establishment (establishment gets a username and password and logs in to use a web application to fill in the questionnaire), followed by the check at NSI and saved for the database.

Data from both ways are merged and together they form a complete set of variables every month, from which the information for national and international purposes are being produced.

Possibilities of using the XML file:

• Upload the XML file to NSI´s system using identifying name and password

At the , there is a new statistical information system being developed by an external software company. They have to create a system including all statistical procedures (from collecting data to the dissemination). We have asked them to consider including receiving data in accommodation statistics in XML format and maybe include it in one of the new tools.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 18

Report on Work Package 6: “Evaluation of NSIs’ information systems” 19 Conclusion

All partners use mainly the same method. There is the possibility to upload the XML-file (automated data collection) or to fill in the questionnaire online (internet questionnaire). Most countries offer also the possibility to fill in a paper questionnaire or to send the data by fax (more traditional options). The data from the XML-files is stored in a temporary database, where data control and validations are performed. Eventually, the data is transferred to a production database.

Report on Work Package 6: “Evaluation of NSIs’ information systems” 20