ESSnet Project “Automated data collection and reporting in accommodation statistics” Report on Work Package 6: “Evaluation of NSIs’ information systems” Introduction The objective of Work Package 6 is to analyze the different ways of introducing the data collected by XML files in the structure already existing in each country. The deliverable of Work Package 6 is a report describing the way each country introduces the data collected via XML in the NSI database. Report on Work Package 6: “Evaluation of NSIs’ information systems” 2 1. Information from Spain Paper Q, MANUAL fax RECORDING IN NSI TEMPORARY DATATRANSFER SOFTWARE IN DATABASE e-Surveys NSI (ARCE) (GRECO DB) Prices & personnel variables DATA EDITING XML Oracle DB Rest of variables Text files DATA TRANSFER (SIRENET) DATA CONTROL AND VALIDATION PRODUCTION DATABASE (GENTUR DB) Once it is checked that the schema of the XML file is correct and the validation rules are met, this file is uploaded (either manually or through the Web Service) in the Oracle DB and is divided into two parts: the mandatory one and the optional one. If the schema is wrong or the validation rules are not achieved, data are not uploaded in the NSI database. The mandatory part is automatically transformed to a txt format, so it can be easily processed in the production database. This plain text file is downloaded using a web application in the INE intranet. The optional part is also automatically transformed to be uploaded in the temporary database (where these data are merged with data from e-survey, paper and fax questionnaires). Prices and personnel data from XML files included in this database are edited. In some cases these variables are not included in the XML file and are collected in this phase of the process. After the editing, another plain text file is transferred to the production database using another web application (SIRENET) in the INE intranet. Report on Work Package 6: “Evaluation of NSIs’ information systems” 3 2. Information from Finland 1. Introduction Statistics Finland’s data collection system in accommodation statistics consists of three different data collection methods. About 70 % of the reports are delivered using the electronic data collection which is composed of the automated data collection and Internet-based questionnaire. Other options are more traditional such as answering by faxing reports generated from hotel management systems or filling in paper questionnaires. The following figure depicts the overall design of the Finnish architecture in data collection Figure 1. The Finnish architecture in accommodation statistics’ data collection The process in both electronic collection modes is more or less similar. If the respondent delivers data by using automated data collection, the received data are transferred within a short delay to the Internet questionnaire. Otherwise, the respondent must manually key the data into Internet questionnaire. After this stage, the process is similar between two electronic alternatives. Data is transferred from the Internet questionnaire to the temporary database after data control and logical validations. The team responsible of the accommodation statistics does these procedures. Report on Work Package 6: “Evaluation of NSIs’ information systems” 4 Figure 2 shows the more detailed architecture of handling XML files in automated data reporting. The diagram consists of the data integration process before data is transferred to the Internet questionnaire. There are two types of XML files that can be sent to Statistics Finland. One is based on XML format and another in XBRL format. Nevertheless, the process is similar between both approaches. First software suppliers implement the XML file into their hotel management software. After this, respondents send the reports automatically in every month. The files are handled in Cinereus server using Internet Information Services and ASP.NET Webservices. Authentication is done in Maggie server. After basic handling and authentication, the files are delivered to temporary database situated in SQL server. The last step is to use StatTransCopy and StatTransCopyXBRL software. This is the first stage in which the team responsible of the accommodation statistics steps into picture. All the earlier steps are done automatically. The files are sent manually in this step back into SQL server and stored into “XCola database” which, in practice, means the common Internet questionnaire. Figure 2. The handling of XML files in Statistics Finland Report on Work Package 6: “Evaluation of NSIs’ information systems” 5 Respondents can check their reports once the data is transferred to the Internet questionnaire. After this the team responsible of the accommodation statistics transfer reports to the temporary database. Data is logically checked and verified in “Accommodation statistics’ software” by the team both automatically and manually. After these procedures data can be transferred to the production database. In the end of the whole process data is once more checked before the publication. Report on Work Package 6: “Evaluation of NSIs’ information systems” 6 2. Delivery of the XML file Respondents can transmit the XML file to Statistics Finland direct from the management system of the accommodation establishment simply “by pressing the button”. This function is meant to be implemented utilising the Web service interface. When submitting accommodation statistics data the following parameters are entered in the Web service (WSDL): - User ID of the establishment (provided by Statistics Finland) - Password (provided by Statistics Finland) - Collection identifier (provided by Statistics Finland) - XML file (character string input, which follows the XML Schema) Once the software supplier has implemented the feature to their hotel management systems, data can be sent almost automatically by just choosing the right year and month and pressing the send button. Report on Work Package 6: “Evaluation of NSIs’ information systems” 7 3. Information from Poland Introduction Since 2009 the collection of official statistics in Poland has been carried out through the Reporting Portal which is an integrated communication platform for electronic reporting. Reporting Portal provides an electronic form and ensures to deliver a data file in XML format at the time specifeid by the dates from the survey calendar. Currently, the data from the KT-1 survey are submitted to the statistical office in the following ways: 1. electronically a. by filling in the form directly on the page https://raport.stat.gov.pl b. by providing a file in XML format 2. in the traditional manner, i.e. by providing a paper form. The basic method of submitting forms is filling them in on the Reporting Portal. Forms submitted in paper form are recorded by Statisticians in the database Reporting Portal through the same electronic form which is used by the Reporting Units. The interval of data collection in the case of Reporting Unit is defined by the variables [dataOd] and [dataDoFormJS], in the case of Statisticians who enter paper forms using the interface of the electronic form the end date is different (variable [dataDoStatystyk]). The different end dates for the form being active, results from expected longer period for entering data from paper forms. Report on Work Package 6: “Evaluation of NSIs’ information systems” 8 Diagram showing the paths for providing reports to the statistical office sd Interaction refill form using paper Offline Application print Statistical Office Consultant send validate Form Application Manager Paper Survey fill form validateAndSaveSurvey fill form Survey Site saveAs fill form Form Form Management Database Reporter Process upload generate saveAs validateAndSaveSurvey Web Service Tourism Establishment generate XML Software is used for validateAndSaveSurvey generation XML either saveAs Common Survey in Tourism Specific Format XML Format (XSD definition created in this project) or Common Survey Format (currently used XSL XSD definition). Legend transform Objects/Flows created in this project Tourism Survey Specyfic Format TourismSurvey XML 1. Use of the Reporting Portal for sending KT-1 questionnaires. In order to log in the Reporting Portal one should enter in a web browser the following address: https://raport.stat.gov.pl On the login page, one should enter authorization data: ID and password obtained from the Statistical Office (or generated by the person in charge of reporting in the company/institution). All the data in the KT-1 report are mandatory and must be completed. Additional data are required in the reports for July – in the remaining months the fields for these data are not active. While working with the form the correctness of information entered is being controlled. Control procedures are activated automatically when going through pages, as well as when the user clicks on the Zapisz strone (Save page) or Zatwierd ź (Approve) buttons. Report on Work Package 6: “Evaluation of NSIs’ information systems” 9 Errors identified within the form data can be either absolute or discretionary . The absolute error prevents the form from being approved, whereas the discretionary error serves only information purposes only and does not block such possibility. In the absence of absolute errors one can save the form in a database system. After the form has been successfully submitted there is no need to take any additional steps. The Reporting Portal sends a confirmation e-mail of submitting the report. 2. Saving the form on a user's computer Apart from saving the form data automatically in the database the CSO, the user of the Portal application has the ability to save the data on his/her computer and import them later (paste) into the form of the same structure. This means that after entering data and approving the form the user can save the data so that in the next edition of the survey (e.g. in a month) he can paste these data and modify only these values that have changed since the previous reporting period. The data are saved in XML format and are fully readable to the user only when imported to the structure of the form.