21 th International PC-AXIS meeting Stockholm 30 May-1 June 2012
Agenda Wednesday, 30 May Welcome/opening by Cecilia Westström, Statistics Sweden. Practical matters from organizing committee. Introduction by Raitis Sedlenieks and Åsa Arrhén from Statistics Sweden. Short presentation: name, organization, country. Chosen country report PX-Web developments plan, Lena Gustafsson, Statistics Sweden General information, Cecilia Westström and Åsa Olsson, Statistics Sweden. Presentations from the participants on the experiences and the planned usage of PC-AXIS product family in their countries after the selection process. PX-Web2012 by Petros Likidis and Mikael Nordberg, Statistics Sweden. News in PX-Edit by Veli-Matti Jantunen, Statistics Finland. Cont. Country reports. 2(2)
Thursday, 31 May News on the Swiss Portal by Alain Nadeau, Statistics Switzerland. SDMX project by Lars Nordbäck, Statistics Sweden. Visualisation and Mapping in the Modern Web Browser by Fiachra O Donoghue, Central Statistics Office Ireland. Statistics eXplorer – Web Publishing and Storytelling with PC-Axis data files using Flash OR new HTML5 by Mikael Jern and Tobias Åström NComVA.
Statistikatlasen by Stefan Palmelius, Statistics Sweden. Latest news on PX-Map by Markku Koskela, Tieto Info regarding Fridays group discussions. 18.30 Evening activity and dinner: Meet at Museum.
Friday, 1 June Group discussions in four groups: Developer’s group I Developer’s group II User’s group I User’s group II Presentations from group discussions Summary of the meeting and conclusions, Proposal for next meeting.
Participants See annex 1
1 List of content 21 th International PC-AXIS meeting Stockholm 30 May-1 June 2012...... 1 Country reports...... 2 PX-Web 2012...... 5 PX-Edit 3.0 2012...... 6 Wishlist and problems...... 7 Minding the data gaps in data cubes...... 7 PC-Axis SDMX integration project...... 8 Mapping and Visualisation in a modern web browser...... 8 Statistics eXplorer, Geovisual Analytics...... 9 Statistical atlas...... 9 PX-Map...... 9 Annex 1 Participants...... 11
Welcome Director Cecilia Westström welcomed the PC-Axis group to the 21th International PC-Axis meeting.
Country reports
Albania Burkina Faso Croatia Denmark Estonia Greenland Finland Iceland Ireland Macedonia Norway Slovenia Sweden Switzerland Ukraine UNECE South Africa
There were no possibilities to present all country reports, so the group expressed wishes to learn more about some of the work described in the reports: Finland, Greenland, Ukraine, Albania, Ireland
1. Finland: 20 different database in pc-axis. A system based on saved queries is build, FinIndicators, with a new interface. Around 100
2 indicators updated from the databank. Sharing with Facebook and Twitter is possible, or you can save or show graphics in vector or like a bit map. A version where external customers of px-web can get dynamic graphs on their web site has been developed. Example: http://verti2.stat.fi/VERTI-jak/graph/viewpage.aspx? ifile=quicktables/Kuntaliitto/E_Vaesto/E06_k&lang=1&rind=1 Open data: px-files med HTML links to every file were listed and made accessible for everyone. Pc-Axis file format was however a showstopper for many. That is why the Cossi XML (PXML) model and format was “invented”. In 5 hours all files were converted (in PX-edit). http://pxweb2.stat.fi/database/StatFin/StatFin_rap_xml.csv SDMX has been considered but it is not in the plan. Some keywords needed in SDMX do not exist in PC-Axis and PX-edit does not support that (yet?).
2. Greenland: Regional population forecast is under development A graph is shown together with access to details and selections from a px table. PX-Web “StatBank to-go” (mobile version). At the planning stage. During the coming two month this possibility will be examined. A mock-up was presented. WPS is much cheaper than SAS (does the same costs 5%) The solution will be in HTML5 not an APP. It I expected to see an increase in the need and use in a 1-2 years.
3. Ukraine: PC-Axis I widely used at the Lviv office. Used for quick tables with indicators, some also presented with graphics. StatBank multi lingual px-files. Regional statistics, Regiostat, presented with maps (PX-iMap). In the case of the use of SQL database in the corporate network for selected statistical table quite simple and convenient looks its straight representation by PC-Axis tools in such a way that PC-Axis applications would get started automatically every time through direct link from the corresponding newly created px-file. In other words, px- file is generated on-the-fly for chosen from SQL database information and switch to the corresponding hyperlink is made, for example: PXWEB2008/Dialog/varval.asp? ma=
4. Albania: PX-web to be launched September 2012. The metadata database is implemented. A new website is build and it will contain PX- files instead of only excel. Data loading to the sql database was a challenge and a tool from UNECE was used, but unsuccessful. Now on the basis of 9 sheet Excel templates a successful loading takes place. One has to be aware of the risk of duplicating metadata. However, a check is being done on this.
5. Ireland: Five projects have been important last year: Census data to the StatBank, Key indicator application, Coordination with Google to
3 create visualization via Google Data explorer. Hosting databases for other organizations and the, “Link Data project”: Wikipedia describes Linked data : linked data describes a method of publishing structured data so that it can be interlinked and become more useful. It builds upon standard Web technologies such as HTTP and URIs, but rather than using them to serve web pages for human readers, it extends them to share information in a way that can be read automatically by computers. This enables data from different sources to be connected and queried. The uniform resource identifier (URI) is a string of characters used to identify a name or a resource. Such identification enables interaction with representations of the resource over a network (typically the World Wide Web) using specific protocols. Schemes specifying a concrete syntax and associated protocols define each URI. In a pilotproject data from Census 2006 was put out on the web and linked to data not yet published. This means re-use of data, free of charge, data linked to the metadata and, transparency. More than 200 data catalogs are included. Semantics is important in linking: (Tim Berners-Lee 5 star model) 1) Make data available 2) Make structured data available 3) Use non-priority format (csv , not excel) 4) Use linked data format (uri -be the first to do it) 5) Link to others
Progress on PX-Web Released 2011 version in April, now a beta 2012 is out. Statistical Atlas with px-files to eXplorer The Google Forum was started.
PX-Web developments plan Official version 2011 is still basic (no search, no saved queries, no graph). It is possible to connect with a CNMM database Version 2012 is a beta version and released in May. Includes Graphs, it is security tested. Expected final in the autumn 2012.
2013 plans Saved queries and search are planned for Maps not decided. Additional functionality not decided. The product plan is on the website www.scb.se/pc-axis
The group is asked to bring up any activities for the wish list.
General information: plans and challenges PC-Axis Forum since January at http://www.scb.se/Pages/List____314030.aspx Register and use! Sweden: New price policy is planned: Discussions on pricing of the PC-Axis are going on in Statistics Sweden. The rule is that it should cover costs – but no profit. The former considerations of making a consortium with a steering board are no longer valid. Instead Statistics Sweden keeps the full responsibility for Pc- Axis /Px-Web, supported by the Nordic countries that will meet in connection with the yearly PC-Axis meetings to discuss the vision of the PX-family, prioritize the changes and plans the development, and look at incoming wishes and needs.
4 PX-Web 2012
Table of Contents Web controls in the Menu Tree is loaded dynamically. StartNode property ExpandNode property UrlLink property (display URL links as tree nodes or images)
Aggregation and valuesets: The way content variable is displayed is changed and the content variable will be shown as the first. If only one it can be hidden. ValueSetMustBeSelectedFirst is a new property.
Cropped table MaxRows and MaxColumns
New keyword: OFFICIAL-STATISTICS (displays in the information web control)
Support for logging user behavior: PxAction
W3c compliance for webcontrols Percent operation optimized Prn file renamed csv In PXweb support for multiple cNMM databases General handling of errors, improved handling of settings
Security tests and bugs carried out, now security is very good (ASP.NET gives a lot of “free” security.) Potentially vulnerable code in PCAxis.sql will be fixed by Statistics Norway.
Search Values and select them- or show all values.
2012 version Views could be considered as different perspectives of data: tables, graphs, footnotes etc. Displaying as a table is not necessary as the first display. Go directly to a graph for instance. The same kind of manipulations can be done from the graph as from the table view. By opening a tool box editing of text, titles, etc. can be in effect. The charts are also multi lingual. A demo of PX-Web 2012 beta version was given.
The new administration interface: Features can be enabled or disabled. This is also where settings for the chart are edited. Saved queries will be the same (later on). Defaults are set in the Selection Settings Command bar.
Work is going on with attributes at cell level (necessary for SDMX to Eurostat) (Should there be “layers” of attributes?)The attributes should be visible also in print- out . API:s New version of the Nordic Data Model 2:3 (before summer 2012)
The longer Customization manual run/Next: New selection page (you see only one variable at a time – or even both ways?) Input from this meeting and from the mail Google group and Forum.
5 PX-Edit 3.0 2012
New Features Quicker file read and save (20-30%) Edit/convert to XML, used in open data distribution Enhanced eXist database interface for table standardization Enhanced structured file format at cell level information, (notes, comments attributes) Wild character search (*?) Select and Add tables Better support for hierarchical key words File names with national characters be converted to ASCII Default used language selection list sett Russian and Italian are new interface languages. Because the Unicode support PX-Edit may be translated to practically every language. Google translate could be a start to edit from. Finland provides that to the ones that wishes. Bug fixes have been done
Still waiting PX-Edit Batch (command line interface ) Multilingual eXplorer support Settings/Lowercase filenames New controls in Join Window Works under Linux Px-Tool needs upgrade
Future New development depends on feed back from this group. Things to consider: ODBC interface vs/agg support Xdf input Full support hierarchical keywords Variable joining and splitting User levels Administrator interface DotNet integration Portable px-edit. Which features are needed?
Wishlist and problems
Some countries had suggestions for the future development: Adapt Px-Web to mobile devices (UNECE) Cell notes shown as tips or popups when mouse-over (UNECE) The strategy for the Nordic data model?? (UNECE)
Classic PX-Web software should be freeware (Greenland) Statistics-to-go (mobile) focus on HTML5 and JavaScript (Greenland)
Editing tables in hierarchies give problems “loosing quotation marks” (Burkino Faso)
Coordination with Google to create visualization directly from the Nordic data model in Google Data Explorer (Ireland) SDMX Integration a great wish (Ireland)
6 Enable the automatic flow of data from SAS to PC-Axis through PX-Edit. (South Africa) The GIS unit has accomplished this type of integration between SAS and and ARCGIS by use of web services (refer to diagram below) and hopes to achieve the same level of integration with PX Web (South Africa)
Px-Edit: User defined parameters for validation-standardization in PX-Edit (Slovenia) New PX-Tool with deeper metadata management options (Slovenia)
Minding the data gaps in data cubes
(Alain Nadeau) Output from the data warehouse used to be HTML tables manually made, excel files and px-web produced from the excel files. Quality not in top and errors might occur because of manual handling. So a new system is set up with PX-Generator. Data from the data warehouse and metadata comes in via SDMX and the generator creates templates where the selected dimensions are combined with data. From PX- web an API CubeCore produces dynamic HTML and RDF. And it is ready for open linked data production.
CubeCore uses Semantic/natural language processing and maintains/updates the tables on the web. Access data through a query in YCQL (query language). The data flow goes like this: Source (PX, CSV, …) Data management/Views (procedures) Publication/representation (JSON, CSV) The CubeCore works as a data warehouse as well as a publication service making data visualization, embed tables on other web sites and ready for openData . Yoocos made a demo of all the steps. (demo on cubecore.yoocos.com)
PC-Axis SDMX integration project
(Lars Nordbäck & Ulf Sandberg) The objective is reporting to international organizations, and to set up routines in accordance with Eurostat Reference Infrastructure (RI). The project has in 2012 been restricted to cover only the Census Hub project as no other domains are defined with DSDs. There will be no encryption facility. That will only be in Eurostat RI. In the project manuals for users and administrators will be made. Links to visualization tools based on generic SDMX from PC-Axis. From pc-Axis you make save in SDMX structure file and SDMX data file. This can be imported to visualization tools Data Express has been tested OK, eXplorer is tested and showed that changes has to be done. There will be a work shop on PC-AXIS SDMX integration with hands on training in Rome end of June.
PC-Axis format needs to have implemented attributes on cell level needed for SDMX.
7 Re-usability and links to SDMX by Nordic data model (MS-SQL, Oracle, Sybase) and PC-Axis files. The software will be accessible on the PC-Axis web site www.scb.se/pc-axis PX-Edit has to be tested regarding the attributes at cell level, in Px-web it will be developed in the dotNet version. Some adjustments of publishing software for CensusHub data are needed. CensusHub data will reside in a separate server for Eurostat access. A suggestion is made to Eurostat for an admission restriction to the RI tools, so access is only to own data.
A demo showing the different tools on screen dumps that one needs to work on throughout all processes in the transformation was given.
Mapping and Visualisation in a modern web browser
(Fiachra O Donoghue, Ireland) Modern browsers are supporting SVG support, fast javaScript, Filesystem API’s Good browsers: anything but Explorer 8 Polymaps www.polymaps.org based on geoJSON . It is open source. But the documentation is not too good. There are different datasets for each zoom level. Data can come from a database or PC-axis files. It is downloaded to the browser. It can be connected with other data from outside. Calculations can be made on the fly. TileStache can be used to make new maps.
Statistics eXplorer, Geovisual Analytics
(Mikael Jern, NComVA) Working with a lot of customers like OECD, Worldbank, National statistical offices and local administrations. In OECD their database is directly linked to the eXplorer maps. It is all about: Information visualization, geographic, cognitive perceptual science, time animation, storytelling.
It uses GAV Flash/HTML5with .JavaScript framework and tool box. Input API, Excel, csv, PC-Axis, SDMX. The “system” covers what (graphics), when (time series), where (map), why (storytelling wizlet). Some features and possibilities are listed below. Dynamic queries using indicator filters is a possibility: ex show the ten regions with most elderly people on the map. Fish eye technique: zooming on a graph to look closer to some parts in a bar chart. Parallel Cordinates with applied Filter mechanism Scatter plot to explore multivariate data Multi layers maps Flow maps to show examples are: trade (import/export), commuting, migration
Version for mobile devices is completed. A demo with pc-axis file as input was given. Long titles in PX-file is a problem when displayed in the eXplorer- in particular when combined with other data SDMX-ML, xml based format can be read by eXplorer. Is now integrated with OECDstat.
8 Most commonly used format is Excel, PC-axis is used by a certain group of people while SDMX is only used by technical people. If someone is interested contact the www.ncomva.se website. There are desk top licenses and server licenses. A server license is about 5000 Euro, but it depends on the size of the organization.
Statistical atlas
(Stefan Palmelius, Sweden) The atlas covers seven subject areas. The atlas with counties was published March 2012 The plan is to update data three times a year. A demo showing importing of data from Excel and from PC-Axis files were given. Users can make their own local story and share it with others.
PX-Map
(Markku Koskela, Tieto) Verti is a graphic tool developed by Tieto. It handles even very large tables and calculations and presents the results in graphics. Uses PC-Axis files. Statistics Finland has used this for the FinDicator. (www.findicator.fi) A new project is to build a map into Verti. PX-Map2 uses the same technology and they got the source code. The plan is to build 2 versions: a basic version for end users with same functionality as PXMap2. An advanced version is meant as a tool for administrators. The agreement is ready between Tieto and Statistics Norway, but not signed yet. The basic version will have bug fixed but not further developed. It can be used by the reference group free of charge.
Work shops 1. Developer’s group (Petros) 2. User’s group (Tatiana) Connection SDMX, PC-Axis and open linked data Integration PX-Edit and PC-Axis PC-Axis files as input to SQL database Prioritizing of wish list Google forum /PC-Axis forum Problems? The wish list 2012 should be prioritized
Reports from the User’s group two groups SDMX from PC-Axis will be possible in the near future, but the data structure definitions are crucial and needed. The international organizations have to produce these. New key words for attributes in Pc-Axis and PX-Edit.. Integration PX-edit and PC-axis would be useful with a basic user PX-Edit. “Someone” has to come up with requirements to such development. PC-Axis input to sql database. The group discussed on using a light version/ a core of the metadata model, where additional modules could be used or not used. Discussion forum has been improved by creating different roles so bug reporters can see each other’s bug reports.
9 A moderator of the Forum should handle discussions
Also see annex 3 for notes from this group.
Developer’s group Open data, discussions on using semantic web in the data model. A description will be made for further discussion. PC-Axis and PX-edit integration. Finland asked for ideas for improving PX- edit functionality. PC-Axis files to the Nordic data model. It might be a waste of time as data are structured differently. Everything should be discussed in the forum. Security issues can be handled by sending mails.
The list with reported wishes for development was discussed and the result of prioritizing is listed in annex 2
10 11 Annex 1 Participants
12 Frenki Ktona INSTAT Albania [email protected] Shuka Klajd INSTAT Albania [email protected] Abdou Ouattara Institut National de la Statistique et de la Démographie Burkina Faso [email protected] Gordana Hocurscak Central Bureau of Statistics Croatia [email protected] Dinka Popovic Central Bureau of Statistics Croatia [email protected] Dejan Bozic Central Bureau of Statistics Croatia [email protected] Lars Knudsen Danmarks Statistik Denmark [email protected] Annegrete Wulff Danmarks Statistik Denmark [email protected] Alan Ingel Statistics Estonia Estonia [email protected] Harri Lehtinen Statistics Finland Finland [email protected] Hans Baumgartner Statistics Finland Finland [email protected] Veli-Matti Jantunen Statistics Finland Finland [email protected] Huuhko Kim Statistics Finland Finland [email protected] Markku Koskela TietoEnator Oyj/Government Services Finland Finland [email protected] Lars Pedersen Grønland Statistik Greenland [email protected] Thorbjorg Magnusdottir Statistics Iceland Iceland [email protected] Björgvin Sigurðsson Statistics Iceland Iceland [email protected] Healy Kevin Central Statistics Office Ireland [email protected] Eoin MacCuirc Central Statistics Office Ireland [email protected] O'Donoghue Fiachra Central Statistics Office Ireland [email protected] Macedonia Damir Puh State Statistical Office of the Republic of Macedonia FYR [email protected] Anselmo Nhane INE - Mozambique Mozambique [email protected] Paulo Matusse INE - Mozambique Mozambique [email protected] Marianne Vik Dysterud Statistisk sentralbyrå Norway [email protected] Per Inge Vaaje Statistisk sentralbyrå Norway [email protected] Boštjan Kukovec Statistical Office of the Republic of Slovenia Slovenia [email protected] Raitis Sedlenieks Statistics Sweden Sweden [email protected] Åsa Arrhén Statistics Sweden Sweden [email protected] Lena Gustafsson Statistics Sweden Sweden [email protected] Petros Likidis Statistics Sweden Sweden [email protected] Mikael Nordberg Statistics Sweden Sweden [email protected] Birgitta Edberg Statistics Sweden Sweden [email protected] Lars Nordbäck Statistics Sweden Sweden [email protected] Tim Forsström Statistics Sweden Sweden [email protected] Cecilia Westström Statistics Sweden Sweden [email protected] Åsa Olsson Statistics Sweden Sweden [email protected] Kersti Leary Statistics Sweden Sweden [email protected] Amanda Werngren Statistics Sweden Sweden [email protected] Doris Carlström Statistics Sweden Sweden [email protected] Ulf Sandberg Statistics Sweden Sweden [email protected] Paula Ljungberg Statistics Sweden Sweden [email protected] Stefan Palmelius Statistics Sweden Sweden [email protected] Alain Nadeau Swiss Federal Statistical Office Switzerland [email protected] Jigitekov Marlen UN/ECE, Statistical Division Switzerland [email protected] Maxime Winkler Yoocos Switzerland [email protected] David Roon Yoocos Switzerland [email protected]
13