Creating dashboards and data stories within the Data & Analytics Framework (DAF) Michele Petitoc, Francesca Fallucchia,b and De Luca Ernesto Williama,b a DIII, Guglielmo Marconi University, Via Plinio 44, 00193 Roma RM, Italy E-mail: [email protected], [email protected] b DIFI, Georg Eckert Institute Braunschweig, Celler Str. 3, 38114 Braunschweig, German E-mail: [email protected], [email protected] c DIFI, Università̀ di Pisa, Lungarno Antonio Pacinotti 43, 56126, Pisa PI, Italy E-mail: [email protected] Abstract. In recent years, many data visualization tools have appeared on the market that can potentially guarantee citizens and users of the Public Administration (PA) the ability to create dashboards and data stories with just a few clicks, using open and unopened data from the PA. The Data Analytics Framework (DAF), a project of the Italian government launched at the end of 2017 and currently being tested, integrates data based on the semantic web, data analysis tools and open source business intelli- gence products that promise to solve the problems that prevented the PA to exploit its enormous data potential. The DAF favors the spread of linked open data (LOD) thanks to the integration of OntoPiA, a network of controlled ontologies and vocabularies that allows us to describe the concepts we find in datasets, such as "sex", "organization", "people", "addresses", "points of inter- est", "events" etc. This paper contributes to the enhancement of the project by introducing the process of creating a dashboard in the DAF in 5 steps, starting from the dataset search on the data portal, to the creation phase of the real dashboard through Superset and the related data story. The case study created by the author, concerns tourism of Sardinia (a region of Italy). This case study is one of the few demonstrations of use on a real case of DAF and highlights the ability of the DAF to transform the analysis of a large amount of data into simple visual representations with clear and effective language. Keywords: Data & Analytics Framework, Data Visualization, Dashboard, Ontology, Controlled vocabulary, Open Data. 1. Introduction (AgID) [1] at the end of 2012 through the national guidelines [2] that paved the way to the use of LOD as Open and Closed Public Administration (PA) Data, the data paradigm for enabling semantic interoperabil- are one of the major digital useful assets for interpret- ity in the collaboration between PAs.. ing social and economic phenomena, making deci- In this scenario, the big data platform DAF [3] de- sions, for improving services to citizens and compet- signed by the Digital Transformation Team [4], a in- ing on the international scene. To better exploit their frastructure of the Italian Government established in potential, it is necessary to leave the approach based September 2016, represents Italy's latest effort to val- on silos in favor of a systemic vision: a platform that orize public information assets. The objective of the supports data sharing and access. This is especially DAF is to overcome these difficulties by channeling true of open data, whose spread has slowed in recent data flows from the various local and central PAs into years because large number of databases and open a large container (data lake) where, thanks to a frame- data portals are only available as silos in the various work for distributed applications such as Apache Ha- PAs, so non-interoperable between them. Furthermore, doop [5], data can be explored and analyzed, also us- very few PAs have adopted standardized production, ing machine learning techniques, to fully exploit in- analysis or data storage, according to the methodolog- sights and information hidden in it. Thanks to a very ical approach defined by the Agency for Digital Italy simple interface and an API (Application Program Interface) system, the DAF allows individual admin- speed, and show the DAF project which integrates two istrations to communicate and share data, analysis and business intelligence tools for data representation Su- dashboards using open standards and open source perset [9] and Metabase [10]. There are many other technologies. Furthermore, the DAF promotes seman- platforms in the same category such as Microsoft tic interoperability, according to the new European In- Power BI [11], Tableau [12], Google Data Studio [13] teroperability Framework (EIF) [6]. To enhance in- e Ploty.ly [14] that offer the possibility of use in the teroperability DAF make use of an ecosystem of on- cloud via API and that could allow integration with the tologies and controlled vocabularies (OntoPiA) [7], a DAF, even if with important limitations. set of ingestion pipelines that performs standardiza- Public Tableau allows the creation of complex tion to incoming data, and a rich set of metadata. Every dashboards with great flexibility, do not require spe- dataset in DAF is accompanied by metadata that de- cific technical skills nevertheless an application needs scribes the dataset and its internal structure. It will be to be installed in locale. the user's responsibility to define the ontological infor- Google Data Studio is very intuitive but does not mation and controlled vocabularies associated with allow the use of more than one dataset within the same the data structure, through the meaning of the semantic dashboard. It has few data connectors (only for MySql tags. A tagging system will allow to drive the user to [15], and for PostgreSQL [16]) and the quality of the the correct use of controlled vocabularies and to en- dashboards is not comparable to that of the competi- sure that all datasets can be effectively connected to- tors in the sector. gether. Plot.ly allows the creation and sharing of quick in- In this paper, we focus on DAF data visualization teractive dashboards. But it only accepts datasets with tools, illustrating the weaknesses of some of the most a maximum size of 5 MB and the published graphics famous data visualization tools on the market and the must be public (for them to be private it is necessary advantages of the integration in the DAF of two open to pay a subscription). Among the open source catego- source software (Metabase and Superset, see Section ries we have instead distinguished three data visuali- 2). In Section 3, we present the general architecture of zation tools: Superset [9], Metabase [10], and Redash the DAF, its main functionalities and the semantic [17]. technologies used, with particular reference to Onto- All these projects meet the requirements [18] that PiA, the network of controlled ontologies and vocab- an OGD2 visualization tool should possess. Superset ularies. Section 4 focuses on the process of building a and Redash are very similar. Both are powerful and dashboard through Apache Superset. Finally, a case of give the possibility to connect to a large number of use of the DAF will be presented for the construction data sources. In addition, they have a powerful inter- of a dashboard starting from a dataset on tourism in face for writing and executing SQL queries. the Sardinia Region (see Section 5). We conclude the Once saved, queries can also be used as a basis for paper with some future developments (see Section 6). the creation of dashboards. Superset supports a larger number of authentication systems than Redash and 2. Related works Metabase. For example, it includes LDAP3 , the sys- tem used for the unique authentication of all modules William S. Cleveland said “The human brain is in the DAF. Although Redash is an excellent project more able to identify and comprehend relationships in rapid evolution, Superset [9] has been chosen for and patterns if data is encoded into visual forms. three main reasons: the presence of LDAP authentica- Graphs allow us to explore data and observe patterns tion, the high number of views, and the support of the that no other approach can achieve.” [8] Python language (used throughout the DAF project). According to this, we believe that the identification In addition, Superset was preferred to Metabase, alt- of information needs is useful for the definition of data hough Metabase is more intuitive, because its graphs driven policy and the realization of interface for ana- are less parametrizable and offers much less types of lytical tools. visual representations. Therefore, we consider the importance of data vis- ualization1 and the goal of the PA to obtain useful trends and information with maximum simplicity and 1 The science of visual representation of ‘data’, which has been 2 Open Government Data. http://www.oecd.org/gov/digital-gov- abstracted in some schematic form, including attributes or variables ernment/open-government-data.htm for the units of information. 3 Lightweight Directory Access Protocol. http://foldoc.org/ldap Superset is an open source product hosted on the national catalog data. Instead, in the private section, Apache Foundation GitHub4 platform and is devel- the acces is done through a login, thus allowing only oped using Flask [19] a very lean Python framework accredited users to exploit the functionality of query- [20] for web development. The part that generates the ing, analyzing and sharing data. interactive graphs instead makes use of NVD3 [21] a javascript library built on D3.js [22]. Any dashboard created with Superset consists of a series of graphs (called slices in the current version of Superset installed in the DAF). Each of these can be resized, moved relative to the others, or shown in full screen. In addition, each dataset represented in a graph can also be exported in CSV or JSON format or through SQL queries. The "slices" are created starting from a table available in the many data sources that Superset is able to manage. Superset provides two main inter- faces: the first is the Rich SQL IDE (Interactive De- velopment Environment) called Sql Lab5 with which Fig.