A Consumer Focused Open Data Platform
Total Page:16
File Type:pdf, Size:1020Kb
A Consumer Focused Open Data Platform Cherlton Millette and Patrick Hosein Department of Computer Science The University of the West Indies St. Augustine, Trinidad [email protected], [email protected] Abstract—Open data has been around for many years but data consumer and seeks to leverage the data already present with the advancement of technology and its steady adoption by in existing data repositories and open data platforms. businesses and governments it promises to create new opportu- Data-TAP will be the first consumer oriented open data nities for the advancement of society as a whole. Many popular open data platforms have focused on solving the problems faced platform that will (1) Leverage the data resources of existing by the data producer and offers limited support for the data Open Data platforms, (2) Provide a means to standardize consumers in the way of tools and supported processes for easy and structure data, (3) Use a rule-based method for parsing access, modeling and mining of data within the various platforms. resources by providing an easy to understand interface for In this paper we present the specifications for a data consumer defining parsing rules, (4) Allow for the merging of data oriented platform for open data, the Data-TAP. We demonstrate how this platform is able to (a) meet the goal of leveraging the resources from differing data repositories, (5) Provide raw data resources of existing Open Data platforms, (b) provide a API access to data stored on the platform in JSON and CSV means to standardize and structure data to allow for the merging formats in alignment with the standards defined for data-sets. of related data resources and (c) use a rule-based method for parsing resources. This paper will show how Data-TAP provides II. REQUIREMENTS OF AN OPEN DATA PLATFORM an easy to use and understand interface for making open data friendlier for consumers. To best serve the needs of the public, open data platforms Index Terms—open data, data transformation, data aggrega- are required to be highly available services that provide tion, data analytics, data platform reusable data which is universally available and consumable. In addition, the data published to such platforms must ad- I. INTRODUCTION here to the following seven principles Data must be Com- In the age of Big Data processing, the power of the Open plete, Primary, Timely, Accessible, Machine readable, Non- Data resource has yet to be fully harnessed to the betterment discriminatory and Non- Proprietary [3]. A fully fledged open of humankind. This has resulted in skeptical views on the data platform solution that meets all the criteria for providing usefulness of Open Data. The Open Data resource consists of open data and promotes the seven principle for openness will data anyone can access, use or share. It is a powerful tool that cater to the needs of both the data publisher and the data can be used to enable citizens, researchers and academics to consumer. develop key resources and make crucial decisions to improve the quality of life for many around the world. A. Highly Available Data With the recent initiatives of governments to increase the Any system or component that can remain operational and availability of open data to the public, the Open Data resource accessible for an indefinite period of time or has sufficient fail has steadily grown. However, the approach of many data pub- over components to avoid any service outages is considered to lishers to opening their data is likely to produce a sea of inter- be Highly Available. When the term high availability (HA) is operable but related raw data streams, only decipherable by mentioned, one may create associations with large data centers arXiv:1606.05832v1 [cs.CY] 19 Jun 2016 the technological elite [1]. This digital divide will effectively and with redundant servers and 100% up-time on services. alienate many data consumers who are unable to acquire or This concept of always available services is translated into employ the technical skills to access or decipher open data. always available data. This means open data platform design- Over the years, there has been the development of numerous ers must choose the right technologies that would help their open data platforms that adhere to may of the open data respective open data platforms perform as highly available policy guidelines [2]. Unfortunately, many of the platforms services. developed have focused on solving the problems faced by the data publisher, exasperating the problem outlined. On the other B. Reusable and Distributive Data hand, platforms that have attempted to provide solutions that In a programming context, re-usability is the extent to cater to both the data publisher and consumer have seen uphill which code or program assets that can be used in different adoption rates by data publishers as extra work is given to data applications with minimal changes being made to the original publishers in the form of preparing data for upload. In this code or asset. Distribution of any code or program asset is paper, we present the Data Transformation and Aggregation limited by the legalities involved in product distribution and Platform (Data-TAP). The Data-TAP is aimed for use by the the rights to the code or asset. Data can be viewed as a program asset as programs require meta-data. It also offers a powerful API that allows third-party data to function and be useful. Thus, licensing for parts or applications and services to be built around it. whole data-sets will allow for the proper distribution of data that protects the rights of data owners. It is also noted, re- B. Socrata usability is the ability for data to be accessed for use in many Socrata is a cloud-based SaaS (Software as a Service) open applications of differing functionality with limited changes data catalog platform that provides API, catalog, and data required to the data. The way to achieve this objective is manipulation tools. On its website, Socrata is described as: through data standardization. Cloud-based solution for federal agencies and governments of all sizes to transform their data into this kind of vital and C. Universally Consumable vibrant asset. It is the only solution that is dedicated to meeting This concern deals more closely with data quality and ease the needs of data publishers as well as data consumers” [7]. of use. The data itself must be applicable to real issues and This platform presents one of the few attempts at a holistic solution formation, and must be easy to digest and understand approach in building Open Data solutions for both the data by users who are not expert in the field of study presented by a publisher and data consumer. given dataset. This would require that data come accompanied One distinguishing feature of Socrata is that it allows users with sufficient meta-data tags and descriptions that would help to create views and visualizations based on published data describe the dataset and its contents to non-expert users of the and save those for others to use. This feature allows for data data. creation from within the platform itself. Additionally, Socrata From an application development perspective, this would offers an open source version of their API that is intended to speak to the need to have all the related data served in a facilitate transitions for customers that decide to migrate away single format along with all the required meta-data needed from the SaaS model. to understand the purpose of the data. This reduces the complexity of applications’ data consumption since all data C. Junar sources are expected to be delivered in a single, predictable Junar is another cloud-based SaaS open data platform so format and helps developers understand how best to use the data is typically managed within Junars infrastructure (the data. all-in-one model). Junar can either provide a complete data catalog or can provide data via an API to a separate user III. REVIEW OF AVAILABLE OPEN DATA PLATFORMS catalog. It has some of the most advanced publishing features In this section we review some of the more popular open as it allows users to scrape data from almost any source (web data solutions [4] as well as a novel one. Platforms such as pages, spread sheets, databases and other file formats). On CKAN contain features such as Data-Proxy [5] which became their website it is described as: Junar provides everything the CKAN datastore that provides partial support for Data- you need to open data with confidence. Its built for massive TAP features. The review presented in Table I summarizes the scale and supports local and global data-driven organizations. differences. With Junar, you can easily choose what data to collect, how you want to present it, and when it should be made publicly A. CKAN available. You can also determine which data-sets are made CKAN is the most popular Open Data platform in adoption. available to the public and which data-sets are available only Developed and maintained by The Open Data Foundation, for your internal use. Think of it as a next generation data CKANs goal is to provide tools to streamline the publishing, management system for sharing information” [8]. sharing and finding of data. CKAN is aimed at national and Junar is especially aimed at governments and national insti- regional governments, companies and organizations wanting tutions. It has very close comparisons to the CKAN platform to make their data open and available. with more polished features available. It also incorporates The platform’s mission statement is presented on their web- various analytic and usage tracing features, leveraging trusted site as follows: CKAN is a powerful data management system platforms such as Google Analytic.