Creating Dashboards and Data Stories Within the Data & Analytics Framework (DAF)

Total Page:16

File Type:pdf, Size:1020Kb

Creating Dashboards and Data Stories Within the Data & Analytics Framework (DAF) Creating dashboards and data stories within the Data & Analytics Framework (DAF) Michele Petitoc, Francesca Fallucchia,b and De Luca Ernesto Williama,b a DIII, Guglielmo Marconi University, Via Plinio 44, 00193 Roma RM, Italy E-mail: [email protected], [email protected] b DIFI, Georg Eckert Institute Braunschweig, Celler Str. 3, 38114 Braunschweig, German E-mail: [email protected], [email protected] c DIFI, Università̀ di Pisa, Lungarno Antonio Pacinotti 43, 56126, Pisa PI, Italy E-mail: [email protected] Abstract. In recent years, many data visualization tools have appeared on the market that can potentially guarantee citizens and users of the Public Administration (PA) the ability to create dashboards and data stories with just a few clicks, using open and unopened data from the PA. The Data Analytics Framework (DAF), a project of the Italian government launched at the end of 2017 and currently being tested, integrates data based on the semantic web, data analysis tools and open source business intelli- gence products that promise to solve the problems that prevented the PA to exploit its enormous data potential. The DAF favors the spread of linked open data (LOD) thanks to the integration of OntoPiA, a network of controlled ontologies and vocabularies that allows us to describe the concepts we find in datasets, such as "sex", "organization", "people", "addresses", "points of inter- est", "events" etc. This paper contributes to the enhancement of the project by introducing the process of creating a dashboard in the DAF in 5 steps, starting from the dataset search on the data portal, to the creation phase of the real dashboard through Superset and the related data story. The case study created by the author, concerns tourism of Sardinia (a region of Italy). This case study is one of the few demonstrations of use on a real case of DAF and highlights the ability of the DAF to transform the analysis of a large amount of data into simple visual representations with clear and effective language. Keywords: Data & Analytics Framework, Data Visualization, Dashboard, Ontology, Controlled vocabulary, Open Data. 1. Introduction (AgID) [1] at the end of 2012 through the national guidelines [2] that paved the way to the use of LOD as Open and Closed Public Administration (PA) Data, the data paradigm for enabling semantic interoperabil- are one of the major digital useful assets for interpret- ity in the collaboration between PAs.. ing social and economic phenomena, making deci- In this scenario, the big data platform DAF [3] de- sions, for improving services to citizens and compet- signed by the Digital Transformation Team [4], a in- ing on the international scene. To better exploit their frastructure of the Italian Government established in potential, it is necessary to leave the approach based September 2016, represents Italy's latest effort to val- on silos in favor of a systemic vision: a platform that orize public information assets. The objective of the supports data sharing and access. This is especially DAF is to overcome these difficulties by channeling true of open data, whose spread has slowed in recent data flows from the various local and central PAs into years because large number of databases and open a large container (data lake) where, thanks to a frame- data portals are only available as silos in the various work for distributed applications such as Apache Ha- PAs, so non-interoperable between them. Furthermore, doop [5], data can be explored and analyzed, also us- very few PAs have adopted standardized production, ing machine learning techniques, to fully exploit in- analysis or data storage, according to the methodolog- sights and information hidden in it. Thanks to a very ical approach defined by the Agency for Digital Italy simple interface and an API (Application Program Interface) system, the DAF allows individual admin- speed, and show the DAF project which integrates two istrations to communicate and share data, analysis and business intelligence tools for data representation Su- dashboards using open standards and open source perset [9] and Metabase [10]. There are many other technologies. Furthermore, the DAF promotes seman- platforms in the same category such as Microsoft tic interoperability, according to the new European In- Power BI [11], Tableau [12], Google Data Studio [13] teroperability Framework (EIF) [6]. To enhance in- e Ploty.ly [14] that offer the possibility of use in the teroperability DAF make use of an ecosystem of on- cloud via API and that could allow integration with the tologies and controlled vocabularies (OntoPiA) [7], a DAF, even if with important limitations. set of ingestion pipelines that performs standardiza- Public Tableau allows the creation of complex tion to incoming data, and a rich set of metadata. Every dashboards with great flexibility, do not require spe- dataset in DAF is accompanied by metadata that de- cific technical skills nevertheless an application needs scribes the dataset and its internal structure. It will be to be installed in locale. the user's responsibility to define the ontological infor- Google Data Studio is very intuitive but does not mation and controlled vocabularies associated with allow the use of more than one dataset within the same the data structure, through the meaning of the semantic dashboard. It has few data connectors (only for MySql tags. A tagging system will allow to drive the user to [15], and for PostgreSQL [16]) and the quality of the the correct use of controlled vocabularies and to en- dashboards is not comparable to that of the competi- sure that all datasets can be effectively connected to- tors in the sector. gether. Plot.ly allows the creation and sharing of quick in- In this paper, we focus on DAF data visualization teractive dashboards. But it only accepts datasets with tools, illustrating the weaknesses of some of the most a maximum size of 5 MB and the published graphics famous data visualization tools on the market and the must be public (for them to be private it is necessary advantages of the integration in the DAF of two open to pay a subscription). Among the open source catego- source software (Metabase and Superset, see Section ries we have instead distinguished three data visuali- 2). In Section 3, we present the general architecture of zation tools: Superset [9], Metabase [10], and Redash the DAF, its main functionalities and the semantic [17]. technologies used, with particular reference to Onto- All these projects meet the requirements [18] that PiA, the network of controlled ontologies and vocab- an OGD2 visualization tool should possess. Superset ularies. Section 4 focuses on the process of building a and Redash are very similar. Both are powerful and dashboard through Apache Superset. Finally, a case of give the possibility to connect to a large number of use of the DAF will be presented for the construction data sources. In addition, they have a powerful inter- of a dashboard starting from a dataset on tourism in face for writing and executing SQL queries. the Sardinia Region (see Section 5). We conclude the Once saved, queries can also be used as a basis for paper with some future developments (see Section 6). the creation of dashboards. Superset supports a larger number of authentication systems than Redash and 2. Related works Metabase. For example, it includes LDAP3 , the sys- tem used for the unique authentication of all modules William S. Cleveland said “The human brain is in the DAF. Although Redash is an excellent project more able to identify and comprehend relationships in rapid evolution, Superset [9] has been chosen for and patterns if data is encoded into visual forms. three main reasons: the presence of LDAP authentica- Graphs allow us to explore data and observe patterns tion, the high number of views, and the support of the that no other approach can achieve.” [8] Python language (used throughout the DAF project). According to this, we believe that the identification In addition, Superset was preferred to Metabase, alt- of information needs is useful for the definition of data hough Metabase is more intuitive, because its graphs driven policy and the realization of interface for ana- are less parametrizable and offers much less types of lytical tools. visual representations. Therefore, we consider the importance of data vis- ualization1 and the goal of the PA to obtain useful trends and information with maximum simplicity and 1 The science of visual representation of ‘data’, which has been 2 Open Government Data. http://www.oecd.org/gov/digital-gov- abstracted in some schematic form, including attributes or variables ernment/open-government-data.htm for the units of information. 3 Lightweight Directory Access Protocol. http://foldoc.org/ldap Superset is an open source product hosted on the national catalog data. Instead, in the private section, Apache Foundation GitHub4 platform and is devel- the acces is done through a login, thus allowing only oped using Flask [19] a very lean Python framework accredited users to exploit the functionality of query- [20] for web development. The part that generates the ing, analyzing and sharing data. interactive graphs instead makes use of NVD3 [21] a javascript library built on D3.js [22]. Any dashboard created with Superset consists of a series of graphs (called slices in the current version of Superset installed in the DAF). Each of these can be resized, moved relative to the others, or shown in full screen. In addition, each dataset represented in a graph can also be exported in CSV or JSON format or through SQL queries. The "slices" are created starting from a table available in the many data sources that Superset is able to manage. Superset provides two main inter- faces: the first is the Rich SQL IDE (Interactive De- velopment Environment) called Sql Lab5 with which Fig.
Recommended publications
  • IPS Signature Release Note V9.17.79
    SOPHOS IPS Signature Update Release Notes Version : 9.17.79 Release Date : 19th January 2020 IPS Signature Update Release Information Upgrade Applicable on IPS Signature Release Version 9.17.78 CR250i, CR300i, CR500i-4P, CR500i-6P, CR500i-8P, CR500ia, CR500ia-RP, CR500ia1F, CR500ia10F, CR750ia, CR750ia1F, CR750ia10F, CR1000i-11P, CR1000i-12P, CR1000ia, CR1000ia10F, CR1500i-11P, CR1500i-12P, CR1500ia, CR1500ia10F Sophos Appliance Models CR25iNG, CR25iNG-6P, CR35iNG, CR50iNG, CR100iNG, CR200iNG/XP, CR300iNG/XP, CR500iNG- XP, CR750iNG-XP, CR2500iNG, CR25wiNG, CR25wiNG-6P, CR35wiNG, CRiV1C, CRiV2C, CRiV4C, CRiV8C, CRiV12C, XG85 to XG450, SG105 to SG650 Upgrade Information Upgrade type: Automatic Compatibility Annotations: None Introduction The Release Note document for IPS Signature Database Version 9.17.79 includes support for the new signatures. The following sections describe the release in detail. New IPS Signatures The Sophos Intrusion Prevention System shields the network from known attacks by matching the network traffic against the signatures in the IPS Signature Database. These signatures are developed to significantly increase detection performance and reduce the false alarms. Report false positives at [email protected], along with the application details. January 2020 Page 2 of 245 IPS Signature Update This IPS Release includes Two Thousand, Seven Hundred and Sixty Two(2762) signatures to address One Thousand, Nine Hundred and Thirty Eight(1938) vulnerabilities. New signatures are added for the following vulnerabilities: Name CVE–ID
    [Show full text]
  • ECM Platforms: the Next Generation of Enterprise Content Management
    ECM Platforms: The Next Generation of Enterprise Content Management The Evolution of ECM: Platform Oriented, Flexible, Architected for the Cloud and Designed for Technologists Over the last decade, content technology, use and expectations have signicantly evolved - redening the requirements for architects, project managers and developers responsible for implementing ECM solutions. Emerging best practices suggest that a well-designed ECM platform supports solution development and deployment in a manner that is predictable, sustainable and aordable. This white paper explores these content technology changes, and explains the value and benets to business and technical adopters of embracing a new generation of ECM platform technology. Eric Barroca, Roland Benedetti - August 12, 2011 © Copyright 2011 - 2012 Nuxeo. All Rights Reserved. www.nuxeo.com Taking a Platform Approach to Building Content Centric Applications Contents Executive Summary_______________________________________________________4 Scope and Goals__________________________________________________________4 Target Audience__________________________________________________________4 Defining Enterprise Content and Enterprise Content Management_______________5 What Is Enterprise Content Management?_____________________________________5 What Is Enterprise Content?_________________________________________________6 Drivers for ECM Adoption___________________________________________________6 Enterprise Content Trends__________________________________________________8 ECM: It's not Just
    [Show full text]
  • State of the Art Technologies and Architectures for EMPOWER
    Deliverable D3.1.1 State of the Art Technologies and Architectures for EMPOWER Project title: Support of Patient Empowerment by an intelligent self- management pathway for patients Project acronym: EMPOWER Project identifier: FP7-ICT-2011-288209 Project instrument: STREP Web link: www.empower-fp7.eu Dissemination level: PU (public) Contractual delivery: 2012-06-30 Actual delivery: 2012-06-28 Leading partner: HMGU The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/2007-2013) under grant agreement No 288209, EMPOWER Project. FP7-288209 EMPOWER Document History Version Date Changes From Review HMGU, V0.1 2012-02-29 Initial Document All Partners SRFG V0.2 2012-05-15 Content added All Partners All Partners V0.4 2012-05-25 Comments SRFG All Partners MOH, SRFG, V0.5 2012-06-14 Content added All Partners GOIN HMGU, V0.6 2012-06-19 Content added All Partners ICOM HMGU, V0.7 2012-06-22 Content added SRFG, MOH, All Partners GOIN, USI V1.0 2012-06-27 Final revision All Partners - EMPOWER Consortium Contacts Beneficiary Name Phone E-Mail SRFG Manuela Plößnig +43 662 2288 402 [email protected] HMGU Claudia Hildebrand +49 89 3187 4182 [email protected] GOIN Siegfried Jedamzik +49 8 41956161 [email protected] USI Peter J. Schulz +41586664724 [email protected] SRDC Asuman Dogac +90 312 210 13 93 [email protected] ICOM Ilias Lamprinos +302106677953 [email protected] MOH Ali Kemal Caylan +903125851907 [email protected] d311_empower_technologies-and-architectures-v10.docx 2 / 136 FP7-288209 EMPOWER Table of Contents 1 Summary .......................................................................................................................
    [Show full text]
  • HDP 3.1.4 Release Notes Date of Publish: 2019-08-26
    Release Notes 3 HDP 3.1.4 Release Notes Date of Publish: 2019-08-26 https://docs.hortonworks.com Release Notes | Contents | ii Contents HDP 3.1.4 Release Notes..........................................................................................4 Component Versions.................................................................................................4 Descriptions of New Features..................................................................................5 Deprecation Notices.................................................................................................. 6 Terminology.......................................................................................................................................................... 6 Removed Components and Product Capabilities.................................................................................................6 Testing Unsupported Features................................................................................ 6 Descriptions of the Latest Technical Preview Features.......................................................................................7 Upgrading to HDP 3.1.4...........................................................................................7 Behavioral Changes.................................................................................................. 7 Apache Patch Information.....................................................................................11 Accumulo...........................................................................................................................................................
    [Show full text]
  • Full-Graph-Limited-Mvn-Deps.Pdf
    org.jboss.cl.jboss-cl-2.0.9.GA org.jboss.cl.jboss-cl-parent-2.2.1.GA org.jboss.cl.jboss-classloader-N/A org.jboss.cl.jboss-classloading-vfs-N/A org.jboss.cl.jboss-classloading-N/A org.primefaces.extensions.master-pom-1.0.0 org.sonatype.mercury.mercury-mp3-1.0-alpha-1 org.primefaces.themes.overcast-${primefaces.theme.version} org.primefaces.themes.dark-hive-${primefaces.theme.version}org.primefaces.themes.humanity-${primefaces.theme.version}org.primefaces.themes.le-frog-${primefaces.theme.version} org.primefaces.themes.south-street-${primefaces.theme.version}org.primefaces.themes.sunny-${primefaces.theme.version}org.primefaces.themes.hot-sneaks-${primefaces.theme.version}org.primefaces.themes.cupertino-${primefaces.theme.version} org.primefaces.themes.trontastic-${primefaces.theme.version}org.primefaces.themes.excite-bike-${primefaces.theme.version} org.apache.maven.mercury.mercury-external-N/A org.primefaces.themes.redmond-${primefaces.theme.version}org.primefaces.themes.afterwork-${primefaces.theme.version}org.primefaces.themes.glass-x-${primefaces.theme.version}org.primefaces.themes.home-${primefaces.theme.version} org.primefaces.themes.black-tie-${primefaces.theme.version}org.primefaces.themes.eggplant-${primefaces.theme.version} org.apache.maven.mercury.mercury-repo-remote-m2-N/Aorg.apache.maven.mercury.mercury-md-sat-N/A org.primefaces.themes.ui-lightness-${primefaces.theme.version}org.primefaces.themes.midnight-${primefaces.theme.version}org.primefaces.themes.mint-choc-${primefaces.theme.version}org.primefaces.themes.afternoon-${primefaces.theme.version}org.primefaces.themes.dot-luv-${primefaces.theme.version}org.primefaces.themes.smoothness-${primefaces.theme.version}org.primefaces.themes.swanky-purse-${primefaces.theme.version}
    [Show full text]
  • Presto: the Definitive Guide
    Presto The Definitive Guide SQL at Any Scale, on Any Storage, in Any Environment Compliments of Matt Fuller, Manfred Moser & Martin Traverso Virtual Book Tour Starburst presents Presto: The Definitive Guide Register Now! Starburst is hosting a virtual book tour series where attendees will: Meet the authors: • Meet the authors from the comfort of your own home Matt Fuller • Meet the Presto creators and participate in an Ask Me Anything (AMA) session with the book Manfred Moser authors + Presto creators • Meet special guest speakers from Martin your favorite podcasts who will Traverso moderate the AMA Register here to save your spot. Praise for Presto: The Definitive Guide This book provides a great introduction to Presto and teaches you everything you need to know to start your successful usage of Presto. —Dain Sundstrom and David Phillips, Creators of the Presto Projects and Founders of the Presto Software Foundation Presto plays a key role in enabling analysis at Pinterest. This book covers the Presto essentials, from use cases through how to run Presto at massive scale. —Ashish Kumar Singh, Tech Lead, Bigdata Query Processing Platform, Pinterest Presto has set the bar in both community-building and technical excellence for lightning- fast analytical processing on stored data in modern cloud architectures. This book is a must-read for companies looking to modernize their analytics stack. —Jay Kreps, Cocreator of Apache Kafka, Cofounder and CEO of Confluent Presto has saved us all—both in academia and industry—countless hours of work, allowing us all to avoid having to write code to manage distributed query processing.
    [Show full text]
  • Hortonworks Data Platform Date of Publish: 2018-09-21
    Release Notes 3 Hortonworks Data Platform Date of Publish: 2018-09-21 http://docs.hortonworks.com Contents HDP 3.0.1 Release Notes..........................................................................................3 Component Versions.............................................................................................................................................3 New Features........................................................................................................................................................ 3 Deprecation Notices..............................................................................................................................................4 Terminology.............................................................................................................................................. 4 Removed Components and Product Capabilities.....................................................................................4 Unsupported Features........................................................................................................................................... 4 Technical Preview Features......................................................................................................................4 Upgrading to HDP 3.0.1...................................................................................................................................... 5 Before you begin.....................................................................................................................................
    [Show full text]
  • Kyriakakisalexandros2019.Pdf
    TECHNOLOGICAL EDUCATIONAL INSTITUTE OF CRETE SCHOOL OF ENGINEERING DEPARTMENT OF INFORMATICS ENGINEERING THESIS ONTOLOGY-BASED SEARCH ENGINE WITH SPEECH RECOGNITION ALEXANDROS KYRIAKAKIS SUPERVISOR PROFESSOR MANOLIS TSIKNAKIS HERAKLION FEBRUARY 2019 Kyriakakis Alexandros Copyright ALEXANDROS KYRIAKAKIS 2019 Formal Declaration: We assure you that we are authors of this bachelor thesis and that the assistance we received during its preparation is fully recognized and it is stated in the thesis. Also, we have stated all the sources which we made use of data, ideas or words from, either in their exact form or changed. We also assure you that this thesis was prepared by us personally for the curriculum of the Department of Informatics Engineering of TEI of Crete. Υπεύθυνη Δήλωση: Βεβαιώνουμε ότι είμαστε συγγραφείς αυτής της πτυχιακής εργασίας και ότι κάθε βοήθεια την οποία είχαμε για την προετοιμασία της, είναι πλήρως αναγνωρισμένη και αναφέρεται στην πτυχιακή εργασία. Επίσης έχουμε αναφέρει τις όποιες πηγές από τις οποίες κάναμε χρήση δεδομένων, ιδεών ή λέξεων, είτε αυτές αναφέρονται ακριβώς είτε παραφρασμένες. Επίσης βεβαιώνουμε ότι αυτή η πτυχιακή εργασία προετοιμάστηκε από εμάς προσωπικά ειδικά για τις απαιτήσεις του προγράμματος σπουδών του Τμήματος Μηχανικών Πληροφορικής του Τ.Ε.Ι. Κρήτης. 2 Ontology-based search engine with speech recognition Abstract Searching for data throughout the internet is not always an easy task. The reason for that is that the internet is flooded with data and the most common searching techniques often fail to provide the best possible results for the users’ needs. Therefore, researchers have for years been trying to find alternative ways of searching. Through that research, searching techniques based on semantic technologies have emerged.
    [Show full text]
  • Developer Documentation
    Developer Documentation Nuxeo Platform 5.5 Table of Contents 1. Technical Documentation Center . 4 1.1 Overview and Architecture . 5 1.2 Overview . 6 1.3 Architecture . 9 1.3.1 Architecture overview . 9 1.3.2 About the content repository . 12 1.3.2.1 VCS Architecture . 21 1.3.3 Platform features quick overview . 30 1.3.4 Component model overview . 32 1.3.5 API and connectors . 36 1.3.6 UI frameworks . 39 1.3.7 Deployment options . 42 1.3.8 Performance management for the Nuxeo Platform . 47 1.4 Customization and Development . 53 1.4.1 Learning to customize Nuxeo EP . 54 1.4.2 Document types . 55 1.4.3 Document, form and listing views . 62 1.4.3.1 Layouts (forms and views) . 62 1.4.3.1.1 Manage layouts . 63 1.4.3.1.2 Document layouts . 70 1.4.3.1.3 Layout display . 71 1.4.3.1.4 Standard widget types . 72 1.4.3.1.5 Custom templates . 73 1.4.3.1.6 Custom widget types . 85 1.4.3.1.7 Generic layout usage . 93 1.4.3.2 Content views . 93 1.4.3.2.1 Custom Page Providers . 100 1.4.3.2.2 Page Providers without Content Views . 101 1.4.3.3 Views on documents . 102 1.4.4 Versioning . 104 1.4.5 User Actions (links, buttons, icons, tabs) . 105 1.4.6 Events and Listeners . 108 1.4.6.1 Scheduling periodic events . 113 1.4.7 Tagging .
    [Show full text]
  • Structuring Medical Records with Apache Stanbol
    Structuring Medical Records with Apache Stanbol Rafa Haro, Senior Software Engineer, Athento Antonio Pérez Morales, Senior Software Engineer, Ixxus • Committer, PMC Member @ Apache Stanbol, Apache ManifoldCF • Topics: Document Analysis, NLP, Machine Learning, Semantic Technologies, ECM • Committer @ Apache Stanbol, Apache ManifoldCF • Topics: ECM, Semantic Search, ETL, Machine Learning Apache Stanbol provides a set of reusable components for semantic content management. It extends existing CMSs with a number of semantic services. Traditional Semantic CMS Software Architecture for Semantically Enabled CM and ECM systems Apache Stanbol Story • Started within FP7 European Project IKS (Interactive Knowledge Stack. 2009 - 2012) • IKS project brought together an Open Source Community for Defining and Building Platforms in the Semantic CMS Space • Incubated in November 2010 • Successfully promoted within CMS and ECM industry through IKS Early Adopters Program • Graduated to Top-Level Apache Project in October 2012 What is a Semantic CMS? Traditional CMS Semantic CMS Atomic Unit: Document Atomic Unit: Entity Properties as meta-data Semantic meta-data (key-value schemas) (RDF) Keyword Search Semantic Search Document Management Knowledge Management Document Types Entity Management Document Workflow Ontologies Source: What Apache Stanbol Can Do for You?. Fabian Christ. ApacheCon Europe 2012 Key Points • Designed to bring Semantic Technologies to existing CMS • Non-intrusive set of RESTful ‘Semantic’ Services • Extremely Modular: Use only the modules
    [Show full text]
  • Code Smell Prediction Employing Machine Learning Meets Emerging Java Language Constructs"
    Appendix to the paper "Code smell prediction employing machine learning meets emerging Java language constructs" Hanna Grodzicka, Michał Kawa, Zofia Łakomiak, Arkadiusz Ziobrowski, Lech Madeyski (B) The Appendix includes two tables containing the dataset used in the paper "Code smell prediction employing machine learning meets emerging Java lan- guage constructs". The first table contains information about 792 projects selected for R package reproducer [Madeyski and Kitchenham(2019)]. Projects were the base dataset for cre- ating the dataset used in the study (Table I). The second table contains information about 281 projects filtered by Java version from build tool Maven (Table II) which were directly used in the paper. TABLE I: Base projects used to create the new dataset # Orgasation Project name GitHub link Commit hash Build tool Java version 1 adobe aem-core-wcm- www.github.com/adobe/ 1d1f1d70844c9e07cd694f028e87f85d926aba94 other or lack of unknown components aem-core-wcm-components 2 adobe S3Mock www.github.com/adobe/ 5aa299c2b6d0f0fd00f8d03fda560502270afb82 MAVEN 8 S3Mock 3 alexa alexa-skills- www.github.com/alexa/ bf1e9ccc50d1f3f8408f887f70197ee288fd4bd9 MAVEN 8 kit-sdk-for- alexa-skills-kit-sdk- java for-java 4 alibaba ARouter www.github.com/alibaba/ 93b328569bbdbf75e4aa87f0ecf48c69600591b2 GRADLE unknown ARouter 5 alibaba atlas www.github.com/alibaba/ e8c7b3f1ff14b2a1df64321c6992b796cae7d732 GRADLE unknown atlas 6 alibaba canal www.github.com/alibaba/ 08167c95c767fd3c9879584c0230820a8476a7a7 MAVEN 7 canal 7 alibaba cobar www.github.com/alibaba/
    [Show full text]
  • Data Quality Centric Application Framework for Big Data
    ALLDATA 2016 : The Second International Conference on Big Data, Small Data, Linked Data and Open Data (includes KESA 2016) Data Quality Centric Application Framework for Big Data Venkat N. Gudivada∗, Dhana Rao y, and William I. Grosky z ∗Department of Computer Science, East Carolina University, USA yDepartment of Biology, East Carolina University, USA zDepartment of Computer and Information Science, University of Michigan - Dearborn, USA email: [email protected], [email protected], and [email protected] Abstract—Risks associated with data quality in Big and load (ETL) process and analyze large datasets. Pig Data have wide ranging adverse implications. Current re- generates MapReduce jobs that perform the dataflows search in Big Data primarily focuses on the proverbial and thus provides a high level abstract interface to harvesting of low hanging fruit and applications are de- veloped using the Hadoop Ecosystem. In this paper, we MapReduce. The Pig Latin enhances the Pig through a discuss the risks and attendant consequences emanating programming language extension. It provides common from data quality in Big Data. We propose a data quality data manipulation operations such as grouping, joining, centric framework for Big Data applications and describe an and filtering. Hive is a tool for enabling data summa- approach to implementing it. rization, ad hoc query execution, and analysis of large Keywords—Big Data; Data Quality; Data Analytics; Application Framework. datasets stored in HDFS-compatible file systems. In other words, Hive serves as a SQL-based data warehouse for the Hadoop Ecosystem. I. Introduction The other widely used tools in the Hadoop Ecosys- Big Data, which has emerged in the last five years, tem include Cascading, Scalding, and Cascalog.
    [Show full text]