Modern ETL Tools for Cloud and Big Data

Total Page:16

File Type:pdf, Size:1020Kb

Modern ETL Tools for Cloud and Big Data Modern ETL Tools for Cloud and Big Data Ken Beutler, Principal Product Manager, Progress Michael Rainey, Technical Advisor, Gluent Inc. Agenda ▪ Landscape ▪ Cloud ETL Tools ▪ Big Data ETL Tools ▪ Best Practices ▪ QnA © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 2 Landscape © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 3 Data Warehousing Has Transformed Enterprise BI and Analytics © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 4 ETL Tools Are At The Heart Of EDW © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 5 Rise Of The Modern EDW Solutions Cloud Data Warehouse Big Data Warehouse © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 6 Enterprises Are Increasingly Adopting Cloud and Big Data Cloud Big Data NoSQL Amazon S3 Amazon Web Services 24% MongoDB 27% 44% Hadoop Hive 22% Cassandra Microsoft Azure 39% Spark SQL 11% 15% Redis 7% VMware 22% Hortonworks 9% Amazon EMR 9% Oracle NoSQL 7% Google Cloud 18% Apache Solr 8% Google Cloud… 6% Salesforce 14% Cloudera CDH 7% HBase 6% Oracle 10% Oracle BDA 7% DynamoDB 5% Apache Sqoop 7% Couchbase 4% IBM 8% 7% IBM BigInsights Google Cloud… 3% 6% SAP HANA 6% SAP Cloud Platform Big Data… DocumentDB Cloudera Impala 2% Rackspace 6% 5% MapR 6% SimpleDB 2% RedHat OpenStack 4% Azure HDInsights 6% MarkLogic 1% DataStax Enterprise RedHat OpenShift 3% Apache Storm 5% 1% Google Cloud DataProc 5% Aerospike 1% Digital Ocean 2% Apache Drill 4% Riak 0% CenturyLink 1% Apache Phoenix 3% Scylla 0% Linode 0% Presto 2% ArangoDB 0% Pivotal HD 2% Other (please… 2% Other (please specify): 6% GemFireXD 1% None 32% None 13% Other (please specify): 2% Not sure 10% Not sure 3% None 32% Not sure 12% Source: Progress DataDirect’s Data Connectivity Outlook Survey 2018 © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 7 Cloud ETL Tools © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 8 Poll Question 1 ▪ Which cloud provider are you currently using / plan to use? ▪ AWS ▪ Google Cloud Platform ▪ Azure ▪ Other ▪ None © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 9 AWS Glue Serverless ETL AWS Glue ▪ Components: • Data Catalog • Crawlers • ETL jobs/scripts • Job scheduler ▪ Useful for… • …running serverless queries against S3 buckets and relational data • …creating event-driven ETL pipelines • …automatically discovering and cataloging your enterprise data assets © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 11 AWS Glue architecture Source: https://docs.aws.amazon.com/athena/latest/ug/glue-athena.html © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 12 Data catalog - the central component ▪ Can act as metadata repository for other Amazon services ▪ Tables - Added to “databases” using the wizard or a crawler ▪ Data sources: Amazon S3, Redshift, Aurora, Oracle, PostgreSQL, MySQL, MariaDB, MS SQL Server, JDBC, DynamoDB ▪ Crawlers connect to one or more data stores, determine the data structures, and write tables into the Data Catalog © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 13 Jobs ▪ PySpark or Scala scripts, generated by AWS Glue ▪ Visual dataflow can be generated, but not used for development ▪ Execute ETL using the job scheduler, events, or manually invoke ▪ Built-in transforms used to process data ▪ ApplyMapping ▪ SelectFields • Maps source and target columns • Output selected fields to new DynamicFrame ▪ Filter ▪ SplitRows • Load new DynamicFrame based on • Split rows into two new DynamicFrames filtered records based on a predicate ▪ Join ▪ SplitFields • Joins two DynamicFrames • Split fields into two new DynamicFrames © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 14 Azure Data Factory Visual Cloud ETL Azure Data Factory ▪ Build data pipelines using a visual ETL user interface • Visual Studio Team Services (VSTS) Git integration for collaboration, source control, and versioning ▪ Drag, drop, link activities • Copy Data: Source to Target • Transform: Spark, Hive, Pig, streaming on HDInsight, Stored Procedures, ML Batch Execution, etc. • Control flow: If-then-else, For-each, Lookup, etc. Source: https://azure.microsoft.com/en-us/blog/continuous-integration-and-deployment-using-data-factory/ © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 16 Linked services ▪ Allow connection to many data sources • Source: – Azure DBs, Azure Blob Storage, Azure File Storage, Oracle, SQL Server, SAP HANA, Sybase, DB2, Impala, Drill, MySQL, Google BigQuery, Cassandra, Amazon S3, HDFS, Salesforce, etc. • Target: – Azure DBs, Azure Blob Storage, Azure File Storage, Oracle, SQL Server, SAP HANA, File System, Generic ODBC, Salesforce, etc. © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 17 Integration runtime environment ▪ Compute environment required for Azure Data Factory • Fully managed in Azure cloud infrastructure • Automatic scaling, high availability, etc. • Allows execution of SSIS packages ▪ Can be hosted on a local private network • Organizations can control, and manage, compute resources © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 18 Google Cloud Dataflow Unified Programming Model for Data Processing Google Cloud Dataflow Source: https://cloud.google.com/dataflow/ © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 20 Google Cloud Dataflow overview ▪ Unified programming model for batch (historical) and streaming (real-time) data pipelines and distributed compute platform • Reuse code across both batch and streaming pipelines • Java or Python based ▪ Programming model an open source project - Apache Beam (https://beam.apache.org/) • Runs on multiple different distributed processing back-ends: Spark, Flink, Cloud Dataflow platforms ▪ Fully managed service • Automated resource management and scale-out Source: https://cloud.google.com/dataflow/ © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 21 Dataflow I/O transforms JDBC allows custom connection to Language File-based Messaging Databaseadditional databases Java Beam Java supports Apache HDFS, Amazon Amazon Kinesis Apache Cassandra S3, Google Cloud Storage, and local AMQP Apache Hadoop InputFormat filesystems Apache Kafka Apache Hbase FileIO, AvroIO, TextIO, TFRecordIO, XmlIO, TikaIO, ParquetIO Google Cloud Pub/Sub Apache Hive (HCatalog) JMS Apache Solr MQTT JDBC ElasticSearch Many different Google BigQuery Google Cloud BigTable filesystems and cloud Google Cloud DatastoreOther I/O object stores supported Google Cloud Spannertransforms are MongoDB Redis already under Python Beam Python supports Google Cloud Storage Google Cloud Pub/Sub Google BigQuery development and local filesystems Google Cloud Datastore avroio, textio, tfrecordio, vcfio © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 22 Cloud ETL tools compared AWS Glue Google Dataflow Azure Data Factory Batch ETL X X X Streaming - X - User Interface - - * X Compute data platform - X - Cross-platform support X X X Custom connector support X X X Metadata catalog X - - Monitoring tools available X X X Fully managed X X X *Cloud Dataprep © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 23 Big Data ETL Tools © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 24 Poll Question 2 ▪ Which of the following tools are you currently using / plan to use? ▪ Apache Sqoop ▪ Apache Nifi ▪ Apache Flume ▪ Other ▪ None © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 25 Apache Sqoop Batch Data Transfer Sqoop Overview ▪ Sqoop is a tool designed to transfer data between Hadoop and RDBMS or mainframes (bi-directional) ▪ Open source (Apache) command line tool ▪ Batch/Bulk transfer of structured data ▪ Requires Hadoop • Leverages MapReduce to provide parallel execution and data partitioning © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 27 Sqoop Details Importing Data Exporting Data ▪ Bulk move data from RDBMS or mainframe to ▪ Exports a set of files in HDFS to RDBMS Hadoop ▪ Uses JDBC* for connectivity • Persist in HDFS, Hive, HBase or Accumulo ▪ Leverages either update or insert mode or ▪ Uses JDBC* for connectivity incremental imports ▪ Able to perform column and row filtering or free- ▪ Limited transformation support – column form queries filtering • Must use MapReduce, Pig or other scripting ▪ Constraints: language • Table must exist • Operation is not atomic * Very limited number of databases are supported and version specific © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 28 Sqoop Details ▪ Sqoop jobs can created and saved to be used as templates ▪ Sqoop jobs can be scheduled using Oozie ▪ sqoop-merge can be used to combine two datasets into one and overwrite values of an older dataset ▪ Plugin architecture $ sqoop import --connect
Recommended publications
  • Working with Storm Topologies Date of Publish: 2018-08-13
    Apache Storm 3 Working with Storm Topologies Date of Publish: 2018-08-13 http://docs.hortonworks.com Contents Packaging Storm Topologies................................................................................... 3 Deploying and Managing Apache Storm Topologies............................................4 Configuring the Storm UI.................................................................................................................................... 4 Using the Storm UI.............................................................................................................................................. 5 Monitoring and Debugging an Apache Storm Topology......................................6 Enabling Dynamic Log Levels.............................................................................................................................6 Setting and Clearing Log Levels Using the Storm UI.............................................................................6 Setting and Clearing Log Levels Using the CLI..................................................................................... 7 Enabling Topology Event Logging......................................................................................................................7 Configuring Topology Event Logging.....................................................................................................8 Enabling Event Logging...........................................................................................................................8
    [Show full text]
  • Apache Flink™: Stream and Batch Processing in a Single Engine
    Apache Flink™: Stream and Batch Processing in a Single Engine Paris Carboney Stephan Ewenz Seif Haridiy Asterios Katsifodimos* Volker Markl* Kostas Tzoumasz yKTH & SICS Sweden zdata Artisans *TU Berlin & DFKI parisc,[email protected][email protected][email protected] Abstract Apache Flink1 is an open-source system for processing streaming and batch data. Flink is built on the philosophy that many classes of data processing applications, including real-time analytics, continu- ous data pipelines, historic data processing (batch), and iterative algorithms (machine learning, graph analysis) can be expressed and executed as pipelined fault-tolerant dataflows. In this paper, we present Flink’s architecture and expand on how a (seemingly diverse) set of use cases can be unified under a single execution model. 1 Introduction Data-stream processing (e.g., as exemplified by complex event processing systems) and static (batch) data pro- cessing (e.g., as exemplified by MPP databases and Hadoop) were traditionally considered as two very different types of applications. They were programmed using different programming models and APIs, and were exe- cuted by different systems (e.g., dedicated streaming systems such as Apache Storm, IBM Infosphere Streams, Microsoft StreamInsight, or Streambase versus relational databases or execution engines for Hadoop, including Apache Spark and Apache Drill). Traditionally, batch data analysis made up for the lion’s share of the use cases, data sizes, and market, while streaming data analysis mostly served specialized applications. It is becoming more and more apparent, however, that a huge number of today’s large-scale data processing use cases handle data that is, in reality, produced continuously over time.
    [Show full text]
  • Administration and Configuration Guide
    Red Hat JBoss Data Virtualization 6.4 Administration and Configuration Guide This guide is for administrators. Last Updated: 2018-09-26 Red Hat JBoss Data Virtualization 6.4 Administration and Configuration Guide This guide is for administrators. Red Hat Customer Content Services Legal Notice Copyright © 2018 Red Hat, Inc. This document is licensed by Red Hat under the Creative Commons Attribution-ShareAlike 3.0 Unported License. If you distribute this document, or a modified version of it, you must provide attribution to Red Hat, Inc. and provide a link to the original. If the document is modified, all Red Hat trademarks must be removed. Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law. Red Hat, Red Hat Enterprise Linux, the Shadowman logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries. Linux ® is the registered trademark of Linus Torvalds in the United States and other countries. Java ® is a registered trademark of Oracle and/or its affiliates. XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries. MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and other countries. Node.js ® is an official trademark of Joyent. Red Hat Software Collections is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.
    [Show full text]
  • Download File
    Annex 2: List of tested and analyzed data sharing tools (non-exhaustive) Below are the specifications of the tools surveyed, as to February 2015, with some updates from April 2016. The tools selected in the context of EU BON are available in the main text of the publication and are described in more details. This list is also available on the EU BON Helpdesk website, where it will be regularly updated as needed. Additional lists are available through the GBIF resources page, the DataONE software tools catalogue, the BioVel BiodiversityCatalogue and the BDTracker A.1 GBIF Integrated Publishing Toolkit (IPT) Main usage, purpose, selected examples The Integrated Publishing Toolkit is a free open source software tool written in Java which is used to publish and share biodiversity data sets and metadata through the GBIF network. Designed for interoperability, it enables the publishing of content in databases or text files using open standards, namely, the Darwin Core and the Ecological Metadata Language. It also provides a 'one-click' service to convert data set metadata into a draft data paper manuscript for submission to a peer-reviewed journal. Currently, the IPT supports three core types of data: checklists, occurrence datasets and sample based data (plus datasets at metadata level only). The IPT is a community-driven tool. Core development happens at the GBIF Secretariat but the coding, documentation, and internationalization are a community effort. New versions incorporate the feedback from the people who actually use the IPT. In this way, users can help get the features they want by becoming involved. The user interface of the IPT has so far been translated into six languages: English, French, Spanish, Traditional Chinese, Brazilian Portuguese, Japanese (Robertson et al, 2014).
    [Show full text]
  • System and Organization Controls (SOC) 3 Report Over the Google Cloud Platform System Relevant to Security, Availability, and Confidentiality
    System and Organization Controls (SOC) 3 Report over the Google Cloud Platform System Relevant to Security, Availability, and Confidentiality For the Period 1 May 2020 to 30 April 2021 Google LLC 1600 Amphitheatre Parkway Mountain View, CA, 94043 650 253-0000 main Google.com Management’s Report of Its Assertions on the Effectiveness of Its Controls Over the Google Cloud Platform System Based on the Trust Services Criteria for Security, Availability, and Confidentiality We, as management of Google LLC ("Google" or "the Company") are responsible for: • Identifying the Google Cloud Platform System (System) and describing the boundaries of the System, which are presented in Attachment A • Identifying our service commitments and system requirements • Identifying the risks that would threaten the achievement of its service commitments and system requirements that are the objectives of our System, which are presented in Attachment B • Identifying, designing, implementing, operating, and monitoring effective controls over the Google Cloud Platform System (System) to mitigate risks that threaten the achievement of the service commitments and system requirements • Selecting the trust services categories that are the basis of our assertion We assert that the controls over the System were effective throughout the period 1 May 2020 to 30 April 2021, to provide reasonable assurance that the service commitments and system requirements were achieved based on the criteria relevant to security, availability, and confidentiality set forth in the AICPA’s
    [Show full text]
  • Apache Sentry
    Apache Sentry Prasad Mujumdar [email protected] [email protected] Agenda ● Various aspects of data security ● Apache Sentry for authorization ● Key concepts of Apache Sentry ● Sentry features ● Sentry architecture ● Integration with Hadoop ecosystem ● Sentry administration ● Future plans ● Demo ● Questions Who am I • Software engineer at Cloudera • Committer and PPMC member of Apache Sentry • also for Apache Hive and Apache Flume • Part of the the original team that started Sentry work Aspects of security Perimeter Access Visibility Data Authentication Authorization Audit, Lineage Encryption, what user can do data origin, usage Kerberos, LDAP/AD Masking with data Data access Access ● Provide user access to data Authorization ● Manage access policies what user can do ● Provide role based access with data Agenda ● Various aspects of data security ● Apache Sentry for authorization ● Key concepts of Apache Sentry ● Sentry features ● Sentry architecture ● Integration with Hadoop ecosystem ● Sentry administration ● Future plans ● Demo ● Questions Apache Sentry (Incubating) Unified Authorization module for Hadoop Unlocks Key RBAC Requirements Secure, fine-grained, role-based authorization Multi-tenant administration Enforce a common set of policies across multiple data access path in Hadoop. Key Capabilities of Sentry Fine-Grained Authorization Permissions on object hierarchie. Eg, Database, Table, Columns Role-Based Authorization Support for role templetes to manage authorization for a large set of users and data objects Multi Tanent Administration
    [Show full text]
  • Use Splunk with Big Data Repositories Like Spark, Solr, Hadoop and Nosql Storage
    Copyright © 2016 Splunk Inc. Use Splunk With Big Data Repositories Like Spark, Solr, Hadoop And Nosql Storage Raanan Dagan, May Long Big Data Architect, Splunk Disclaimer During the course of this presentaon, we may make forward looking statements regarding future events or the expected performance of the company. We cauJon you that such statements reflect our current expectaons and esJmates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward- looking statements made in the this presentaon are being made as of the Jme and date of its live presentaon. If reviewed aer its live presentaon, this presentaon may not contain current or accurate informaon. We do not assume any obligaon to update any forward looking statements we may make. In addiJon, any informaon about our roadmap outlines our general product direcJon and is subject to change at any Jme without noJce. It is for informaonal purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligaon either to develop the features or funcJonality described or to include any such feature or funcJonality in a future release. 2 Agenda Use Cases: Fraud With Solr, Splunk, And Splunk AnalyJcs For Hadoop Business AnalyJcs With Cassandra, Splunk Cloud, And Splunk AnalyJcs For Hadoop Document Classificaon With Spark And Splunk Network IT With Kaa And Splunk Kaa Add On Demo 3 Fraud With Solr, Splunk, And Splunk AnalyJcs For Hadoop Use Case: Fraud – Why Apache Solr Apache Solr is an open source enterprise search plaorm from the Apache Lucene API.
    [Show full text]
  • Google Cloud Platform Integration
    Solidatus FACTSHEET Google Cloud Platform Integration The Solidatus Google Cloud Platform (GCP) integration suite helps to discover data structures and lineage in GCP and automatically create and maintain Solidatus models describing these assets when they are added to GCP and when they are changed. As of January 2019, the GCP integration supports the following scenarios: • Through the Solidatus UI: – Load BigQuery dataset schemas as Solidatus objects on-demand. • Automatically using a Solidatus Agent: – Detect new BigQuery schemas and add to a Solidatus model. – Detect changes to BigQuery schemas and update a Solidatus model. – Detect new files in Google Cloud Storage (GCS) and add to a Solidatus model. – Automatically detect changes to files in GCS and update a Solidatus model. • Automatically at build time: – Extract structure and lineage from a Google Cloud Dataflow and create or update a Solidatus model. FEATURES BigQuery Loader Apache Beam (GCP Dataflow) Lineage A user can import a BigQuery table definition, directly Mapper from Google, as an object into a Solidatus model. A developer can visualise their Apache Beam job’s The import supports both nested and flat structures, pipeline in a Solidatus model. The model helps both and also includes meta data about the table and developers and analysts to see that data from sources dataset. Objects created via the BigQuery Loader is correctly mapped through transforms to their sinks, can be easily updated by a right-clicking on an providing a data lineage model of the pipeline. object in Solidatus. Updating models using this Generating the models can be ad-hoc (on-demand by feature provides the ability to visualise differences in the developer) or built into a CI/CD process.
    [Show full text]
  • Google's Mission
    & Big Data & Rocket Fuel Dr Raj Subramani, HSBC Reza Rokni, Google Cloud, Solutions Architect Adrian Poole, Google Cloud, Google’s Mission Organize the world’s information and make it universally accessible and useful Eight cloud products with ONE BILLION Users Increasing Marginal Cost of Change $ Traditional Architectures Prohibitively Expensive change Marginal cost of 18 years of Google R&D / Investment Google Cloud Native Architectures (GCP) Increasing complexity of systems and processes Containers at Google Number of running jobs Enabled Google to grow our fleet over 10x faster than we grew our ops team Core Ops Team 2004 2016 4 Google’s innovation in data Millwheel F1 Spanner TensorFlow MapReduce Dremel Flume GFS Bigtable Colossus Megastore Pub/Sub Dataflow 2002 2004 2006 2008 2010 2012 2013 2016 Proprietary + Confidential5 Google’s innovation in data Dataflow Spanner NoSQL Spanner Cloud ML Dataproc BigQuery Dataflow GCS Bigtable GCS Datastore Pub/Sub Dataflow 2002 2004 2006 2008 2010 2012 2013 2016 Proprietary + Confidential6 Now available on Google Cloud Platform Compute Storage & Databases App Engine Container Compute Storage Bigtable Spanner Cloud SQL Datastore Engine Engine Big Data Machine Learning BigQuery Pub/Sub Dataflow Dataproc Datalab Vision API Machine Speech API Translate API Learning Lesson of the last 10 years... ● Democratise ML ● Big datasets beat fancy algorithms ● Good Models ● Lots of compute Google BigQuery BigQuery is Google's fully managed, petabyte scale, low cost enterprise data warehouse for analytics. BigQuery is serverless. There is no infrastructure to manage and you don't need a database administrator, so you can focus on analyzing data to find meaningful insights using familiar SQL.
    [Show full text]
  • A Comprehensive Study of Bloated Dependencies in the Maven Ecosystem
    Noname manuscript No. (will be inserted by the editor) A Comprehensive Study of Bloated Dependencies in the Maven Ecosystem César Soto-Valero · Nicolas Harrand · Martin Monperrus · Benoit Baudry Received: date / Accepted: date Abstract Build automation tools and package managers have a profound influence on software development. They facilitate the reuse of third-party libraries, support a clear separation between the application’s code and its ex- ternal dependencies, and automate several software development tasks. How- ever, the wide adoption of these tools introduces new challenges related to dependency management. In this paper, we propose an original study of one such challenge: the emergence of bloated dependencies. Bloated dependencies are libraries that the build tool packages with the application’s compiled code but that are actually not necessary to build and run the application. This phenomenon artificially grows the size of the built binary and increases maintenance effort. We propose a tool, called DepClean, to analyze the presence of bloated dependencies in Maven artifacts. We ana- lyze 9; 639 Java artifacts hosted on Maven Central, which include a total of 723; 444 dependency relationships. Our key result is that 75:1% of the analyzed dependency relationships are bloated. In other words, it is feasible to reduce the number of dependencies of Maven artifacts up to 1=4 of its current count. We also perform a qualitative study with 30 notable open-source projects. Our results indicate that developers pay attention to their dependencies and are willing to remove bloated dependencies: 18/21 answered pull requests were accepted and merged by developers, removing 131 dependencies in total.
    [Show full text]
  • Synthesis and Development of a Big Data Architecture for the Management of Radar Measurement Data
    1 Faculty of Electrical Engineering, Mathematics & Computer Science Synthesis and Development of a Big Data architecture for the management of radar measurement data Alex Aalbertsberg Master of Science Thesis November 2018 Supervisors: dr. ir. Maurice van Keulen (University of Twente) prof. dr. ir. Mehmet Aks¸it (University of Twente) dr. Doina Bucur (University of Twente) ir. Ronny Harmanny (Thales) University of Twente P.O. Box 217 7500 AE Enschede The Netherlands Approval Internship report/Thesis of: …………………………………………………………………………………………………………Alexander P. Aalbertsberg Title: …………………………………………………………………………………………Synthesis and Development of a Big Data architecture for the management of radar measurement data Educational institution: ………………………………………………………………………………..University of Twente Internship/Graduation period:…………………………………………………………………………..2017-2018 Location/Department:.…………………………………………………………………………………435 Advanced Development, Delft Thales Supervisor:……………………………………………………………………………R. I. A. Harmanny This report (both the paper and electronic version) has been read and commented on by the supervisor of Thales Netherlands B.V. In doing so, the supervisor has reviewed the contents and considering their sensitivity, also information included therein such as floor plans, technical specifications, commercial confidential information and organizational charts that contain names. Based on this, the supervisor has decided the following: o This report is publicly available (Open). Any defence may take place publicly and the report may be included in public libraries and/or published in knowledge bases. • o This report and/or a summary thereof is publicly available to a limited extent (Thales Group Internal). tors . It will be read and reviewed exclusively by teachers and if necessary by members of the examination board or review ? committee. The content will be kept confidential and not disseminated through publication or inclusion in public libraries and/or knowledge bases.
    [Show full text]
  • Cómo Citar El Artículo Número Completo Más Información Del
    DYNA ISSN: 0012-7353 Universidad Nacional de Colombia Iván-Herrera-Herrera, Nelson; Luján-Mora, Sergio; Gómez-Torres, Estevan Ricardo Integración de herramientas para la toma de decisiones en la congestión vehicular DYNA, vol. 85, núm. 205, 2018, Abril-Junio, pp. 363-370 Universidad Nacional de Colombia DOI: https://doi.org/10.15446/dyna.v85n205.67745 Disponible en: https://www.redalyc.org/articulo.oa?id=49657889045 Cómo citar el artículo Número completo Sistema de Información Científica Redalyc Más información del artículo Red de Revistas Científicas de América Latina y el Caribe, España y Portugal Página de la revista en redalyc.org Proyecto académico sin fines de lucro, desarrollado bajo la iniciativa de acceso abierto Integration of tools for decision making in vehicular congestion• Nelson Iván-Herrera-Herreraa, Sergio Luján-Morab & Estevan Ricardo Gómez-Torres a a Facultad de Ciencias de la Ingeniería e Industrias, Universidad Tecnológica Equinoccial, Quito, Ecuador. [email protected], [email protected] b Departamento de Lenguajes y Sistemas Informáticos, Universidad de Alicante, Alicante, España. [email protected] Received: September 15th, 2017. Received in revised form: March 15th, 2018. Accepted: March 21th, 2018. Abstract The purpose of this study is to present an analysis of the use and integration of technological tools that help decision making in situations of vehicular congestion. The city of Quito-Ecuador is considered as a case study for the done work. The research is presented according to the development of an application, using Big Data tools (Apache Flume, Apache Hadoop, Apache Pig), favoring the processing of a lot of information that is required to collect, store and process.
    [Show full text]