DOCSLIB.ORG

How to Connect to Apache Drill from Denodo

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
How to Connect to Apache Drill from Denodo How to connect to Apache Drill from Denodo Revision 20200610 NOTE This document is confidential and proprietary of Denodo Technologies. No part of this document may be reproduced in any form by any means without prior written authorization of Denodo Technologies. Copyright © 2021 Denodo Technologies Proprietary and Confidential How to connect to Apache Drill from Denodo 20200610 2 of 20 CONTENTS 1 GOAL...............................................................................3 2 OVERVIEW........................................................................4 2.1 DRILL SETUP.................................................................................4 2.2 CREATING A JDBC DATA SOURCE.....................................................6 2.3 CREATING BASE VIEWS................................................................13 How to connect to Apache Drill from Denodo 20200610 3 of 20 1 GOAL This document explains how to integrate Apache Drill into the Denodo Platform using its JDBC driver. How to connect to Apache Drill from Denodo 20200610 4 of 20 2 OVERVIEW Apache Drill is a distributed and schema-free SQL query engine that can connect to data stored in multiple formats (avro, parquet, csv, json,...) and datastores and provide a common SQL interface. Apache Drill As Drill was kicked off by MapR (many of the committers and contributors to Drill have come from MapR), and it is included in the MapR Data Platform this document will take MapR as the reference environment. 2.1 DRILL SETUP Check which storage plugins are configured by visiting the Drill Web Console at https://<HOST>:8047/storage. For this guide we have enabled the storage plugins that allows MapR Filesystem, MapR Database and Hive to serve as data sources for Drill: How to connect to Apache Drill from Denodo 20200610 5 of 20 Drill Storage Plugins configuration You can find here the supported data sources and formats in Drill on MapR: https://mapr.com/docs/61/Drill/drill_storage_and_format_plugin_support_matrix.html How to connect to Apache Drill from Denodo 20200610 6 of 20 2.2 CREATING A JDBC DATA SOURCE Download the JDBC driver Before being able to connect to your Apache Drill instance, you need to download the JDBC driver. There are two different options that you could use as JDBC driver: 1. Drill JDBC driver provided by MapR a. This driver supports MapR-SASL and Plain authentication mechanism b. Note: The current version MapRDrill_jdbc_v1.6.0.1001 of this driver does not work with Java 9 and newer due to a bug in the driver. Although the bug was fixed in Drill version 1.15, the MapR driver was not updated yet to include the latest fixes. If you are on Denodo 8.0 (which uses Java 11) and while the driver was not updated yet, you should use the next option. See https://issues.apache.org/jira/browse/DRILL-6349 2. The JDBC driver (drill-jdbc-all.jar file) included with Drill distribution from http://apache.osuosl.org/drill a. This driver supports Kerberos and Plain authentication mechanisms, but does not support the MapR-SASL authentication mechanism. There is no need to install both of them, but in case you face issues with one, you could try the other one. Modify and Install the JDBC driver Before we can go ahead and install the JDBC driver, we need to modify them as the drivers may include dependencies that conflict with the ones included in the Denodo Platform. If you do not follow these steps, you will receive different kinds of error messages when testing the connection and will not be able to connect. So, depending on the driver you use, please consider these notes: 1. Drill JDBC driver provided by MapR ! Important note The slf4j dependency should be deleted from the MapRDrillJDBC41- 1.<version> directory, otherwise Denodo will fail: How to connect to Apache Drill from Denodo 20200610 7 of 20 The jar file that you need to delete from that directory is slf4j-api- <version>.jar 2. Apache Drill driver ! Important note The org.slf4j, javax.xml, org.w3c and org.xml dependencies need to be deleted from the drill-jdbc-all.jar file, otherwise Denodo will fail with errors like these: How to connect to Apache Drill from Denodo 20200610 8 of 20 For deleting these dependencies, you should open the drill-jdbc-all.jar file as an archive and delete the following folders from it: ● org/slf4j ● org/w3c ● org/xml ● javax/xml Starting from Denodo 8.0, the preferred way of installing the JDBC driver is to upload the JDBC driver via the Administration Tool. This can be done under File > Extension Management > Libraries > Import. Select jdbc_other and provide a custom name for using it later. Import all the necessary jars for connecting to Drill. How to connect to Apache Drill from Denodo 20200610 9 of 20 In previous versions, the recommended way is to copy the driver jars to the folder <DENODO_HOME>/lib-external/jdbc-drivers/database name - version Later, you need to provide the 'database name - version' in the driver class path. Create the JDBC data source From the Web Design Studio or from the Virtual DataPort Administration tool, create a new JDBC data source by selecting File > New > Data source > JDBC. This will open a wizard to create a connection with a JDBC driver. Fill all the required fields: 1. Database adapter: Generic 2. Driver class: Depending on your chosen driver, you need to specify either com.mapr.drill.jdbc41.Driver for the MapR driver or org.apache.drill.jdbc.Driver for the Apache Drill driver. 3. Driver class path: Select the driver class path that you have uploaded previously or, for previous versions, copy the driver jars to the folder <DENODO_HOME>/lib-external/jdbc-drivers/database name - version and fill the wizard field with the value ‘database name - version’ ! Important The MapR native library is included in the MapR Drill driver and can be loaded only once. Therefore, if you plan to access to other MapR sources with Denodo, like: ● MapR FileSystem with HDFS Custom Wrapper ● MapR Database with HBase Custom Wrapper ● MapR Event Store with Kafka Custom Wrapper you have to use the same classpath to configure all the custom wrappers and the Drill JDBC driver. With this configuration Denodo can reuse the same classloader and load the native library only once. 4. Database URI. There are two possibilities: i. Recommended: Connect to the ZooKeeper. ZooKeeper returns to the client the available Drillbits in the cluster to which the query can be submitted. The URI is of the form: How to connect to Apache Drill from Denodo 20200610 10 of 20 jdbc:drill:zk=<zk.connect>/<drill_directory_in_zookeepe r>/<cluster_ID> Being the configuration file <DRILL-HOME>/conf/drill- distrib.conf: the JDBC URI will be: jdbc:drill:zk=maprdemo:5181/drill/demo_mapr_com- drillbits ii. Connect directly to a specific Drillbit, though this is generally not recommended due to the hardcoding that can happen in client. jdbc:drill:drillbit=<host>:<port> c. Delimiter. The default setting for identifier quotes in Drill is backticks. Therefore, in the Read & Write tab of the data source configuration, set backticks (`) as the Delimiter Identifier. How to connect to Apache Drill from Denodo 20200610 11 of 20 Drill Delimiter configuration d. Ping query. Enable Test connections and configure Select 1 in Connection Pool Configuration > Ping query Drill Ping Query configuration e. Source Configuration. Drill does not support Prepared-statement dynamic parameters. Therefore, set Allow Literal as Parameters to false in the Source Configuration tab of the data source. Drill Source Configuration Otherwise, you will receive the following errors when adding a WHERE clause to filter the data returned by a base view: How to connect to Apache Drill from Denodo 20200610 12 of 20 ● MapR Driver: - [*]SQLE [MapR][JDBC](10940) Invalid parameter index: 1 - [MapR][DrillJDBCDriver](500980) Encountered error while creating prepared statement. Details: PLAN ERROR: Cannot convert RexNode to equivalent Drill expression. RexNode Class: org.apache.calcite.rex.RexDynamicParam, RexNode Digest: ?0 ● Apache driver: Received exception with message 'Prepared-statement dynamic parameters are not supported.' Drill data source with MapR JDBC Driver Note: Due to the aforementioned bug in the driver MapRDrill_jdbc_v1.6.0.1001 this shows an example configuration working for Denodo 7.0, but not for Denodo 8.0. See https://issues.apache.org/jira/browse/DRILL-6349. If you plan to use the MapR driver, make sure that the latest MapR includes this drill fix. As of time of writing, there is no updated MapR driver working with Java 9 or newer versions. How to connect to Apache Drill from Denodo 20200610 13 of 20 Drill data source with Apache Drill JDBC Driver 2.3 CREATING BASE VIEWS Once we have configured our data source pointing to Drill, we can create different views for accessing the desired entities. Denodo will display a tree with the schemas available in Drill. In this example it will include the Hive and MapR Database schemas, as well as the workspaces configured in the file system storage plugin: How to connect to Apache Drill from Denodo 20200610 14 of 20 Drill schemas ● For Hive databases you can inspect the full schema, its tables and their fields, as the metadata is available in the Hive metastore. Then, you have to select the tables you want to import and click the Create selected button. ● For MapR Database you can inspect the schema partially. Wide-column NoSQL databases can be schema-less by design; every row has its own set of column name-value pairs in a given column family, and the column value can be of any data type. A MAP complex type represents this variable column name-value structure. As shown in the image above, day, hour and total are column families and are represented as MAP. Then, you have to select the tables you want to import and click the Create selected button.
Load more

Recommended publications
  • A Comprehensive Study of Bloated Dependencies in the Maven Ecosystem
    Noname manuscript No. (will be inserted by the editor) A Comprehensive Study of Bloated Dependencies in the Maven Ecosystem César Soto-Valero · Nicolas Harrand · Martin Monperrus · Benoit Baudry Received: date / Accepted: date Abstract Build automation tools and package managers have a profound influence on software development. They facilitate the reuse of third-party libraries, support a clear separation between the application’s code and its ex- ternal dependencies, and automate several software development tasks. How- ever, the wide adoption of these tools introduces new challenges related to dependency management. In this paper, we propose an original study of one such challenge: the emergence of bloated dependencies. Bloated dependencies are libraries that the build tool packages with the application’s compiled code but that are actually not necessary to build and run the application. This phenomenon artificially grows the size of the built binary and increases maintenance effort. We propose a tool, called DepClean, to analyze the presence of bloated dependencies in Maven artifacts. We ana- lyze 9; 639 Java artifacts hosted on Maven Central, which include a total of 723; 444 dependency relationships. Our key result is that 75:1% of the analyzed dependency relationships are bloated. In other words, it is feasible to reduce the number of dependencies of Maven artifacts up to 1=4 of its current count. We also perform a qualitative study with 30 notable open-source projects. Our results indicate that developers pay attention to their dependencies and are willing to remove bloated dependencies: 18/21 answered pull requests were accepted and merged by developers, removing 131 dependencies in total.
    [Show full text]
  • Performance Tuning Apache Drill on Hadoop Clusters with Evolutionary Algorithms
    Performance tuning Apache Drill on Hadoop Clusters with Evolutionary Algorithms Proposing the algorithm SIILCK (Society Inspired Incremental Learning through Collective Knowledge) Roger Bløtekjær Thesis submitted for the degree of Master of science in Informatikk: språkteknologi 30 credits Institutt for informatikk Faculty of mathematics and natural sciences UNIVERSITY OF OSLO Spring 2018 Performance tuning Apache Drill on Hadoop Clusters with Evolutionary Algorithms Proposing the algorithm SIILCK (Society Inspired Incremental Learning through Collective Knowledge) Roger Bløtekjær c 2018 Roger Bløtekjær Performance tuning Apache Drill on Hadoop Clusters with Evolutionary Algorithms http://www.duo.uio.no/ Printed: Reprosentralen, University of Oslo 0.1 Abstract 0.1.1 Research question How can we make a self optimizing distributed Apache Drill cluster, for high performance data readings across several file formats and database architec- tures? 0.1.2 Overview Apache Drill enables the user to perform schema-free querying of distributed data, and ANSI SQL programming on top of NoSQL datatypes like JSON or XML - typically in a Hadoop cluster. As it is with the core Hadoop stack, Drill is also highly customizable, with plenty of performance tuning parame- ters to ensure optimal efficiency. Tweaking these parameters however, requires deep domain knowledge and technical insight, and even then the optimal con- figuration may not be evident. Businesses will want to use Apache Drill in a Hadoop cluster, without the hassle of configuring it, for the most cost-effective implementation. This can be done by solving the following problems: • How to apply evolutionary algorithms to automatically tune a distributed Apache Drill configuration, regardless of cluster environment.
    [Show full text]
  • Apache Calcite: a Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources
    Apache Calcite: A Foundational Framework for Optimized Query Processing Over Heterogeneous Data Sources Edmon Begoli Jesús Camacho-Rodríguez Julian Hyde Oak Ridge National Laboratory Hortonworks Inc. Hortonworks Inc. (ORNL) Santa Clara, California, USA Santa Clara, California, USA Oak Ridge, Tennessee, USA [email protected] [email protected] [email protected] Michael J. Mior Daniel Lemire David R. Cheriton School of University of Quebec (TELUQ) Computer Science Montreal, Quebec, Canada University of Waterloo [email protected] Waterloo, Ontario, Canada [email protected] ABSTRACT argued that specialized engines can offer more cost-effective per- Apache Calcite is a foundational software framework that provides formance and that they would bring the end of the “one size fits query processing, optimization, and query language support to all” paradigm. Their vision seems today more relevant than ever. many popular open-source data processing systems such as Apache Indeed, many specialized open-source data systems have since be- Hive, Apache Storm, Apache Flink, Druid, and MapD. Calcite’s ar- come popular such as Storm [50] and Flink [16] (stream processing), chitecture consists of a modular and extensible query optimizer Elasticsearch [15] (text search), Apache Spark [47], Druid [14], etc. with hundreds of built-in optimization rules, a query processor As organizations have invested in data processing systems tai- capable of processing a variety of query languages, an adapter ar- lored towards their specific needs, two overarching problems have chitecture designed for extensibility, and support for heterogeneous arisen: data models and stores (relational, semi-structured, streaming, and • The developers of such specialized systems have encoun- geospatial). This flexible, embeddable, and extensible architecture tered related problems, such as query optimization [4, 25] is what makes Calcite an attractive choice for adoption in big- or the need to support query languages such as SQL and data frameworks.
    [Show full text]
  • Umltographdb: Mapping Conceptual Schemas to Graph Databases
    Citation for published version Daniel, G., Sunyé, G. & Cabot, J. (2016). UMLtoGraphDB: Mapping Conceptual Schemas to Graph Databases. Lecture Notes in Computer Science, 9974(), 430-444. DOI https://doi.org/10.1007/978-3-319-46397-1_33 Document Version This is the Submitted Manuscript version. The version in the Universitat Oberta de Catalunya institutional repository, O2 may differ from the final published version. https://doi.org/10.1007/978-3-319-61482-3_16 Copyright and Reuse This manuscript version is made available under the terms of the Creative Commons Attribution Non Commercial No Derivatives licence (CC-BY-NC-ND) http://creativecommons.org/licenses/by-nc-nd/3.0/es/, which permits ​ others to download it and share it with others as long as they credit you, but they can’t change it in any way or use them commercially. Enquiries If you believe this document infringes copyright, please contact the Research Team at: [email protected] Universitat Oberta de Catalunya Research archive UMLtoGraphDB: Mapping Conceptual Schemas to Graph Databases Gwendal Daniel1, Gerson Sunyé1, and Jordi Cabot2;3 1 AtlanMod Team Inria, Mines Nantes & Lina {gwendal.daniel,gerson.sunye}@inria.fr 2 ICREA [email protected] 3 Internet Interdisciplinary Institute, UOC Abstract. The need to store and manipulate large volume of (unstructured) data has led to the development of several NoSQL databases for better scalability. Graph databases are a particular kind of NoSQL databases that have proven their efficiency to store and query highly interconnected data, and have become a promising solution for multiple applications. While the mapping of conceptual schemas to relational databases is a well-studied field of research, there are only few solutions that target conceptual modeling for NoSQL databases and even less focusing on graph databases.
    [Show full text]
  • Technology Overview
    Big Data Technology Overview Term Description See Also Big Data - the 5 Vs Everyone Must Volume, velocity and variety. And some expand the definition further to include veracity 3 Vs Know and value as well. 5 Vs of Big Data From Wikipedia, “Agile software development is a group of software development methods based on iterative and incremental development, where requirements and solutions evolve through collaboration between self-organizing, cross-functional teams. Agile The Agile Manifesto It promotes adaptive planning, evolutionary development and delivery, a time-boxed iterative approach, and encourages rapid and flexible response to change. It is a conceptual framework that promotes foreseen tight iterations throughout the development cycle.” A data serialization system. From Wikepedia, Avro Apache Avro “It is a remote procedure call and serialization framework developed within Apache's Hadoop project. It uses JSON for defining data types and protocols, and serializes data in a compact binary format.” BigInsights Enterprise Edition provides a spreadsheet-like data analysis tool to help Big Insights IBM Infosphere Biginsights organizations store, manage, and analyze big data. A scalable multi-master database with no single points of failure. Cassandra Apache Cassandra It provides scalability and high availability without compromising performance. Cloudera Inc. is an American-based software company that provides Apache Hadoop- Cloudera Cloudera based software, support and services, and training to business customers. Wikipedia - Data Science Data science The study of the generalizable extraction of knowledge from data IBM - Data Scientist Coursera Big Data Technology Overview Term Description See Also Distributed system developed at Google for interactively querying large datasets. Dremel Dremel It empowers business analysts and makes it easy for business users to access the data Google Research rather than having to rely on data engineers.
    [Show full text]
  • Code Smell Prediction Employing Machine Learning Meets Emerging Java Language Constructs"
    Appendix to the paper "Code smell prediction employing machine learning meets emerging Java language constructs" Hanna Grodzicka, Michał Kawa, Zofia Łakomiak, Arkadiusz Ziobrowski, Lech Madeyski (B) The Appendix includes two tables containing the dataset used in the paper "Code smell prediction employing machine learning meets emerging Java lan- guage constructs". The first table contains information about 792 projects selected for R package reproducer [Madeyski and Kitchenham(2019)]. Projects were the base dataset for cre- ating the dataset used in the study (Table I). The second table contains information about 281 projects filtered by Java version from build tool Maven (Table II) which were directly used in the paper. TABLE I: Base projects used to create the new dataset # Orgasation Project name GitHub link Commit hash Build tool Java version 1 adobe aem-core-wcm- www.github.com/adobe/ 1d1f1d70844c9e07cd694f028e87f85d926aba94 other or lack of unknown components aem-core-wcm-components 2 adobe S3Mock www.github.com/adobe/ 5aa299c2b6d0f0fd00f8d03fda560502270afb82 MAVEN 8 S3Mock 3 alexa alexa-skills- www.github.com/alexa/ bf1e9ccc50d1f3f8408f887f70197ee288fd4bd9 MAVEN 8 kit-sdk-for- alexa-skills-kit-sdk- java for-java 4 alibaba ARouter www.github.com/alibaba/ 93b328569bbdbf75e4aa87f0ecf48c69600591b2 GRADLE unknown ARouter 5 alibaba atlas www.github.com/alibaba/ e8c7b3f1ff14b2a1df64321c6992b796cae7d732 GRADLE unknown atlas 6 alibaba canal www.github.com/alibaba/ 08167c95c767fd3c9879584c0230820a8476a7a7 MAVEN 7 canal 7 alibaba cobar www.github.com/alibaba/
    [Show full text]
  • Apache Drill
    Apache Drill 0 Apache Drill About the Tutorial Apache Drill is first schema-free SQL engine. Unlike Hive, it does not use MR job internally and compared to most distributed query engines, it does not depend on Hadoop. Apache Drill is observed as the upgraded version of Apache Sqoop. Drill is inspired by Google Dremel concept called BigQuery and later became an open source Apache project. This tutorial will explore the fundamentals of Drill, setup and then walk through with query operations using JSON, querying data with Big Data technologies and finally conclude with some real-time applications. Audience This tutorial has been prepared for professionals aspiring to make a career in Big Data Analytics. This tutorial will give you enough understanding on Drill process, about how to install it on your system and its other operations. Prerequisites Before proceeding with this tutorial, you must have a good understanding of Java, JSON and any of the Linux operating system. Copyright & Disclaimer © Copyright 2016 by Tutorials Point (I) Pvt. Ltd. All the content and graphics published in this e-book are the property of Tutorials Point (I) Pvt. Ltd. The user of this e-book is prohibited to reuse, retain, copy, distribute or republish any contents or a part of contents of this e-book in any manner without written consent of the publisher. We strive to update the contents of our website and tutorials as timely and as precisely as possible, however, the contents may contain inaccuracies or errors. Tutorials Point (I) Pvt. Ltd. provides no guarantee regarding the accuracy, timeliness or completeness of our website or its contents including this tutorial.
    [Show full text]
  • Technical Expertise
    www.ultantechnologies.com Technical Expertise Subject: Ultan Technologies Technical Expertise Author: Cathal Brady Date Published: 01/03/2016 Version Number: Version 1 www.ultantechnologies.com Contents 1 INTRODUCTION ..................................................................................................................... 1 2 .NET ....................................................................................................................................... 1 3 DATABASES ........................................................................................................................... 2 4 BIG DATA ............................................................................................................................... 2 5 JAVA ...................................................................................................................................... 3 6 PHP, RUBY, PYTHON .............................................................................................................. 3 7 FRONT END............................................................................................................................ 4 8 3RD PARTY INTEGRATION, APIs, PLUGINS ............................................................................. 4 9 CONTINUOUS INTEGRATION / BUILD AUTOMATION / VERSION CONTROL .......................... 4 10 MOBILE DEVELOPMENT ........................................................................................................ 5 11 CRM CUSTOMISATION .........................................................................................................
    [Show full text]
  • Apache Calcite for Enabling SQL Access to Nosql Data Systems Such As Apache Geode Christian Tzolov Whoami Christian Tzolov
    Apache Calcite for Enabling SQL Access to NoSQL Data Systems such as Apache Geode Christian Tzolov Whoami Christian Tzolov Engineer at Pivotal, Big-Data, Hadoop, Spring Cloud Dataflow, Apache Geode, Apache HAWQ, Apache Committer, Apache Crunch PMC member [email protected] blog.tzolov.net twitter: @christzolov https://nl.linkedin.com/in/tzolov Disclaimer This talk expresses my personal opinions. It is not read or approved by Pivotal and does not necessarily reflect the views and opinions of Pivotal nor does it constitute any official communication of Pivotal. Pivotal does not support any of the code shared here. 2 Big Data Landscape 2016 • Volume • Velocity • Varity • Scalability • Latency • Consistency vs. Availability (CAP) 3 Data Access • {Old | New} SQL • Custom APIs – Key / Value – Fluent APIs – REST APIs • {My} Query Language Unified Data Access? At What Cost? 4 SQL? • Apache Apex • SQL-Gremlin • Apache Drill … • Apache Flink • Apache Geode • Apache Hive • Apache Kylin • Apache Phoenix • Apache Samza • Apache Storm • Cascading • Qubole Quark 5 Geode Adapter - Overview SQL/JDBC/ODBC Parse SQL, converts into relational expression and Apache Calcite optimizes Push down the relational expressions supported by Geode Spring Data API for Enumerable OQL and falls back to the Calcite interacting with Geode Adapter Enumerable Adapter for the rest Convert SQL relational Spring Data Geode Adapter expressions into OQL queries Geode (Geode Client) Geode API and OQL Geode Server Geode Server Geode Server Data Data Data SQL Relational Expressions
    [Show full text]
  • A Big Data Revolution Cloud Computing, 22 Aerospike, 91, 217 Competing Definitions, 21 Aerospike Query Language (AQL), 218 Industrial Revolution, 22 AJAX
    Index A Big Data revolution cloud computing, 22 Aerospike, 91, 217 competing definitions, 21 Aerospike query language (AQL), 218 industrial revolution, 22 AJAX. See Asynchronous JavaScript and IoT, 22 XML (AJAX) social networks and Alternative persistence model, 92 smartphones, 22 Amazon Binary JSON (BSON), 157 ACID RDBMS, 46 Blockchain, 212 Dynamo, 14, 45–46 Bloom filters, 161 DynamoDB, 219 Boolean bit logic, 214 hashing, 47–48 B-tree index structure, 158–159 key–value stores, 51 Business intelligence (BI) practices, 193 NWR notation, 49–50 SOA, 45 Amazon Web Services (AWS), 15 C Apache Cassandra , 218. See also Cassandra Cache-less architecture, 92 Apache HBase, 220 CAP theorem Apache Kudu, 211. See also Hbase partition tolerance, 44 Append only file (AOF), 94 RAC solution, 44 Asynchronous JavaScript and Cascading Style Sheets (CSS), 54 XML (AJAX), 15 Cassandra, 211 Atomic, Consistent, Independent, and Durable cluster node, 120 (ACID) transactions, 9–10, 128 consistent hashing, 120–121 AWS. See Amazon Web Services (AWS) data model, 153, 155 gossip, 119 node adding, 122 B order-preserving Berkeley analytics data stack and spark partitioners, 124 AMPlab, 99 replicas, 124–125 BDAS, 100 snitches, 126 DAG, 101 virtual nodes, 122–123 Hadoop, 99–100 Cassandra consistency JDBC-compliant database, 101 hinted handoff, 136–137 MapReduce, 99 LWT (see Lightweight MLBase component, 100 transactions (LWT)) RDD, 101 read consistency, 135 spark architecture, 101 read repair, 136–137 spark processing replication factor, 134 elements, 102 timestamps and granularity, 137 spark SQL, 100 vector clocks, 138–140 spark streaming, 100 write consistency, 134–135 229 ■ INDEX Cassandra Query Language (CQL), 218 D column structures, 175 cqlsh program, 175 DAG.
    [Show full text]
  • Bright Cluster Manager 8.0 Big Data Deployment Manual
    Bright Cluster Manager 8.0 Big Data Deployment Manual Revision: 97e18b7 Date: Wed Sep 29 2021 ©2017 Bright Computing, Inc. All Rights Reserved. This manual or parts thereof may not be reproduced in any form unless permitted by contract or by written permission of Bright Computing, Inc. Trademarks Linux is a registered trademark of Linus Torvalds. PathScale is a registered trademark of Cray, Inc. Red Hat and all Red Hat-based trademarks are trademarks or registered trademarks of Red Hat, Inc. SUSE is a registered trademark of Novell, Inc. PGI is a registered trademark of NVIDIA Corporation. FLEXlm is a registered trademark of Flexera Software, Inc. ScaleMP is a registered trademark of ScaleMP, Inc. All other trademarks are the property of their respective owners. Rights and Restrictions All statements, specifications, recommendations, and technical information contained herein are current or planned as of the date of publication of this document. They are reliable as of the time of this writing and are presented without warranty of any kind, expressed or implied. Bright Computing, Inc. shall not be liable for technical or editorial errors or omissions which may occur in this document. Bright Computing, Inc. shall not be liable for any damages resulting from the use of this document. Limitation of Liability and Damages Pertaining to Bright Computing, Inc. The Bright Cluster Manager product principally consists of free software that is licensed by the Linux authors free of charge. Bright Computing, Inc. shall have no liability nor will Bright Computing, Inc. provide any warranty for the Bright Cluster Manager to the extent that is permitted by law.
    [Show full text]
  • Modern ETL Tools for Cloud and Big Data
    Modern ETL Tools for Cloud and Big Data Ken Beutler, Principal Product Manager, Progress Michael Rainey, Technical Advisor, Gluent Inc. Agenda ▪ Landscape ▪ Cloud ETL Tools ▪ Big Data ETL Tools ▪ Best Practices ▪ QnA © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 2 Landscape © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 3 Data Warehousing Has Transformed Enterprise BI and Analytics © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 4 ETL Tools Are At The Heart Of EDW © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 5 Rise Of The Modern EDW Solutions Cloud Data Warehouse Big Data Warehouse © 2018 Progress Software Corporation and/or its subsidiaries or affiliates. All rights reserved. 6 Enterprises Are Increasingly Adopting Cloud and Big Data Cloud Big Data NoSQL Amazon S3 Amazon Web Services 24% MongoDB 27% 44% Hadoop Hive 22% Cassandra Microsoft Azure 39% Spark SQL 11% 15% Redis 7% VMware 22% Hortonworks 9% Amazon EMR 9% Oracle NoSQL 7% Google Cloud 18% Apache Solr 8% Google Cloud… 6% Salesforce 14% Cloudera CDH 7% HBase 6% Oracle 10% Oracle BDA 7% DynamoDB 5% Apache Sqoop 7% Couchbase 4% IBM 8% 7% IBM BigInsights Google Cloud… 3% 6% SAP HANA 6% SAP Cloud Platform Big Data… DocumentDB Cloudera Impala 2% Rackspace 6% 5% MapR 6% SimpleDB 2% RedHat OpenStack 4% Azure HDInsights 6% MarkLogic 1% DataStax Enterprise RedHat OpenShift 3% Apache Storm 5% 1% Google Cloud DataProc 5% Aerospike 1% Digital Ocean 2% Apache Drill 4% Riak 0% CenturyLink 1% Apache Phoenix 3% Scylla 0% Linode 0% Presto 2% ArangoDB 0% Pivotal HD 2% Other (please… 2% Other (please specify): 6% GemFireXD 1% None 32% None 13% Other (please specify): 2% Not sure 10% Not sure 3% None 32% Not sure 12% Source: Progress DataDirect’s Data Connectivity Outlook Survey 2018 © 2018 Progress Software Corporation and/or its subsidiaries or affiliates.
    [Show full text]