CDP DATA CENTER 7.1 Laurent Edel : Solution Engineer Jacques Marchand : Solution Engineer Mael Ropars : Principal Solution Engineer

30 Juin 2020 SPEAKERS

•

• CDP DATA CENTER OVERVIEW • DETAILS ABOUT MAJOR COMPONENTS • PATH TO CDP DC && SMART MIGRATION • Q/A

CDP Cloud Public CDP On-Prem (platform-as-a-service) (installable software)

CDP Data Center (installable software) NEW CDP Data Center features include: Cloudera Manager • High-performance SQL analytics • Real-time stream processing, analytics, and management • Fine-grained security, enterprise metadata, and scalable data lineage • Support for object storage (tech preview) • Single pane of glass for management - multi-cluster support

Enterprise analytics and data management platform, built for hybrid cloud, optimized for bare metal and ready for

private cloud

Cloudera Runtime

Deprecate competitive Merge overlapping Keep complementary Upgrade shared technologies technologies technologies technologies

CDP Data Center 7.1(May) 2020

• Cloudera Manager 7.1 • HBase 2.2 • Key HSM 7.1 • Kafka Schema Registry 0.8 • Hadoop 3.1 • Phoenix 5.0 • Knox 1.3 • Streams Messaging Mgr 1.0 • Spark 2.4 / Spark 3(b2) • Kudu 1.12 • Livy 0.7 • Streams Replication Mgr 2.1 • Hive 3.1 • Sqoop 1.4.7 • Navigator Encrypt 7.1 • Ozone (Beta) 0.6 • Impala 3.4 • Parquet 1.10 • Ranger KMS 7.1 • Kafka Connect 2.4 • Oozie 5.1 • Avro 1.8 • Zeppelin • Cruise Control 2.0 • Hue 4.5 • ORC 1.5 • Hive Warehouse Connector 1.0 • Tez 0.9 • Ranger 2.0 • Zookeeper 3.5 • Kafka 2.4 • Key Trustee Server 7 • Atlas 2.0 • Solr 8.4

• RHEL/CENTOS/OEL 7.7 • MySQL 5.7 • Upgrades from CDP DC 7.0 • Postgres 10 • Oracle DB 12 (Fresh Install Only) • Upgrades from CDH 5.13-5.16

• JDK 8 • PostgreSQL 10 • Upgrades from HDP 2.6.5 • JDK 11 Runtime • Maria DB 10.2

• CDP DATA CENTER OVERVIEW • DETAILS ABOUT MAJOR COMPONENTS • PATH TO CDP DC && SMART MIGRATION • Q/A

Collect Report Predict Impala Spark Hive Zeppelin 02 Kudu 04

01 03 05 Enrich Serve Spark Hbase Hive Phoenix SolR

SECURITY | GOVERNANCE | LINEAGE | MANAGEMENT | AUTOMATION

Collect Report Predict Impala Spark Hive Zeppelin 02 Kudu 04

01 03 05 Enrich Serve Spark Hbase Hive Phoenix SolR

SECURITY | GOVERNANCE | LINEAGE | MANAGEMENT | AUTOMATION

• Kafka 2.4 • Ranger & Atlas Integration

• Support of Kafka Connect, Kafka Streams • Cruise Control for load balancing

• Create multiple Kafka compute clusters using shared Security Data Lake with Ranger & Atlas

Schema Registry Streams Messaging Manager (SMM) Streams Replication Manager (SRM) New Kafka Schema Governance New Kafka Monitoring Service New Kafka Replication Engine powered by MirrorMaker2

Collect Report Predict Impala Spark Hive Zeppelin 02 Kudu 04

01 03 05 Enrich Serve Spark Hbase Hive Phoenix SolR

SECURITY | GOVERNANCE | LINEAGE | MANAGEMENT | AUTOMATION

• Spark 2.4

• Integration with Ranger for Fine Grained Authorizations

• Coming soon: Spark 3 ! • Better performance • Enhanced support for Deep Learning • New modules • MLLib replaced with SparkML • Tech Preview available

Collect Report Predict Impala Spark Hive Zeppelin 02 Kudu 04

01 03 05 Enrich Serve Spark Hbase Hive Phoenix SolR

SECURITY | GOVERNANCE | LINEAGE | MANAGEMENT | AUTOMATION

Apache Hive 3 • Comprehensive ANSI SQL 2016 coverage • GDPR: new ACID v2 as fast as regular tables, transactions, UPDATE/DELETE/MERGE • Cloud-ready: optimized for S3/WASB/GCP • Support for JDBC/Kafka/Druid out of the box • EDW oﬄoad: – “DBA” tooling: surrogate keys, materialized views, constraints – information schema • Performance: – workload management – query result cache

Apache Impala Apache Kudu • Leading MPP SQL Engine for DW - • Leading columnar storage engine for fast optimized for Parquet/Kudu analytics on fast data • Ideal for Data Mart Implementations that • Ideal for Low latency time series data require Interactive/Ad-hoc BI ingest and analytics (with Impala SQL • 1000+ enterprise customers - many engine) running on 10s of PBs and 100s of nodes • Strength of fast ingest with single rows like • Certiﬁed with leading BI tools with broad HBASE and allows large scans like HDFS SQL coverage • ACID (insert/update/delete) semantics • Latest release adds improvements for with single rows resiliency, concurrency, and metadata

Global view on Deep Dive Query analysis analytic processing

Collect Report Predict Impala Spark Hive Zeppelin 02 Kudu 04

01 03 05 Enrich Serve Spark Hbase Hive Phoenix SolR

SECURITY | GOVERNANCE | LINEAGE | MANAGEMENT | AUTOMATION

HBASE PHOENIX Flexible, scale-out, no-sql database RDBMS-like, scale-out database

Put put = new Put(Bytes.toBytes(rowKey)); stmt.executeUpdate(“UPSERT INTO TABLE_NAME put.addColumn(COLUMN_FAMILY_NAME, COLUMN_NAME, VALUES(rowKey, GREETINGS) "); Bytes.toBytes(GREETINGS)); stmt.execute(); table.put(put);

• Maximally ﬂexible & customizable • Programmatic ANSI SQL support • SQL only for data remediation • RDBMS-like data architecture • All advanced functionality available • Auto-applies performance best • New async client practices • JDK8/G1GC • Can co-exist with HBase apps • Off-Heap read path • API clean-up, HBCK2

© 2020 Cloudera, Inc. All rights reserved. 22 CDP SEARCH Scalable and Robust Index Storage with SOLR 8.4 Querying API Indexing API Solr Cloud ● Scalable, cost-eﬃcient index storage Distributed processing coordinator ● High availability, Integrated security with Atlas/Ranger Solr

Extraction Mapping ● Shared data store with other processing tools (Spark, Impala..) Indexing engine (Lucene)

● Search AND process data in one platform Shared Data Storage

Collect Report Predict Impala Spark Hive Zeppelin 02 Kudu 04

01 03 05 Enrich Serve Spark Hbase Hive Phoenix SolR

SECURITY | GOVERNANCE | LINEAGE | MANAGEMENT | AUTOMATION

CLOUDERA DATA SCIENCE APACHE ZEPPELIN WORKBENCH

Collect Report Predict Impala Spark Hive Zeppelin 02 Kudu 04

01 03 05 Enrich Serve Spark Hbase Hive Phoenix SolR

SECURITY | GOVERNANCE | LINEAGE | MANAGEMENT | AUTOMATION

• Management of multiple clusters • Knox,Ranger,Atlas,Hive-on-Tez,DAS • Cluster-level conﬁguration history • Improved global search • Resume errors in enabling Kerberos • Scalability improvements • Improved alerts conﬁguration • Upgrade Support • Support for Private Cloud (Beta)

• Data Catalog: a comprehensive catalog of all data sets, spanning on-premises, cloud object stores, structured, unstructured, and semi-structured

• Schema: automatic capture and storage of any and all schema and metadata definitions as they are used and created by platform workloads

• Replication: deliver data as well as data policies there where the enterprise needs to work, with complete consistency and security

• Security: role-based access control applied consistently across the platform. Includes full stack encryption and key management

• Governance: enterprise-grade auditing, lineage, and governance capabilities applied across the platform with rich extensibility for partner integrations

Identity & Perimeter Access Visibility Data Protection Validate users in Deﬁning what users and Reporting on where data Shielding data in the enterprise directory applications can do with came from and how it’s cluster from unauthorized data being used visibility

Technical Concepts: Technical Concepts: Technical Concepts: Authentication Permissions Auditing Technical Concepts: User/group mapping Authorization Lineage Encryption, Key Management

SSL/TLS, HDFS TDE, Kerberos, Apache Ranger Apache Atlas Apache Ranger Apache Knox (KMS, Masking, Filtering)

• Lineage – What data do I consume? – What consumes my Data? • Who uses my data? – Audit who accessed what – Track access events from Apache Ranger – Metadata audit and versioning from Apache Atlas

• Maintain one set of data, control access centrally with ﬁne grained policies down to the column and the row level.

• Anonymize PII with Dynamic column masking

• Customize views for users with Dynamic row ﬁltering

• Manage user access with Role-based Access Control

• Unify policies across many data sets with Attribute-based Access Control

• Ozone is the next generation of HDFS – Based on HDFS architecture, but with some fundamental shifts – Preserve and reuse good parts of HDFS – Addresses HDFS scale limits and small ﬁle problem • Uses an object store architecture to achieve scale. • Provides native Hadoop File System API as well as a native S3 API

Migrate to Public Cloud Migrate to CDP DC Upgrade to CDP DC

Copy data and metadata to a public Build a new CDP Datacenter cluster Upgrade from classic cluster to cloud; implement new, or migrate on-premises; copy data and CDP Datacenter in-place on the existing workloads on CDP Public metadata from existing classic same hardware infrastructure. Cloud. cluster; and migrate existing workloads.

1. PLAN 2. LAUNCH 3. PRODUCTION

PATH2CDP UPGRADE2CDP CONSUME CDP

● ● ● ● ● ● ● ● ● ● ● ●

● ● ● ● ● ● ● CDP Data Center

1+ Weeks 3 + Weeks Custom Estimate

AWS Fundamentals for CDP Public Cloud Admin CDP Security (Q2-Q3) ADMINISTRATOR CDP Pub. Cloud (Q2)

CDP Data Governance (Q2)

DATA ANALYST CDW Hive/Impala (DC - Q1)

Spark Performance Spark (DC - Q1) Workshop DEVELOPER Flow Management with Stream Processing (Q1) Kafka Operations NiFi (CDF) (CDF) (CDF)

DATA SCIENTIST CML Data Science Wkshp (Q2) DS/ML Modules

cloudera.com/training.html

New set of data analytics Management Console applications Featuring use-case optimized interfaces Data Catalog Experiences

Workload Running on a container cloud Machine Data Data Manager DataFlow Fast provisioning & scaling, eﬃcient, simple Learning Warehouse Engineering Replication Manager With access to a shared data lake That is secured and governed Kubernetes

CDP Data Center SDX Bare Metal Security Workloads Metadata Governance

BareMetal