SQL for Nosql and How Apache Calcite Can Help (Slides)

Home , Apache Apex, Apache Calcite, Apache Drill, Apache Hive, Apache Kylin, Apache Phoenix

SQL for NoSQL and how Apache Calcite can help

FOSDEM 2017 Christian Tzolov

Engineer at Pivotal BigData, Hadoop, Spring Cloud Dataflow Apache Committer, PMC member Apache {Crunch, Geode, HAWQ, ...}

blog.tzolov.net twitter.com/christzolov nl.linkedin.com/in/tzolov

Disclaimer This talk expresses my personal opinions. It is not read or approved by Pivotal and does not necessarily reflect the views and opinions of Pivotal nor does it constitute any official communication of Pivotal. Pivotal does not support any of the code shared here.

2 “It will be interesting to see what happens if an established NoSQL database decides to implement a reasonably standard SQL; The only predictable outcome for such an eventuality is plenty of argument.”

2012, Martin Fowler, P.J.Sadalage, NoSQL Distilled

3 Data Big Bang

Why?

4 NoSQL Driving Forces

• Rise of Internet Web, Mobile, IoT – Data Volume, Velocity, Variety challenges ACID & 2PC clash with Distributed architectures. CAP, PAXOS instead..

• Row-based Relational Model. Object-Relational Impedance Mismatch More convenient data models: Datastores, Key/Value, Graph, Columnar, Full-text Search, Schema-on-Load…

• Infrastructure Automation and Elasticity (Cloud Computing) Eliminate operational complexity and cost. Shift from Integration to application databases …

5 Data Big Bang Implications

• Over 150 commercial NoSQL and BigData systems.

• Organizations will have to mix data storage technologies!

• How to integrate such multitude of data systems?

6 “Standard” Data Process/Query Language?

• Functional - Unified Programming pcollection.apply(Read.from(”in.txt")) Model .apply(FlatMapElements.via((String word) -> asList(word.split("[^a-zA-Z']+"))) • Apache {Beam, Spark, Flink, .apply(Filter.by((String word)->!word.isEmpty())) Apex, Crunch}, Cascading .apply(Count.perElement()) • Converging around Apache Batch & Streaming, OLTP Beam

• Declarative - SQL • Adopted by many NoSQL SELECT b."totalPrice", c."firstName” Vendors FROM "BookOrder" as b INNER JOIN "Customer" as c • Most Hadoop tasks: Hive and ON b."customerNumber" = c."customerNumber” SQL-on-Hadoop WHERE b."totalPrice" > 0; • Spark SQL - most used production component for 2016 OLAP, EDW, Exploration • Google F1

SQL for NoSQL?

• Extended Relational Algebra - already present in most NoSql data system • Relational Expression Optimization – Desirable but hard to implement

8 Organization Data - Integrated View

Single Federated DB (M:1:N) Direct (M:N)

Organization Data Tools Organization Data Tools

SQL/JDBC

HAWQ FDBS SQL/JDBC SQL/JDBC SQL/JDBC Apache HAWQ Optimizer, Columnar /browse/HAWQ-1235 (HDFS) Calcite Calcite Calcite jira / SQLAdapter 1 SQLAdapter 2 … SQLAdapter n

NoSQL 1 NoSQL 2 NoSQL n PXF 1 PXF 2 PXF n

Native Native Native API 1 API 2 API n

NoSQL 1 NoSQL 1 … NoSQL n issues.apache.org https://

9 Single Federated Database Federated External Tables with Apache HAWQ - MPP, Shared-Noting, SQL- on-Hadoop

CREATE EXTERNAL TABLE MyNoSQL ( customer_id TEXT, first_name TEXT, last_name TEXT, gender TEXT ) LOCATION ('pxf://MyNoSQL-URL>? FRAGMENTER=MyFragmenter& ACCESSOR=MyAccessor& RESOLVER=MyResolver&') FORMAT 'custom'(formatter='pxfwritable_import');

10 Apache Calcite?

Java framework that allows SQL interface and advanced query optimization, for virtually any data system

• Query Parser, Validator and Optimizer(s) • JDBC drivers - local and remote • Agnostic to data storage and processing Calcite Application

• Apache Apex • Apache Drill • Apache Flink • Apache Hive • Apache Kylin • Apache Phoenix • Apache Samza • Apache Storm • Cascading • Qubole Quark • SQL-Gremlin … • Apache Geode

12 SQL Adapter Design Choices SQL completeness vs. NoSql design integrity

Catalog – namespaces accessed in queries Schema - collection of schemas and tables • Data Type Conversion Table - single data set, collection of rows RelDataType – SQL fields types in a Table

(simple) Predicate Pushdown: Scan, Filter, • Move Computation to Data Projection (complex) Custom Relational Rules and Operations: Sort, Join, GroupBy ...

13 Geode to Calcite Data Types Mapping

Geode Cache is mapped into Calcite Schema

Create Column Types Geode Cache Calcite Schema (RelDataType) from Geode Value class (JavaTypeFactoryImpl) Region 1 Table 1 Col1 Col2 ColN

Key Val Row1 V(1,1) V(1,2) V(1,N)

k1 v1 Row2 V(2,1) V(2,2) V(2,N)

k2 v2 RowM V(M,1) V(M,2) V(M,N)

… Geode Key/Value is mapped … into Table Row

Region K Table K

Regions are mapped into Tables

14 Geode Adapter - Overview

SQL/JDBC/ODBC

Parse SQL, converts into relational expression and optimizes Apache Calcite Push down the relational expressions supported by Geode OQL and falls back to the Calcite Spring Data API for Enumerable Enumerable Adapter for the rest interacting with Geode Adapter

Convert SQL relational expressions into OQL queries Spring Data Geode Adapter Geode (Geode Client)

Geode API and OQL

Geode Server Geode Server Geode Server

Data Data Data Simple SQ L Adapter

Initialize <> <> MySchemaFactory MySchema !connect jdbc:calcite:model=path-to-model.json

+create(operands):Schema +getTableMap():Map)

<> defaultSchema: 'MyNoSQL', schemas: [{ <> name: ’MyNoSQLAdapter, factory: MySchemaFactory’, operand: { myNoSqlUrl: …, } ScannableTable, Uses reflection to builds <> }] RelDataType from your FilterableTable, MyTable value’s class type

ProjectableFilterableTable +getRowType(RelDataTypeFactor) Returns an Enumeration +scan(ctx):Ennumerator over the entire target data store <>

SELECT b."totalPrice” Defined in the Linq4j Query FROM "BookOrder" as b <> WHERE b."totalPrice" > 0; MyEnummerator sub-project Converts MyNoSQL +moveNext() value response into +convert(Object):E Calcite row data

My NoSQL

16 Non-Relational Tables (Simple)

Scanned without intermediate relational expression. • ScannableTable - can be scanned Enumerable scan(DataContext root);

• FilterableTable - can be scanned, applying supplied filter expressions Enumerable scan(DataContext root, List filters);

• ProjectableFilterableTable - can be scanned, applying supplied filter expressions and projecting a given list of columns Enumerable scan(DataContext root, List filters, int[] projects);

17 Calcite Ecosystem

Several “semi-independent” projects.

Local and Remote Port of LINQ (Language-Integrated Query) JDBC driver to Java.

Linq4j JDBC and Avatica Method for translating executable code into data Expression Converts SQL (LINQ/MSN port) queries Into AST SQL Parser & AST Tree (SqlNode …)

Complies Java code Relational generated from linq4j “Expressions”. Part of the Relational Algebra, • Relational Expressions Interpreter physical plan implementer expression, • Row Expression optimizations … • Optimization Rules • Planner … Enumerable Default (In-memory) Data Store Adapter Adapter implementation. Leverages Linq4j

3rd party Adapters

18 Calcite SQL Query Execution Flow

1. On new SQL query JDBC delegates to Prepare to prepare the query execution JDBC

2. Parse SQL, convert to rel. 1 Calcite Framework expressions. Validate and Optimize 2 them Prepare Geode Adapter

3. Start building a physical plan from the relation expressions SQL, 3 5 Interpreter 4. Implement the Geode relations and Relational, Enumerable encode them as Expression tree Planner

5. Pass the Expression tree to the 4 6 7 Interpreter to generate Java code 2 Geode 7 6. Generate and Compile a Binder Binder instance that on ‘bind()’ call runs Adapter Geodes’ query method 7 7. JDBC uses the newly compiled Binder to perform the query on the Geode Geode Cluster Cluster

19 Calcite Relational Expressions

TableScan

Input Column Project Ref

Row-level Relational Literal expressions expression Filter

Struct field * RexlNode RelNode Aggregate access

Project, Sort fields * Join Function call Filter, Join conditions RelTrait Window Intersect expressions Physical attribute of a relation Sort

20 Calcite Relational Expressions

Rule transforms an expression into another. It has a list of Operands, which determine whether the rule can be applied to RelOptRuleOperand a particular section of the tree. * <>

Query optimizer: Transforms a Converts from one calling relational expression according to RelOptRule ConverterRule convention to another a given set of rules and a cost model. * + onMatch(call) + RelNode convert(RelNode) <> RelOptPlanner Indicate that it converts a <> physical attribute only! <> Convertor +findBestExp():RelNode <>

* Calling convention used to represent a single RelNode RelTrait Convention data source. Inputs to a relational + register(RelOptPlander) expression must be in RelOptCluster + List getInputs(); the same convention RexNode * *

<> MyDBConvention NONE EnumberableConvention

21 Calcite Adapter Implementation Patterns

TranslatableTable AbstractQueryableTable Convention.Impl(“MyAdapter”) RelNode

<> 1 2 MyAdapterProject 3 MyAdapterTable MyAdapterConvention MyAdapterRel MyAdapterFilter 9 + toRel(optTable) + implement(implContext)

<> + asQueryable(provider,…) TableScan ImplContext MyAdapterXXX

Can convert queries in 5 + implParm1 Expression AbstractTableQueryable <> + implParm2 … MyAdapterScan <> 4 + register(planer) { MyAdapterQueryable Common interface for all MyAdapter Registers all MyAdapter Rules Relation Expressions. Provides } implementation callback method called + myQuery(params) : as part of physical plan implementation Enumetator MyAdapterProjectRu

6 RelOptRule MyAdapterFilterRule 8 myQuery() implements the call to your DB It is called by the auto generated code. It MyAdapterToEnumerableConvertorRule must return an Enumberable instance MyAdapterXXXRule operands: (RelNode.class, MyAdapterConvention, EnumerableConvention) ConverterRue

<> 7 MyAdapterToEnumerableConvertor EnumerableRel

+ implement(EnumerableRelImplementor) {

Recursively call the implement on each ctx = new MyAdapterRel.ImplContext() ConvertorImpl MyAdapter Relation Expression getImputs().implement(ctx) Calcite Framework return BlockBuild.append( MY_QUERY_REF, MY_QUERY_REF = Types.lookupMethod( MyAdapterQueryable.class, Expressions.constant(ctx.implParms1), ”myQuery”, MyAdapter components Encode the myQuery(params) call as Expressions.constant(ctx.implParms2) … String.class Expressions String.class);

22 Relational Algebra

SELECT b."totalPrice", c."firstName” FROM "BookOrder" as b INNER JOIN "Customer" as c ON b."customerNumber" = c."customerNumber” WHERE b."totalPrice" > 0;

Project (c.firstName, b.totalPrice)

(c.firstName, b.totalPrice) Project

(on customerNumber) optimize Join (b.totalPrice > 0) Filter (totalPrice, Project customerNumber) (on customerNumber) Join

(firstName, Project Filter (totalPrice > 0) customerNumber) Scan Scan

Scan Scan

Customer [c] BookOrder [b] Customer [c] BookOrder [b]

23 Calcite with Geode - Without Implementation

SELECT b."totalPrice", c."firstName” FROM "BookOrder" as b INNER JOIN "Customer" as c ON b."customerNumber" = c."customerNumber” WHERE b."totalPrice" > 0;

24 Calcite with Geode – Scannable Table (Simple)

SELECT b."totalPrice", c."firstName” FROM "BookOrder" as b INNER JOIN "Customer" as c ON b."customerNumber" = c."customerNumber” WHERE b."totalPrice" > 0;

25 Calcite with Geode – Relational (Complex)

SELECT b."totalPrice", c."firstName” FROM "BookOrder" as b INNER JOIN "Customer" as c ON b."customerNumber" = c."customerNumber” WHERE b."totalPrice" > 0;

26 Calcite JDBC Connection

27 References

• Big Data is Four Different Problems, 2016, M.Stonebraker: https://www.youtube.com/watch?v=S79-buNhdhI • Turning Database Inside-Out, 2015 (M. Kleppmann) https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza • NoSQL Distilled, 2012 (Pramod J. Sadalage and M.Fowler) https://martinfowler.com/books/nosql.html • Architecture of a Database System, 2007 (J.M. Hellerstein, M. Stonebraker, J. Hamilton)http://db.cs.berkeley.edu/papers/fntdb07-architecture.pdf • ORCA: A Modular Query Optimizer Architecture for Big Data: http://15721.courses.cs.cmu.edu/spring2016/papers/p337-soliman.pdf • Apache Geode Project (2016) : http://geode.apache.org • Geode Object Query Language (OQL) : http://bit.ly/2eKywgp • Apache Calcite Project (2016) : https://calcite.apache.org • Apache Geode Adapter for Apache Calcite: https://github.com/tzolov/calcite • Relational Algebra Operations: https://www.coursera.org/learn/data-manipulation/lecture/ 4JKs1/relational-algebra-operators-union-difference-selection

Thanks! Apache Geode?

“… in-memory, distributed database with strong consistency built to support low latency transactional applications at extreme scale” Why Apache Geode?

China Railway

5,700 train stations 7,000 stations 4.5 million tickets per day 72,000 miles of track 20 million daily users 23 million passengers daily 1.4 billion page views per day 120,000 concurrent users 40,000 visits per second 10,000 transactions per minute

https://pivotal.io/big-data/case-study/distributed-in-memory-data-management-solution https://pivotal.io/big-data/case-study/scaling-online-sales-for-the-largest-railway-in-the-world-china-railway-corporation

31 Geode Features

• In-Memory Data Storage – Continuous OQL Queries – Over 100TB Memory • Multi-site / Inter-cluster – JVM Heap + Off Heap • Full Text Search (Lucene indexes) • Any Data Format • Embedded and Standalone – Key-Value/Object Store • Top Level Apache Project • ACID and JTA Compliant Transactions • HA and Linear Scalability • Strong Consistency • Streaming and Event Processing – Listeners – Distributed Functions

32 Apache Geode Concepts

Locator – tracks system Client –read and modify the members and provides content of the distributed membership information system

Locator (member) Client Listener – event handler. CacheServer – process Registers for one or connected to the distributed Cache Server (member) more events and notified system with created Cache when they occur

Cache Listeners Region - consistent, distributed Ke Val Functions – distributed, Region 1 y Map (key-value), concurrent data k1 v1 Partitioned or Replicated processing Functions k2 v2 … Cache - In-memory collection … Region N of Regions Geode Topology

Cache Server Cache Server Cache Server

Cache Data Cache Data Cache Data Peer-to-Peer

Client

Local Cache

pool Client-Server

Cache Server Cache Server Cache Server

Cache Data Cache Data Cache Data

… Cache Server Cache Server Cache Server Cache Server

Cache Data Cache Data Cache Data Cache Data

Gateway Receiver Gateway Sender

WAN Multi-site Boundary Multi-Site

… Gateway Sender Gateway Receiver

Cache Data Cache Data Cache Data Cache Data

Cache Server Cache Server Cache Server Cache Server