Bring Your Language (And Libraries) to Your Data

Bring Your Language (and libraries) to Your Data

Stefan Mandl, Oleksandr Kozachuk, Jens Graupmann

. a column store, massively parallel processing (MPP), in-memory analytic database . modern software designed for analytics . runs on standard x86 hardware . ACID compliance . uses standard SQL language (with optional extensions) . suitable for any scale of data & any number of users . mature, proven & very cost effective . quick to implement & easy to operate The World’s Fastest Analytic Database

On 1TB data - an order of magnitude faster than the nearest rival

Sept ´14 5,246,338

Dec ´14 MS SQL Server 588,831

Sep ´14 Actian Vector 585,319

Apr ´14 MS SQL Server 519,976

Jun ´14 Actian Vector 485,242

Sep ´14 MS SQL Server 390,590

Feb ´14 Oracle 326,454

Aug ´14 MS SQL Server 304,361

Nov ´13 Sybase IQ 258,474 Source: www.tpc.org As of Oct 13, 2015

QphH@1000 GB 1,000,000 2,000,000 3,000,000 4.000,000 Queries per hour

Record in Inclusion in Gartner’s TPC-H Benchmark “Magic Quadrant for 100+ customers across EXASOL is founded (“Oracle dethroned“) DW DBMS Systems“ 16 countries/5 continents 2000 2008 2012 2015

2010 90ies 2006 Most successful vendor 2014 Early Research Pilot Customer of analytic database Successful global expansion, Success (University Karstadt-Quelle uses systems in Germany 100TB TPC-H benchmark Erlangen-Nürnberg) EXASOL in production (BARC)

. Leading supplier of payment solutions in Europe . Turned to EXASOL to guarantee safe and fast transactions at payment terminals . Also use EXASOL to analyze product development, product and process quality . Speed, scalability and great EXASOL support were defining factors for CCV . Use EXASOL to run in-depth analysis with “R“ and user-defined functions (UDFs) . Company benefits from an agile development environment

Virtual Schemas • Only metadata of virtually connected data sources are visible. • Whether virtual or “physical”: fully transparent from application perspective. Data Warehouse • Access to these virtual schemas is dynamically 1 2 forwarded to the connected data sources (1). Data is transferred on demand. Virtual Schema • If required, the data can be physically replicated into the DWH on demand without the need for additional ETL tools (2). • Coexistence with ETL

ETL Online Access  Agile access to most recent information  No/reduced redundancy  Less ETL-jobs  No waste of disk space

. Shared nothing

. The language . Role-your-own . Some form of PL/SQL . General purpose programming languages . Integration with the DB . Tight coupling . Loose coupling

Scalar output Set output Scalar input Scalar function Table generating function

Set input Aggregate function m:n mapping Special case: AF . Parallelism . For scalar input: start as many UDF containers as convenient (e.g. one per core), and send arbitrarily buffered rows to the UDFs. For the user it appears as one row at a time. . For set input: start as many UDF containers as convenient. Data is partitioned per group and processed by UDF per group. . → The level of parallelism can be controlled by appropriate GROUP BY clauses

● More precisely

– Open source the involved components

– Provide means to store custom compontents in the cluster

– Provide means to link the new components to SQL

. A filesystem that is automatically synchronized on all nodes . Support straightforward REST API for external or write access . Manages non-transactional resources (models/libraries/languages) . In UDFs: Read-Only access via filesystem

© 2017 EXASOL AG Defining Languages in SQL Sesssions create or replace PYTHON set script train_tree(tree_name varchar(1000), x1 int, x2 int, x3 int, y int) returns varchar(1000) as from sklearn import tree from sklearn.datasets import make_blobs import pickle import redis redis = redis.StrictRedis(host="[Hostname or IP of your redis DB]", port=6379, db=0) def run(ctx): X = [] Y = [] tree_name = None while True: # read all the data from the group if not tree_name: tree_name = ctx.tree_name assert(tree_name == ctx.tree_name) X.append([ctx.x1, ctx.x2, ctx.x3]) Y.append(ctx.y) if not ctx.next(): break clf = tree.DecisionTreeClassifier() clf = clf.fit(X, Y) redis.set(tree_name, pickle.dumps(clf)) return tree_name /

P YT HO N= lo ca lz mq +p ro to bu f: // /b fs de fa ul t/ de fa ul t/ EX AC lu st er OS /S cr ip tL an gu ag es -6 .0 .0 ?l an g= py th on #b uc ke ts /b fs de fa ul t/ de fa ul t/ EX AS ol ut io n- 6. 0. 0/ ex au df cl ie nt

PYTHON=

localzmq+protobuf:///

bfsdefault/default/EXAClusterOS/ScriptLanguages-6.0.0

?lang=python

#buckets/bfsdefault/default/EXASolution-6.0.0/exaudfclient

. Strategy:

– Compile the UDF‘s code on the fly – Dynamically link and call the generated code . Alternative: – Precompile all desired functions „into the language implementation“ – At runtime: only dispatch to the required function implementation

Recipe:

– Download Linux Container from BucketFS – Import into Docker – Mount your source code – Run docker – Build your container – Package as archive – Upload into cluster – Define it as new language in SQL

ALTER SESSION SET SCRIPT_LANGUAGES=’… … … … … … CPP=localzmq+protobuf:///bfsdefault/de fault/EXAClusterOS/ScriptLanguages- 6.0.0#buckets/bfsdefault/cpp/cppclient /cppclient‘

CREATE cpp SCALAR SCRIPT duplicateAndScale(n INT, x DOUBLE, y DOUBLE, z DOUBLE) EMITS (x DOUBLE, y DOUBLE, z DOUBLE) AS %compilerflags -lblas; #include using namespace UDFClient; void run_cpp(Metadata* meta, InputTable* in, OutputTable* out) { double x[] = {in->getDouble(1), in->getDouble(2), in->getDouble(3)}; cblas_dscal(3, 4.323, x, 1);

for (size_t n = 0; n < in->getInt64(0); ++n) { for (size_t i = 0; i < 3; ++i) out->setDouble(i,x[i]); out->next(); } } /

SQL_EXA> SELECT duplicateAndScale(3,1,2,3); EXA: SELECT duplicateAndScale(3,1,2,3);

X Y Z ------4.323 8.646000000000001 12.969 4.323 8.646000000000001 12.969 4.323 8.646000000000001 12.969

3 rows in resultset.

Recipe:

– Download Linux Container from BucketFS – Import into Docker – Mount your a target directory – Run docker, run R – Use R to install the library – Package as archive – Upload into cluster – Use it in R UDFs

Build your own Languages

Easy Integration via BucketFS

Use Your Language in EXASOL