KDB Database Martin Toshev 25.04.2015 1 Agenda KDB Database Overview Q Lang Basics 2 Agenda KDB vs classical RDBMS KDB Java API KDB Use Cases 3 KDB Database Overview 4 KDB database overview KDB: • an in-memory column-based database • can be used to handle large volumes of real-time or historical data • provides OLTP and OLAP capabilities 5 KDB database overview KDB: • provides much of the same functionality of an RDBMS • supports SQL and KSql (for column-based queries) 6 KDB database overview Used by almost any large financial institution … • Goldman Sachs • Morgan Stanley • J.P. Morgan • Deutsche Bank • Commerzbank AG 7 KDB database overview KDB+: • 64bit release of KDB • OLTP: 100 000 – 1 000 000 records per second per CPU • OLAP: 1 000 000 – 10 000 000 records per second per CPU • Backed by the Q general purpose programming language 8 KDB database overview KDB+: • multi-threaded and multi-process (scales vertically) • can be deployed in a distributed environment (scales horizontally) • provides multiple APIs for C++, Java, Perl, Python … • and provides support for JDBC … 9 Q Lang Basics 10 Q Lang Basics Q language: • vector processing language • allows for complex calculations across a large volume of data • merges the K and ksql languages 11 Q Lang Basics Q language: • Interpreted (a mixture of APL and Scheme) • simple rules of precedence (right -> left evaluation of all expressions) • functions are first-class citizens (can be assigned to variables) 12 Q Lang Basics Q language data types: • Integer • time • floating point • datetime • date • char • symbol • lists • dictionaries • others … 13 Q Lang Demo 14 KDB vs classical RDBMS 15 KDB vs classical RDBMS KDB is column-oriented while RDBMS are row-oriented Both support SQL for querying the database 16 KDB vs classical RDBMS Classical RDBMS are in control of the persistence In KDB+ the user is in control of the persistence 17 KDB vs classical RDBMS No concept of transactions in KDB+ Queries in KDB+ are always SERIALIZABLE in terms of isolation levels 18 KDB vs classical RDBMS KDB provides utilities for dealing with time-series data: • time joins (af – asof join, wj – window join) 19 KDB Java API 20 KDB Java API Extremely simple … A single class c that represents a KDB+ connection 21 KDB Java API c.ks() – for executing an operation without waiting for a response c.k() – for execution an operation and waiting for a response c.k() – for waiting for a message from the KDB+ server (naming is somewhat dummy … ) 22 KDB Java API // getting the 'tbl' table c connection = new c("localhost", 5000); Flip tbl = (Flip) connection.k("tbl"); for(String key : tbl.x) { System.out.println(key + " " + tbl.at(key).getClass().getSimpleName()); } 23 KDB Java API // inserting a record in the 'tbl‘ table connection.ks("`tbl insert (`x;10)"); 24 KDB Java API Demo 25 KDB Use Cases 26 KDB Use Cases Capturing, storing and analyzing massive volumes of data … (kdb+tick) Market data Other feeds feeds Kdb+tick tickerplant KDB+ Kdb+tick Java/C/ log files rdb Python Kdb+tick hdb 27 KDB Use Cases Analyzing FIX messages for post trade analytics and compliance (kdb+tick) Market data Other feeds OMS feeds (Order Management System) Kdb+tick tickerplant KDB+ Kdb+tick Java/C/ log files rdb Python Kdb+tick hdb 28 KDB Use Cases Importing and analyzing massive volumes of data from an exchange … (kdb+taq) NYSE Kdb+tick taq 29 Thank you 30 Resources Wikipedia’s entry on KDB http://en.wikipedia.org/wiki/KDB KDB wiki http://code.kx.com/wiki/Main_Page KDB-tick overview http://kx.com/papers/Kdb+tickDiagram.pdf KDB guides http://www.timestored.com/kdb-guides/ 31 Resources K language http://www.math.bas.bg/bantchev/place/k.html KDB for Gods http://www.firstderivatives.com/lecture_series.asp KDB+ function reference http://code.kx.com/wiki/Reference 32 Resources Kdb+ Database and Language Primer http://kx.com/q/d/primer.htm KDB+ Database Reference Manual http://kx.com/q/d/kdb+1.htm Q language reference manual http://kx.com/q/d/q1.htm Abridged KDB+ Database Manual http://kx.com/q/d/kdb+.htm Abridged Q Language Manual http://www.kx.com/q/d/q.htm 33 Resources Kdb database tutorial http://www.thalesians.com/finance/index.php/Knowledge_Base/Databases/Kdb A shallow introduction to the K language http://www.kuro5hin.org/?op=displaystory;sid=2002/11/14/22741/791 K as a prototyping language http://www.cs.nyu.edu/courses/fall02/G22.3033-007/kintro.html 34 .