KDB Database
Martin Toshev 25.04.2015
1 Agenda
KDB Database Overview
Q Lang Basics
2 Agenda
KDB vs classical RDBMS
KDB Java API
KDB Use Cases
3 KDB Database Overview
4 KDB database overview
KDB:
• an in-memory column-based database
• can be used to handle large volumes of real-time or historical data
• provides OLTP and OLAP capabilities
5 KDB database overview
KDB:
• provides much of the same functionality of an RDBMS
• supports SQL and KSql (for column-based queries)
6 KDB database overview
Used by almost any large financial institution …
• Goldman Sachs • Morgan Stanley • J.P. Morgan • Deutsche Bank • Commerzbank AG
7 KDB database overview
KDB+:
• 64bit release of KDB
• OLTP: 100 000 – 1 000 000 records per second per CPU
• OLAP: 1 000 000 – 10 000 000 records per second per CPU
• Backed by the Q general purpose programming language
8 KDB database overview
KDB+:
• multi-threaded and multi-process (scales vertically)
• can be deployed in a distributed environment (scales horizontally)
• provides multiple APIs for C++, Java, Perl, Python …
• and provides support for JDBC …
9 Q Lang Basics
10 Q Lang Basics
Q language:
• vector processing language
• allows for complex calculations across a large volume of data
• merges the K and ksql languages
11 Q Lang Basics
Q language:
• Interpreted (a mixture of APL and Scheme)
• simple rules of precedence (right -> left evaluation of all expressions)
• functions are first-class citizens (can be assigned to variables)
12 Q Lang Basics
Q language data types:
• Integer • time • floating point • datetime • date • char
• symbol • lists • dictionaries • others …
13 Q Lang Demo
14 KDB vs classical RDBMS
15 KDB vs classical RDBMS
KDB is column-oriented while RDBMS are row-oriented
Both support SQL for querying the database
16 KDB vs classical RDBMS
Classical RDBMS are in control of the persistence
In KDB+ the user is in control of the persistence
17 KDB vs classical RDBMS
No concept of transactions in KDB+
Queries in KDB+ are always SERIALIZABLE in terms of isolation levels
18 KDB vs classical RDBMS
KDB provides utilities for dealing with time-series data:
• time joins (af – asof join, wj – window join)
19 KDB Java API
20 KDB Java API
Extremely simple …
A single class c that represents a KDB+ connection
21 KDB Java API c.ks() – for executing an operation without waiting for a response c.k() – for execution an operation and waiting for a response c.k() – for waiting for a message from the KDB+ server
(naming is somewhat dummy … )
22 KDB Java API
// getting the 'tbl' table c connection = new c("localhost", 5000); Flip tbl = (Flip) connection.k("tbl"); for(String key : tbl.x) { System.out.println(key + " " + tbl.at(key).getClass().getSimpleName()); }
23 KDB Java API
// inserting a record in the 'tbl‘ table connection.ks("`tbl insert (`x;10)");
24 KDB Java API Demo
25 KDB Use Cases
26 KDB Use Cases
Capturing, storing and analyzing massive volumes of data … (kdb+tick) Market data Other feeds feeds
Kdb+tick tickerplant
KDB+ Kdb+tick Java/C/ log files rdb Python
Kdb+tick hdb
27 KDB Use Cases
Analyzing FIX messages for post trade analytics and compliance (kdb+tick) Market data Other feeds OMS feeds (Order Management System) Kdb+tick tickerplant
KDB+ Kdb+tick Java/C/ log files rdb Python
Kdb+tick hdb
28 KDB Use Cases
Importing and analyzing massive volumes of data from an exchange … (kdb+taq)
NYSE
Kdb+tick taq
29 Thank you
30 Resources
Wikipedia’s entry on KDB http://en.wikipedia.org/wiki/KDB
KDB wiki http://code.kx.com/wiki/Main_Page
KDB-tick overview http://kx.com/papers/Kdb+tickDiagram.pdf
KDB guides http://www.timestored.com/kdb-guides/
31 Resources
K language http://www.math.bas.bg/bantchev/place/k.html
KDB for Gods http://www.firstderivatives.com/lecture_series.asp
KDB+ function reference http://code.kx.com/wiki/Reference
32 Resources
Kdb+ Database and Language Primer http://kx.com/q/d/primer.htm
KDB+ Database Reference Manual http://kx.com/q/d/kdb+1.htm
Q language reference manual http://kx.com/q/d/q1.htm
Abridged KDB+ Database Manual http://kx.com/q/d/kdb+.htm
Abridged Q Language Manual http://www.kx.com/q/d/q.htm
33 Resources
Kdb database tutorial http://www.thalesians.com/finance/index.php/Knowledge_Base/Databases/Kdb
A shallow introduction to the K language http://www.kuro5hin.org/?op=displaystory;sid=2002/11/14/22741/791
K as a prototyping language http://www.cs.nyu.edu/courses/fall02/G22.3033-007/kintro.html
34