DOCSLIB.ORG

KDB-DATABASE.Pdf

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
KDB-DATABASE.Pdf KDB Database Martin Toshev 25.04.2015 1 Agenda KDB Database Overview Q Lang Basics 2 Agenda KDB vs classical RDBMS KDB Java API KDB Use Cases 3 KDB Database Overview 4 KDB database overview KDB: • an in-memory column-based database • can be used to handle large volumes of real-time or historical data • provides OLTP and OLAP capabilities 5 KDB database overview KDB: • provides much of the same functionality of an RDBMS • supports SQL and KSql (for column-based queries) 6 KDB database overview Used by almost any large financial institution … • Goldman Sachs • Morgan Stanley • J.P. Morgan • Deutsche Bank • Commerzbank AG 7 KDB database overview KDB+: • 64bit release of KDB • OLTP: 100 000 – 1 000 000 records per second per CPU • OLAP: 1 000 000 – 10 000 000 records per second per CPU • Backed by the Q general purpose programming language 8 KDB database overview KDB+: • multi-threaded and multi-process (scales vertically) • can be deployed in a distributed environment (scales horizontally) • provides multiple APIs for C++, Java, Perl, Python … • and provides support for JDBC … 9 Q Lang Basics 10 Q Lang Basics Q language: • vector processing language • allows for complex calculations across a large volume of data • merges the K and ksql languages 11 Q Lang Basics Q language: • Interpreted (a mixture of APL and Scheme) • simple rules of precedence (right -> left evaluation of all expressions) • functions are first-class citizens (can be assigned to variables) 12 Q Lang Basics Q language data types: • Integer • time • floating point • datetime • date • char • symbol • lists • dictionaries • others … 13 Q Lang Demo 14 KDB vs classical RDBMS 15 KDB vs classical RDBMS KDB is column-oriented while RDBMS are row-oriented Both support SQL for querying the database 16 KDB vs classical RDBMS Classical RDBMS are in control of the persistence In KDB+ the user is in control of the persistence 17 KDB vs classical RDBMS No concept of transactions in KDB+ Queries in KDB+ are always SERIALIZABLE in terms of isolation levels 18 KDB vs classical RDBMS KDB provides utilities for dealing with time-series data: • time joins (af – asof join, wj – window join) 19 KDB Java API 20 KDB Java API Extremely simple … A single class c that represents a KDB+ connection 21 KDB Java API c.ks() – for executing an operation without waiting for a response c.k() – for execution an operation and waiting for a response c.k() – for waiting for a message from the KDB+ server (naming is somewhat dummy … ) 22 KDB Java API // getting the 'tbl' table c connection = new c("localhost", 5000); Flip tbl = (Flip) connection.k("tbl"); for(String key : tbl.x) { System.out.println(key + " " + tbl.at(key).getClass().getSimpleName()); } 23 KDB Java API // inserting a record in the 'tbl‘ table connection.ks("`tbl insert (`x;10)"); 24 KDB Java API Demo 25 KDB Use Cases 26 KDB Use Cases Capturing, storing and analyzing massive volumes of data … (kdb+tick) Market data Other feeds feeds Kdb+tick tickerplant KDB+ Kdb+tick Java/C/ log files rdb Python Kdb+tick hdb 27 KDB Use Cases Analyzing FIX messages for post trade analytics and compliance (kdb+tick) Market data Other feeds OMS feeds (Order Management System) Kdb+tick tickerplant KDB+ Kdb+tick Java/C/ log files rdb Python Kdb+tick hdb 28 KDB Use Cases Importing and analyzing massive volumes of data from an exchange … (kdb+taq) NYSE Kdb+tick taq 29 Thank you 30 Resources Wikipedia’s entry on KDB http://en.wikipedia.org/wiki/KDB KDB wiki http://code.kx.com/wiki/Main_Page KDB-tick overview http://kx.com/papers/Kdb+tickDiagram.pdf KDB guides http://www.timestored.com/kdb-guides/ 31 Resources K language http://www.math.bas.bg/bantchev/place/k.html KDB for Gods http://www.firstderivatives.com/lecture_series.asp KDB+ function reference http://code.kx.com/wiki/Reference 32 Resources Kdb+ Database and Language Primer http://kx.com/q/d/primer.htm KDB+ Database Reference Manual http://kx.com/q/d/kdb+1.htm Q language reference manual http://kx.com/q/d/q1.htm Abridged KDB+ Database Manual http://kx.com/q/d/kdb+.htm Abridged Q Language Manual http://www.kx.com/q/d/q.htm 33 Resources Kdb database tutorial http://www.thalesians.com/finance/index.php/Knowledge_Base/Databases/Kdb A shallow introduction to the K language http://www.kuro5hin.org/?op=displaystory;sid=2002/11/14/22741/791 K as a prototyping language http://www.cs.nyu.edu/courses/fall02/G22.3033-007/kintro.html 34 .
Load more

Recommended publications
  • An Array-Oriented Language with Static Rank Polymorphism
    An array-oriented language with static rank polymorphism Justin Slepak, Olin Shivers, and Panagiotis Manolios Northeastern University fjrslepak,shivers,[email protected] Abstract. The array-computational model pioneered by Iverson's lan- guages APL and J offers a simple and expressive solution to the \von Neumann bottleneck." It includes a form of rank, or dimensional, poly- morphism, which renders much of a program's control structure im- plicit by lifting base operators to higher-dimensional array structures. We present the first formal semantics for this model, along with the first static type system that captures the full power of the core language. The formal dynamic semantics of our core language, Remora, illuminates several of the murkier corners of the model. This allows us to resolve some of the model's ad hoc elements in more general, regular ways. Among these, we can generalise the model from SIMD to MIMD computations, by extending the semantics to permit functions to be lifted to higher- dimensional arrays in the same way as their arguments. Our static semantics, a dependent type system of carefully restricted power, is capable of describing array computations whose dimensions cannot be determined statically. The type-checking problem is decidable and the type system is accompanied by the usual soundness theorems. Our type system's principal contribution is that it serves to extract the implicit control structure that provides so much of the language's expres- sive power, making this structure explicitly apparent at compile time. 1 The Promise of Rank Polymorphism Behind every interesting programming language is an interesting model of com- putation.
    [Show full text]
  • Compendium of Technical White Papers
    COMPENDIUM OF TECHNICAL WHITE PAPERS Compendium of Technical White Papers from Kx Technical Whitepaper Contents Machine Learning 1. Machine Learning in kdb+: kNN classification and pattern recognition with q ................................ 2 2. An Introduction to Neural Networks with kdb+ .......................................................................... 16 Development Insight 3. Compression in kdb+ ................................................................................................................. 36 4. Kdb+ and Websockets ............................................................................................................... 52 5. C API for kdb+ ............................................................................................................................ 76 6. Efficient Use of Adverbs ........................................................................................................... 112 Optimization Techniques 7. Multi-threading in kdb+: Performance Optimizations and Use Cases ......................................... 134 8. Kdb+ tick Profiling for Throughput Optimization ....................................................................... 152 9. Columnar Database and Query Optimization ............................................................................ 166 Solutions 10. Multi-Partitioned kdb+ Databases: An Equity Options Case Study ............................................. 192 11. Surveillance Technologies to Effectively Monitor Algo and High Frequency Trading ..................
    [Show full text]
  • Application and Interpretation
    Programming Languages: Application and Interpretation Shriram Krishnamurthi Brown University Copyright c 2003, Shriram Krishnamurthi This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 United States License. If you create a derivative work, please include the version information below in your attribution. This book is available free-of-cost from the author’s Web site. This version was generated on 2007-04-26. ii Preface The book is the textbook for the programming languages course at Brown University, which is taken pri- marily by third and fourth year undergraduates and beginning graduate (both MS and PhD) students. It seems very accessible to smart second year students too, and indeed those are some of my most successful students. The book has been used at over a dozen other universities as a primary or secondary text. The book’s material is worth one undergraduate course worth of credit. This book is the fruit of a vision for teaching programming languages by integrating the “two cultures” that have evolved in its pedagogy. One culture is based on interpreters, while the other emphasizes a survey of languages. Each approach has significant advantages but also huge drawbacks. The interpreter method writes programs to learn concepts, and has its heart the fundamental belief that by teaching the computer to execute a concept we more thoroughly learn it ourselves. While this reasoning is internally consistent, it fails to recognize that understanding definitions does not imply we understand consequences of those definitions. For instance, the difference between strict and lazy evaluation, or between static and dynamic scope, is only a few lines of interpreter code, but the consequences of these choices is enormous.
    [Show full text]
  • Handout 16: J Dictionary
    J Dictionary Roger K.W. Hui Kenneth E. Iverson Copyright © 1991-2002 Jsoftware Inc. All Rights Reserved. Last updated: 2002-09-10 www.jsoftware.com . Table of Contents 1 Introduction 2 Mnemonics 3 Ambivalence 4 Verbs and Adverbs 5 Punctuation 6 Forks 7 Programs 8 Bond Conjunction 9 Atop Conjunction 10 Vocabulary 11 Housekeeping 12 Power and Inverse 13 Reading and Writing 14 Format 15 Partitions 16 Defined Adverbs 17 Word Formation 18 Names and Displays 19 Explicit Definition 20 Tacit Equivalents 21 Rank 22 Gerund and Agenda 23 Recursion 24 Iteration 25 Trains 26 Permutations 27 Linear Functions 28 Obverse and Under 29 Identity Functions and Neutrals 30 Secondaries 31 Sample Topics 32 Spelling 33 Alphabet and Numbers 34 Grammar 35 Function Tables 36 Bordering a Table 37 Tables (Letter Frequency) 38 Tables 39 Classification 40 Disjoint Classification (Graphs) 41 Classification I 42 Classification II 43 Sorting 44 Compositions I 45 Compositions II 46 Junctions 47 Partitions I 48 Partitions II 49 Geometry 50 Symbolic Functions 51 Directed Graphs 52 Closure 53 Distance 54 Polynomials 55 Polynomials (Continued) 56 Polynomials in Terms of Roots 57 Polynomial Roots I 58 Polynomial Roots II 59 Polynomials: Stopes 60 Dictionary 61 I. Alphabet and Words 62 II. Grammar 63 A. Nouns 64 B. Verbs 65 C. Adverbs and Conjunctions 66 D. Comparatives 67 E. Parsing and Execution 68 F. Trains 69 G. Extended and Rational Arithmeti 70 H. Frets and Scripts 71 I. Locatives 72 J. Errors and Suspensions 73 III. Definitions 74 Vocabulary 75 = Self-Classify - Equal 76 =. Is (Local) 77 < Box - Less Than 78 <.
    [Show full text]
  • A Guide to PL/M Programming
    INTEL CORP. 3065 Bowers Avenue, Santa Clara, California 95051 • (408) 246-7501 mcs=a A Guide to PL/M programming PL/M is a new high level programming language designed specifically for Intel's 8 bit microcomputers. The new language gives the microcomputer systems program­ mer the same advantages of high level language programming currently available in the mini and large computer fields. Designed to meet the special needs of systems programming, the new language will drastically cut microcomputer programming time and costs without sacrifice of program efficiency. in addition, training, docu­ mentation, program maintenance and the inclusion of library subroutines will all be made correspondingly easier. PL/M is well suited for all microcomputer program­ ming applications, retaining the control and efficiency of assembly language, while greatly reducing programming effort The PL/M compiler is written in ANSI stand­ ard Fortran I V and thus will execute on most machines without alteration. SEPTEMBER 1973 REV.1 © Intel Corporation 1973 TABLE OF CONTENTS Section Page I. INTRODUCTION TO PL/M 1 II. A TUTORIAL APPROACH TO PL/M ............................ 2 1. The Organization of a PL/M Program ........................ 2 2. Basic Constituents of a PL/M Program ....................... 4 2.1. PL/M Character Set ................................ 4 2.2. Identifiers and Reserved Words ....................... 5 2.3. Comments . 7 3. PL/M Statement Organization . 7 4. PL/M Data Elements . 9 4.1. Variable Declarations ............................... 9 4.2. Byte and Double Byte Constants. .. 10 5. Well-Formed Expressions and Assignments . 11 6. A Simple Example. 15 7. DO-Groups. 16 7.1. The DO-WHILE Group ...........
    [Show full text]
  • Visual Programming Language for Tacit Subset of J Programming Language
    Visual Programming Language for Tacit Subset of J Programming Language Nouman Tariq Dissertation 2013 Erasmus Mundus MSc in Dependable Software Systems Department of Computer Science National University of Ireland, Maynooth Co. Kildare, Ireland A dissertation submitted in partial fulfilment of the requirements for the Erasmus Mundus MSc Dependable Software Systems Head of Department : Dr Adam Winstanley Supervisor : Professor Ronan Reilly June 30, 2013 Declaration I hereby certify that this material, which I now submit for assessment on the program of study leading to the award of Master of Science in Dependable Software Systems, is entirely my own work and has not been taken from the work of others save and to the extent that such work has been cited and acknowledged within the text of my work. Signed:___________________________ Date:___________________________ Abstract Visual programming is the idea of using graphical icons to create programs. I take a look at available solutions and challenges facing visual languages. Keeping these challenges in mind, I measure the suitability of Blockly and highlight the advantages of using Blockly for creating a visual programming environment for the J programming language. Blockly is an open source general purpose visual programming language designed by Google which provides a wide range of features and is meant to be customized to the user’s needs. I also discuss features of the J programming language that make it suitable for use in visual programming language. The result is a visual programming environment for the tacit subset of the J programming language. Table of Contents Introduction ............................................................................................................................................ 7 Problem Statement ............................................................................................................................. 7 Motivation..........................................................................................................................................
    [Show full text]
  • Frederick Phillips Brooks, Jr
    Frederick Phillips Brooks, Jr. Biography Frederick P. Brooks, Jr., was born in 1931 in Durham NC, and grew up in Greenville NC. He received an A.B. summa cum laude in physics from Duke and a Ph.D. in computer science from Harvard, under Howard Aiken, the inventor of the early Harvard computers. He joined IBM, working in Poughkeepsie and Yorktown, NY, 1956-1965. He was an architect of the Stretch and Harvest computers and then was the project manager for the development of IBM's System/360 family of computers and then of the Operating System/360 software. For this work he received a National Medal of Technology jointly with Bob O. Evans and Erich Bloch Brooks and Dura Sweeney in 1957 patented an interrupt system for the IBM Stretch computer that introduced most features of today's interrupt systems. He coined the term computer architecture. His System/360 team first achieved strict compatibility, upward and downward, in a computer family. His early concern for word processing led to his selection of the 8-bit byte and the lowercase alphabet for the System/360, engineering of many new 8-bit input/output devices, and introduction of a character-string datatype in the PL/I programming language. In 1964 he founded the Computer Science Department at the University of North Carolina at Chapel Hill and chaired it for 20 years. Currently, he is Kenan Professor of Computer Science. His principal research is in real-time, three-dimensional, computer graphics—“virtual reality.” His research has helped biochemists solve the structure of complex molecules and enabled architects to “walk through” structures still being designed.
    [Show full text]
  • Symantec™ Endpoint Protection 14.X for Linux Client Guide Contents
    Symantec™ Endpoint Protection 14.x for Linux Client Guide Contents Chapter 1 Protecting Linux computers with Symantec Endpoint Protection ....................................................... 3 About Symantec Endpoint Protection for Linux ..................................... 3 Symantec Endpoint Protection client for Linux system requirements ........................................................................... 4 Getting started on the Linux client ...................................................... 5 About the Linux client graphical user interface ...................................... 7 What happens when a virus is detected ........................................ 7 Importing client-server communication settings into the Linux client ..................................................................................... 8 Uninstalling the Symantec Endpoint Protection client for Linux ................ 9 Appendix A Symantec Endpoint Protection client for Linux command line reference .............................................. 11 rtvscand ..................................................................................... 12 sav ............................................................................................ 15 savtray ....................................................................................... 22 smcd ......................................................................................... 24 symcfg ....................................................................................... 27 symcfgd ....................................................................................
    [Show full text]
  • KDB+ Quick Guide
    KKDDBB++ -- QQUUIICCKK GGUUIIDDEE http://www.tutorialspoint.com/kdbplus/kdbplus_quick_guide.htm Copyright © tutorialspoint.com KKDDBB++ -- OOVVEERRVVIIEEWW This is a complete quide to kdb+ from kx systems, aimed primarily at those learning independently. kdb+, introduced in 2003, is the new generation of the kdb database which is designed to capture, analyze, compare, and store data. A kdb+ system contains the following two components − KDB+ − the database kdatabaseplus Q − the programming language for working with kdb+ Both kdb+ and q are written in k programming language (same as q but less readable). Background Kdb+/q originated as an obscure academic language but over the years, it has gradually improved its user friendliness. APL 1964, AProgrammingLanguage A+ 1988, modifiedAPLbyArthurWhitney K 1993, crispversionofA + , developedbyA. Whitney Kdb 1998, in − memorycolumn − baseddb Kdb+/q 2003, q language – more readable version of k Why and Where to Use KDB+ Why? − If you need a single solution for real-time data with analytics, then you should consider kdb+. Kdb+ stores database as ordinary native files, so it does not have any special needs regarding hardware and storage architecture. It is worth pointing out that the database is just a set of files, so your administrative work won’t be difficult. Where to use KDB+? − It’s easy to count which investment banks are NOT using kdb+ as most of them are using currently or planning to switch from conventional databases to kdb+. As the volume of data is increasing day by day, we need a system that can handle huge volumes of data. KDB+ fulfills this requirement. KDB+ not only stores an enormous amount of data but also analyzes it in real time.
    [Show full text]
  • Software Conceptual Integrity: Deconstruction, Then Reconstruction
    Software Conceptual Integrity: Deconst, Then Reconst Iaakov Exman Software Conceptual Integrity: Deconstruction, Then Reconstruction Iaakov Exman 1 Software Engineering Department The Jerusalem College of Engineering- Azrieli Jerusalem, Israel [email protected] Computer Classification System Software and its engineering → Software creation and management → Designing software {CCS 2012} Abstract Conceptual Integrity is the most important consideration for software system design, as stated by Frederick Brooks. Brooks also suggested that Conceptual Integrity can be attained by means of design principles, such as Propriety, and Orthogonality. However, Brooks’ principles have not been formalized, posing obstacles to their application in practice, and to a deeper comprehension of Conceptual Integrity. This paper has three goals: first, to achieve deeper comprehension of Conceptual Integrity by deconstructing it into two phases, viz. Conceptualization and Modularization , iteratively applied during software system design; second, to show that the algebraic Linear Software Models already provide the hitherto lacking formalization of Brooks’ design principles, which surprisingly belong mainly to the Modularization phase; third, to reconstruct Conceptualization and Modularization, preserving the desirable tension between: a- phases’ separation , each with its own specific formal manipulation techniques; b- precise transition between these phases, consisting of explicit mutual relationships. The tension stems from the Modularity Matrix linking two
    [Show full text]
  • Krust: a Formal Executable Semantics of Rust
    KRust: A Formal Executable Semantics of Rust Feng Wang∗, Fu Song∗, Min Zhang†, Xiaoran Zhu† and Jun Zhang∗ ∗School of Information Science and Technology, ShanghaiTech University, Shanghai, China †Shanghai Key Laboratory of Trustworthy Computing, East China Normal University, Shanghai, China Abstract—Rust is a new and promising high-level system object is a variable binding. When a variable goes out of its programming language. It provides both memory safety and scope, Rust will free the bound resources. The ownership of thread safety through its novel mechanisms such as ownership, an object can be transferred, after which the object cannot moves and borrows. Ownership system ensures that at any point there is only one owner of any given resource. The ownership of be accessed via the outdated owner. This ensures that there a resource can be moved or borrowed according to the lifetimes. is exactly one binding to any object. Instead of transferring The ownership system establishes a clear lifetime for each value ownership, Rust provides borrows which allows an object to and hence does not necessarily need garbage collection. These be shared by multiple references. There are two kinds of novel features bring Rust high performance, fine low-level control borrows in Rust: mutable references and immutable references. of C and C++, and unnecessity in garbage collection, which differ Rust from other existing prevalent languages. For formal A mutable reference can mutate the borrowed object if the analysis of Rust programs and helping programmers learn its object is mutable, while an immutable reference cannot mutate new mechanisms and features, a formal semantics of Rust is the borrowed object even the object itself is mutable.
    [Show full text]
  • A Declarative Metaprogramming Language for Digital Signal
    Vesa Norilo Kronos: A Declarative Centre for Music and Technology University of Arts Helsinki, Sibelius Academy Metaprogramming PO Box 30 FI-00097 Uniarts, Finland Language for Digital [email protected] Signal Processing Abstract: Kronos is a signal-processing programming language based on the principles of semifunctional reactive systems. It is aimed at efficient signal processing at the elementary level, and built to scale towards higher-level tasks by utilizing the powerful programming paradigms of “metaprogramming” and reactive multirate systems. The Kronos language features expressive source code as well as a streamlined, efficient runtime. The programming model presented is adaptable for both sample-stream and event processing, offering a cleanly functional programming paradigm for a wide range of musical signal-processing problems, exemplified herein by a selection and discussion of code examples. Signal processing is fundamental to most areas handful of simple concepts. This language is a good of creative music technology. It is deployed on fit for hardware—ranging from CPUs to GPUs and both commodity computers and specialized sound- even custom-made DSP chips—but unpleasant for processing hardware to accomplish transformation humans to work in. Human programmers are instead and synthesis of musical signals. Programming these presented with a very high-level metalanguage, processors has proven resistant to the advances in which is compiled into the lower-level data flow. general computer science. Most signal processors This programming method is called Kronos. are programmed in low-level languages, such as C, often thinly wrapped in rudimentary C++. Such a workflow involves a great deal of tedious detail, as Musical Programming Paradigms these languages do not feature language constructs and Environments that would enable a sufficiently efficient imple- mentation of abstractions that would adequately The most prominent programming paradigm for generalize signal processing.
    [Show full text]