High Volume Transaction Pro cessing Without Concurrency Control Two Phase Commit SQL or C Arthur Whitney Dennis Shasha Stevan Apter KX Systems Courant Institute NYU Union Bank of Switzerland Harker Avenue Mercer Street Park Avenue Palo Alto CA New York NY New York NY Abstract Imagine an application environment in which subsecond response to thousands of events gives the user a distinct competitive advantage yet transactional guarantees are important Imag ine also that the data ts comfortably into a few gigabytes of Random Access Memory These attributes characterize many nancial trading applications Which engine should one use in such a case IBM FastPath Sybase Oracle or Object Store We argue that an unconventional approach is cal led for we use a listbased language cal led K having optimized support for bulk array operators and that integrates networking and a graphical user interface Locking is unnecessary when singlethreading such applications because the data ts into memory obviating the need to go to disk except for logging purposes Multithreading can be hand led for OLTP applications by analyzing the arguments to transactions The result is a private sizereduced TPCB benchmark that achieves transactions per second with ful l recoverability and TCPIP overhead on an Megahertz UltraSparc I Further hot disaster recovery can be done with far less overhead than required by two phase commit by using a sequential state machine approach We show how to exploit multiple processors without complicating the language or our basic framework A Brief Description of the K Language K is listbased language integrating bulk op erators interpro cess communication and graphical user interface facilities designed and implemented by Arthur Whitney It is extremely concise For example a simple spreadsheet has b een implemented in ab out characters including the graphical user interface as illustrated in Figure Partly supp orted by NSF grants IRI and IRI Authors email addresses nnyaptnyubscom Steve Apter shashacsnyuedu Dennis Shasha asapnetcomcom Arthur Whitney FtD yF FfD The first line executes each formula of F and assigns the results to the corresponding positions of D The second line assigns those values of D to the corresponding display fields of F FDabcd across and deep showF Figure This K program implements a by spreadsheet including a user interface and full access to vector functionality Comments b egin with slashes The program has two data structures a set of formulas and constants F and a set of data D Every time a formula is changed the trigger Ft res This trigger up dates the data D based on the data and for mulas The strings F and data D are initialized to b e by arrays of blanks The last line displays the spreadsheet This program has slightly more functionality b ecause arbi trary formulas are allowed than a simple spreadsheet written in Java presented at lo cation httpwwwjavasoftcomappletsappletsSpreadSheetexamplehtml Unlike most other listbased languages K is extremely fast For example sorting three million records in memory takes two seconds on an IBM K is also small The entire runtime system written in C ts in bytes The basic data structures are arbitrary length multilists and dictionaries asso ciative arrays Binary op erations on equilength multilists include elementbyelement addition multiplication subtraction and division max and min Bo olean comparison op erators are also elementbyelement returning a list of s and s These op erators are overloaded with scalar variants as in APL So one can for example compare to every element in a list L and get a list of s and s of length jLj such that p osition i is if is equal to the ith p osition in the list and otherwise Rearrangement op erators include sort duplicate removal and partition Partition on a list L returns a list of sublists of p ositions where a given sublist of p ositions consists of all those p ositions in L having equal values For example applying partition to the list returns a list consisting of sublists p ositions having value p ositions having value and p ositions having value Lo cation functions include memb er ie is a value in a list nd viz nd rst place in a list having a value and where nd all places in a list having a value Find can b e optimized using hashing Any op erator can b e mo died to form a crosspro duct as well as left and right halfcross pro ducts Using the full crosspro duct one can for example compare every memb er of list L with every memb er of another list L This returns a list of s and s whose length is the pro duct of the cardinalities of L and of L Using the left halfcrosspro duct one can also compare or add or divide or any other binary op erations each memb er of L with all of L returning a list of sublists where each sublist is of the length of L and the numb er of sublists is the length of L Reduction op erators include count and prex scan For example one can mo dify the op erator plus with prex scan on the list and get the result Another op erator called over gives the last element of the plus prex scan for the example ab ove Twodimensional tables are built from multiple lists a fully vertically partitioned representa tion in which each attribute is asso ciated with a list Each value in an attribute can itself b e a list Vertical partitioning works well for decision supp ort b ecause only the attributes necessary for a query are brought in Also the scanning rates in K are very fast For example million records in RAM can b e scanned in seconds on an UltraSparc I to check that a eld value is greater than using the where feature mentioned ab ove The op erators ab ove allow K to implement relational algebra and aggregates including group by and having clauses An ecient library for all of relational algebra ts comfortably on a page Since K treats names as regular data one can p erform secondorder queries Supp ose for example that you want to union certain elds of a group of tables where the group is presented as a list of names SQL do esnt allow this but Ks second order facility makes this a short one liner In addition K has unied le inputoutput with TCPIP supp ort and has a graphical user interface built on top of XWindows in which the values of variables can determine their color This is useful on Wall Street where painting losses red is a useful visual cue K do es not have p ointers so do es not need to worry ab out p ointer swizzling Instead data structures are memorymapp ed from les Having a language with bulk op erators implies that there is no discontinuity b etween data intensive op erators and other op erators In the Wall Street context for example K supp orts b oth trading and analytics in a seamless fashion There is no need for cursors th genera tion languages emb edding calls from C or C SQL OSQL PowerBuilder Motif or Op en ClientOp enServer packages Note The Sigmo d reviewers found it annoying that we had no references for K The language is in fact proprietary even its description We therefore dep end on the readers cultural knowledge of APL as well as the descriptions ab ove to get a sense of what the language is Transaction Pro cessing Strategy The rest of this pap er concentrates on only the most essential asp ects of the K language A full and very fast programming language with bulk op erators that can easily express SQL emb o dies order eg top ten queries and that can b oth access disks and send and receive TCPIP messages How do we construct an online transaction pro cessing from such a language It turns out to b e embarrassingly simple Assume that a transaction program is implemented as a K pro cedure so it can b e describ ed by its name and its arguments Typical Wall Street programs enter a trade and up date the traders p osition as well as many subsidiary risk tables and back oce billclearing databases A message may consist of many transaction instances hereafter just transactions Our basic strategy is to log all incoming messages onto disk and then to p erform the transactions in those messages on the inmemory version of the database Recovery consists of replaying the logged transactions on disk after repro ducing the state from an appropriate dump In our applications so far this is the dump of the state as of last night Message logging is sucient b ecause the database is up dated sequentially Group Logging The basic strategy is slow if every transaction is logged separately So we write many trans actions each transaction is represented by its transaction typ e and all its parameters at a time to disk Once they are written to disk they are secure against main memory failure b ecause one can always recover the state of the database by replaying the log Disaster Recovery Warm start recovery might b e slow even with group logging b ecause one might have to recover from a transaction log that is very long Avoiding this problem requires frequent ie intraday dumping but let us step back a minute In the Wall Street setting we are worried not only ab out warm start recovery pro cessor failure and media failure lo cal disk but site failure as well rememb er the World Trade Center b ombing So we are interested in remote backups Further we want to dump the database state p erio dically for fast recovery in case of a massive p ower failure For these reasons the backup sites b ehave dierently from the primary as seen in gure On the primary a single pro cess logs a batch of transactions and applies them to the inmemory database In every backup a logger pro cess