A Queue-oriented Transaction Processing Paradigm Thamir M. Qadah∗† Exploratory Systems Lab School of Electrical and Computer Engineering, Purdue University, West Lafayette [email protected] Abstract because the final database state cannot be entirely deter- Transaction processing has been an active area of research mined by the input database state and the input set of trans- for several decades. A fundamental characteristic of classical actions. The output database state acceptable as long as the transaction processing protocols is non-determinism, which resulted history of concurrent transaction execution is equiv- causes them to suffer from performance issues on modern alent to some serial history of execution according to serial- computing environments such as main-memory databases izability theory. using many-core, and multi-socket CPUs and distributed The goal of transaction processing protocols is to ensure environments. Recent proposals of deterministic transaction ACID properties and increase the concurrency of executed processing techniques have shown great potential in address- transactions. Serializable isolation ensures anomaly-free ex- ing these performance issues. In this position paper, I argue ecution. Using other isolation levels (e.g., Read-committed) for a queue-oriented transaction processing paradigm that improves concurrency but is prone to producing anomalies leads to better design and implementation of deterministic that defy users’ intentions and leave the database in an un- transaction processing protocols. I support my approach desirable inconsistent state. with extensive experimental evaluations and demonstrate Due to the non-deterministic nature of classical trans- significant performance gains. action processing protocols, they suffer from performance issues on modern computing environments such as main- CCS Concepts • Information systems → Database trans- memory databases that use many-core and multi-socket action processing; Distributed database transactions; CPUs, and cloud-based distributed environment. In this Ph.D. Main memory engines; • Computer systems organiza- dissertation, I look into ways to impose determinism to im- tion → Multicore architectures; Distributed architec- prove the performance of transaction processing in modern tures; computing environments. Keywords database systems, transaction processing, con- currency control, distributed database systems, performance 2 Transaction Processing in Modern evaluation Computing Environments ACM Reference Format: Thamir M. Qadah. 2019. A Queue-oriented Transaction Processing This section describes two major performance issues when Paradigm. In Proceedings of ACM Conference (Conference’17). ACM, running database transactions using non-deterministic trans- New York, NY, USA, 5 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn action processing protocols. In this section, our discussion assumes the requirement of a serializable isolation model. 1 Introduction Transaction processing is an old-aged problem that has been 2.1 High-contention Workloads an active area of research for the past 40 years[8]. Classical Under high-contention workloads, non-deterministic trans- arXiv:1910.10350v1 [cs.DB] 23 Oct 2019 transaction processing is characterized as non-deterministic action processing protocols suffer from high abort rates ∗The author is co-advised by Prof. Mohammad Sadoghi because their concurrency control algorithms need to en- † Also with Umm Al-Qura University, Makkah, Saudi Arabia. sure serializable histories. Pessimistic concurrency control Permission to make digital or hard copies of all or part of this work for algorithms, abort transactions to avoid deadlocks, and op- personal or classroom use is granted without fee provided that copies are not timistic concurrency control algorithms abort transactions made or distributed for profit or commercial advantage and that copies bear during the validation phase. Ensuring deadlock-free execu- this notice and the full citation on the first page. Copyrights for components tion and validating transactions require extensive coordina- of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to tion among concurrent threads executing transactions while redistribute to lists, requires prior specific permission and/or a fee. Request guaranteeing serializability. The main research question for permissions from [email protected]. this problem is: Is it possible to process high-contended work- Conference’17, July 2017, Washington, DC, USA loads in a concurrency control-free and minimal coordination © 2019 Association for Computing Machinery. while ensuring serializability? What is the right abstraction ACM ISBN 978-x-xxxx-xxxx-x/YY/MM...$15.00 and principles to achieve that? https://doi.org/10.1145/nnnnnnn.nnnnnnn Conference’17, July 2017, Washington, DC, USA Thamir M. Qadah 2.2 Distributed Commit Protocols 3 Approach In distributed transaction processing, agreement protocols Our goal is to process transactions efficiently in modern introduce significant overhead to the processing because all computing environments with minimal coordination among participant nodes need to agree on the fate of an executed the threads running in our system. The proposed approach distributed transaction. Achieving this agreement involves addresses the research questions presented in the previous multiple rounds of communication messages to be exchanged section. The answer to these questions relies on three princi- among participating nodes. ples: transaction fragmentation, deterministic two-phase pro- The state-of-the-art to solve the agreement problem on cessing, and priority-based queue-oriented representation the fate of transactions in database systems is the two-phase of transnational workload. The essence of the approach is to commit protocol (2PC) [9]. In general cases, 2PC is required minimize the overhead of transactional concurrency control to ensure atomicity for processing distributed transactions. and coordination across the whole system. A second goal Note that 2PC by itself does not ensure serializable histories. is to provide a unified extensible abstraction for determin- A distributed concurrency control augments it to guarantee istic transaction processing that seamlessly admits various serializable execution of transactions. Therefore, the research configurations (e.g., speculative execution, conservative exe- questions for this problem are as follows: Can we reduce cution, serializable isolation, and read-committed isolation). the cost of commitment in distributed transaction processing To lay the foundations for describing the queue-oriented protocols? What conditions are needed to avoid using the costly transaction processing paradigm, we start by describing the 2PC-based protocol? transaction fragmentation model. Fortunately, in many useful and practical cases, we can do away with 2PC. The work on deterministic transaction 3.1 Transaction Fragmentation Model processing protocols has demonstrated that. The next sec- tion describes how determinism is a step toward overcom- Now, I briefly describe the transaction fragmentation model. ing this obstacle. However, proposed deterministic transac- For more formal specification of this model, I refer the read- tion processing protocols suffer from in-efficiencies. Another ers to [17]. In this model, a transaction is broken into frag- step toward eliminating these in-efficiencies is the proposed ments containing the relevant transaction logic and aborting queue-oriented paradigm, which addresses the following ad- conditions. A fragment can perform multiple operations on ditional research questions: What is the best way to abstract the same record, such as read, modify, and write operations. deterministic transaction processing? Is it possible to provide A fragment can cause the transaction to abort, and in this a unified framework that unifies transaction processing for case, we refer to such fragments as abortable fragments. In centralized and distributed transaction processing? Table 1, a summary is provided on the kinds of dependencies that may exist among fragments. 3.2 Queue-oriented Transaction Processing 2.3 Potentials and Limitations of Determinism The essence of this paradigm is to process batches of transac- Work on deterministic transaction processing protocols has tions in two deterministic phases. Figure 1 depicts the basic demonstrated great potential for improving the performance flow. The first phase is called a planning phase, where threads of transaction processing systems [2]. In distributed trans- deterministically create queues tagged with deterministic action processing systems, recently proposed deterministic priorities containing transaction fragments. Dependencies approaches almost eliminates the need to perform a costly among fragments are not shown in Figure 1. The dependency 2PC protocol [18]. In other words, they rely on commit pro- information is maintained in a shared lock-free and thread- tocols that minimize overhead for committing a distributed safe distributed data structure. In the second execution phase, transaction because they perform agreement ahead of time, execution threads receive their assigned queues (filled with which avoids aborting transactions for non-deterministic fragments) and use the tagged priorities to determine the reasons (e.g., deadlocks, validation, or node failures). processing order of queues from different