Serializable Snapshot Isolation for Replicated Databases in High-Update Scenarios
Hyungsoo Jung† Hyuck Han∗ Alan Fekete† Uwe Rohm¨ † University of Sydney Seoul National University University of Sydney University of Sydney
†{firstname.lastname}@sydney.edu.au ∗[email protected]
ABSTRACT pects of this work. Among these is the value of Snapshot Many proposals for managing replicated data use sites run- Isolation (SI) rather than serializability. SI is widely avail- ning the Snapshot Isolation (SI) concurrency control mech- able in DBMS engines like Oracle DB, PostgreSQL, and Mi- anism, and provide 1-copy SI or something similar, as the crosoft SQL Server. By using SI in each replica, and deliver- global isolation level. This allows good scalability, since only ing 1-copy SI as the global behavior, most recent proposals ww-conflicts need to be managed globally. However, 1-copy have obtained improved scalability, since local SI can be SI can lead to data corruption and violation of integrity con- combined into (some variant of) global 1-copy SI by han- straints [5]. 1-copy serializability is the global correctness dling ww-conflicts, but ignoring rw-conflicts. This has been condition that prevents data corruption. We propose a new the dominant replication approach, in the literature [10, 11, algorithm Replicated Serializable Snapshot Isolation (RSSI) 12, 13, 23, 27]. that uses SI at each site, and combines this with a certifi- While there are good reasons for replication to make use cation algorithm to guarantee 1-copy serializable global ex- of SI as a local isolation level (in particular, it is the best ecution. Management of ww-conflicts is similar to what is currently available in widely deployed systems including Or- done in 1-copy SI. But unlike previous designs for 1-copy se- acle and PostgreSQL), the decision to provide global SI is rializable systems, we do not need to prevent all rw-conflicts less convincing. Global SI (like SI in a single site) does not among concurrent transactions. We formalize this in a theo- prevent data corruption or interleaving anomalies. If data rem that shows that many rw-conflicts are indeed false pos- integrity is important to users, they should expect global (1- itives that do not risk non-serializable behavior. Our pro- copy) serializable execution, which will maintain the truth posed RSSI algorithm will only abort a transaction when it of any integrity constraint that is preserved by each trans- detects a well-defined pattern of two consecutive rw-edges action considered alone. in the serialization graph. We have built a prototype that Thus in this paper, we propose a new replication algo- integrates our RSSI with the existing open-source Postgres- rithm that uses sites running SI, and provides users with a R(SI) system. Our performance evaluation shows that there global serializable execution. is a worst-case overhead of about 15% for getting full 1- Recently, Bornea et al. [6] proposed a replica management copy serializability as compared to 1-copy SI in a cluster of system that uses sites running SI, and provides users with a 8 nodes, with our proposed RSSI clearly outperforming the global serializable execution. They use an essentially opti- previous work [6] for update-intensive workloads. mistic approach (following the basic principles from [17]) in which each transaction is performed locally at one site, and then (for update transactions) the writeset and readset are 1. INTRODUCTION transmitted (by a totally ordered broadcast) to remote sites In 1996, a seminal paper by Gray et al. [15] showed that where a certification step at the end of the transaction can there were performance bottlenecks that limited scalability check for conflicts between the completing transaction and of all the then-known approaches to managing replicated all concurrent committed transactions; if a conflict is found, data. These approaches involved a transaction performing the later transaction is aborted. All the designs that offer SI reads at one site, performing writes at all sites, and ob- as global isolation level, must do essentially the same checks taining locks at all sites. This led to a busy research area, to prevent ww-conflicts; the extra work for 1-copy serializ- proposing novel system designs to get better scalability. In ability lies in preventing rw-conflicts. Bornea et al. follow 2010, in a reflection [19] on winning a Ten-Year-Best-Paper this approach in a middleware-based design, with readsets award, Kemme and Alonso identified some important as- calculated by the middleware from analysis of SQL text, in order to reduce transmission cost. We refer to this conflict management approach, of aborting an update transaction Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are when it has a rw- or ww-conflict with a committed concur- not made or distributed for profit or commercial advantage and that copies rent transaction, as the “conflict-prevention with read-only bear this notice and the full citation on the first page. To copy otherwise, to optimisation” approach (abbreviated as CP-ROO). Our pa- republish, to post on servers or to redistribute to lists, requires prior specific per offers a more subtle approach, that looks at pairs of permission and/or a fee. Articles from this volume were invited to present consecutive conflict edges in order to reduce the number of their results at The 37th International Conference on Very Large Data Bases, unnecessary aborts; as a trade-off however, we must also cer- August 29th - September 3rd 2011, Seattle, Washington. Proceedings of the VLDB Endowment, Vol. 4, No. 11 tify read-only transactions, and so our technique is suited to Copyright 2011 VLDB Endowment 2150-8097/11/08... $ 10.00.
783 cases of high update rate (as we demonstrate in the evalua- Snapshot Isolation. Snapshot Isolation (SI) is a method tion). These cases are not well treated by existing replication of multiversion concurrency control in a single DBMS engine strategies which focus instead on read-mostly environments, that ensures non-blocking reads and avoids many anomalies. and they scale badly when updates are common. SI was described in Berenson et al. [5] and it has been im- A similar issue was previously studied in single-site DBMS plemented in several production systems including Oracle concurrency control. Theory from Adya [1] as extended by DB, PostgreSQL and Microsoft SQL Server. When SI is the Fekete et al. [14] showed that checking each rw-conflict in an concurrency control mechanism, and each transaction Ti be- SI-based system gives many unnecessary aborts (“false pos- gins its execution, it gets a begin timestamp bi. Whenever itives”) in situations where non-serializable execution will Ti reads a data record x, it does not necessarily see the lat- not arise even in presence of a rw-conflict; instead, non- est value written to x; instead Ti sees the version of x that serializable execution only arises from a pair of consecutive was produced by the last to commit among the transactions rw-conflicts within a cycle in the serialization graph. Cahill that committed before Ti started and also modified x (there et al. [7] used this to propose Serializable Snapshot Isola- is one exception to this: if Ti has itself modified x, it sees tion (SSI), a concurrency control mechanism that is a mod- its own version). Owing to this rule, Ti appears to run on a ification of an SI-based (single site) engine. SSI guarantees snapshot of the database, storing the last committed version serializable execution, with performance close to that of SI of each record at bi. SI also enforces another rule on writes, itself (note that a pair of consecutive edges is much less fre- called the First-Committer-Wins (FCW) rule: two concur- quent than a single edge, so SSI has far fewer aborts than a rent transactions that both modify the same data record standard optimistic approach). In this paper we do a similar cannot both commit. This prevents lost updates in a single thing for a replicated system, with SI as the mechanism in site DBMS. each site, and 1-copy serializability as the global property of The theoretical properties of SI have been studied ex- executions. We prevent ww-conflicts in the same way as 1- tensively. Berenson et al. [5] showed that SI allows some copy SI designs, and we prevent certain pairs of consecutive non-serializable executions. Adya [1] showed that in any rw-conflicts. non-serializable execution of SI, the serialization graph has Contributions of This Work. We prove a new the- a cycle with two consecutive rw-edges. This was extended orem that gives a sufficient condition for 1-copy serializ- by Fekete et al. [14], who found that any non-serializable able execution, based on absence of ww-conflicts between execution had consecutive rw-edges where the transactions concurrent transactions, and absence of certain patterns of joined by the edge were concurrent with one another. Our consecutive rw-conflicts. We propose a concurrency con- new theorem is similar in format to those in [1, 14], but trol algorithm that applies the theorem, to abort trans- with different side-conditions that are suited to a replicated actions during a certification step when these situations are system. found. We design and implement a prototype system This theory has been used to enforce serializability based called Postgres-RSSI, using this algorithm, as a modification on SI mechanisms. In Fekete et al. [14] and in Alomari of the existing Postgres-R(SI) open source replication sys- et al. [2], compile-time analysis of conflicts was used to tem. Our prototype uses a combination of existing ideas, to modify the application code so that all executions are seri- be quite efficient. In particular, we use a carefully-arranged alizable. Cahill et al. [7] instead proposed a modified mech- mechanism for total-order broadcast of the transaction com- anism, called Serializable Snapshot Isolation (SSI), that can pletion messages, including the writesets, and dependency be coded in the DBMS engine; this checks at run-time for information about rw-conflicts. That is, we avoid send- the consecutive rw-edges, and aborts a transaction as needed ing potentially large readsets in messages. We measure to prevent non-serializable executions. Being in a single the performance of our prototype and show that there DBMS, [7] can detect conflict edges directly in lock tables, is not too much degradation compared to (non-serializable) while (being in a system with replicas) we need to transmit designs like Postgres-R(SI), and that (when updates trans- information about writesets from node to node. actions are frequent) we do significantly better than a design Replication. The traditional approach to replica man- that prevents all rw-conflicts between update transactions. agement involves read one copy, write all copies (ROWA). Paper Road Map. In Section 2 we summarize related Gray et al. [15] identified a series of performance hazards research. Section 3 has the theorem that is a sufficient con- that locking-based database replication could face. This dition for global serializability, and we give the correspond- inspired much work on system designs to reduce the per- ing concurrency control algorithm; in Section 4 we describe formance bottlenecks. These are classified by three design the details of the efficient certification, without transmitting criteria; whether an approach is based on changes to the readsets. Section 5 presents the implementation details of database kernel or by middleware that uses (nearly) unmod- our prototype, and in Section 6 we evaluate its performance. ified single node DBMS engines for the replicas; whether up- Section 7 concludes. dates are allowed at any site, or only at a “master” primary Acknowledgements. This work was supported by re- site, and whether transaction propagation to other replicas search grant DP0987900 from the Australian Research Coun- is done eagerly (as part of commit processing) or lazily after cil. the commit. Kemme et al. [18] proposed a kernel-based, multi-master and eager replication protocol, which makes 2. BACKGROUND use of an available group communication system to guaran- tee a total order delivery which allows sites to remain con- The field of transaction management, and its subfield of sistent (so, 1-copy serializability can be ensured). Improved replica management, is extensive; authoritative coverage can performance came from doing the database processing at be found in the texts by Weikum and Vossen [26] and by one site only, and applying efficient primary-keyed write op- Kemme et al. [20], respectively. This section points only to erations at other sites [17], with a certification to prevent some especially close work on which we build.
784 conflicts, that was done consistently at all sites. Using sites vidual nodes are using SI, they keep versions of each item. that provide SI as concurrency control, and giving one-copy We use the notation that Xi represents the version of item SI rather than serializability, allows a great improvement X produced by user transaction Ti (during the user-level in performance, since only ww-conflicts need be considered event Wi(Xi)). When Ti reads item X, it will return the in certification. This was done by Wu et al. [27], and the value from some version, produced by some transaction. If implementation of this, called Postgres-R(SI) has continued the read by Ti returns the version produced by Tj , we write as an open-source project. It serves as the baseline of our this event as Ri(Xj ). We also include in the history the prototype system. beginning and the completion (commit or abort) event for Most recent work has focused on middleware designs, as each transaction; we write bi for the event when Ti starts explained in [19]. This began with MIDDLE-R [24] which executing, ci for the commit point of Ti, and ai for its abort supports 1-copy serializability for sites using locking, de- point. The version order for versions of item X will be the pending on a priori knowledge on which data partitions a order of the commit events of the transactions that produce particular application program would access. Lin et al. [23] the versions. provide 1-copy SI as the global isolation level, for sites us- We associate three important numbers, which we call times- ing SI locally; this approach allows the middleware to ig- tamps, with each transaction. The timestamps in our sys- nore the readsets. Various session properties are given by tem are counters, which “tick” once at each transaction that different designs, for example Elnikety et al. [13] provide attempts to commit. We use the symbol ≺ for “less than” GSI, in which the snapshot of transaction T may be earlier among timestamps (or when equality is allowed). Be- than that containing all transactions that commit before T cause our architecture sends transaction completion through starts. Improved performance came with moving durability a total-order broadcast mechanism, the commit-timestamp to the middleware [11], and adding load balancing and fil- of any transaction is just its position in the broadcast total tering [12]. Daudjee et al. [10] proposed a lazy-propagation order. We write commit(Ti) for the commit-timestamp of design that ensures a strong session property; they evaluated Ti. The begin-timestamp begin(Ti) captures the most recent its correctness using simulation and compared the costs to commit that has occurred at the site where Ti runs, when designs giving other variants of SI. A simpler design is possi- Ti starts to execute. Because each site uses SI, this means ble with a master-slave design as in Ganymed [25] where all that each read in Ti sees the version that includes all writes updates happen at one primary site; this has other benefits by transactions whose commit-timestamp is up to, and in- in allowing heterogeneity in the hardware and software of cluding, the begin-timestamp of Ti. Clearly begin(Ti) ≺ the slaves. commit(Ti). A new work by Bornea et al. [6] achieves 1-copy serial- We also define a latest-snapshot-version timestamp lsv(Ti), izability from sites with SI locally. This is a middleware which gives the highest commit-timestamps among the trans- design, where the middleware uses program analysis to de- actions whose versions of items are accessed by Ti. If we termine the predicates involved in SELECT statements, as simplify by ignoring blind writes, so each transaction reads a proxy for the readsets. The certification step, like that an item before writing it, this is just the maximum commit- in [24], uses an approach where any detected ww-edge or timestamp among the versions that are read by Ti. Notice rw-edge among update transactions leads to the abort of that lsv(Ti) begin(Ti), but the equality may or may not one of the transactions involved. The main differences be- occur, depending on whether there is any item in the in- tween our new work and [6] are the overall architecture (ker- tersection of readset(Ti) and writeset(Tj ), where Tj is the nel modifications with a broadcast that is integrated with transaction such that commit(Tj ) = begin(Ti), that is, Tj is the dependency checks, rather than a middleware layer that the most recently committed transaction at the node when adds a separate certifier site and a proxy to act as a ve- Ti starts running. neer over each unmodified replica DBMS), the way read- As usual in multiversion concurrency theory, we consider sets are obtained (modification of the kernel to remember a serialization graph SG(h) with three types of dependency ww which records are read, rather than predicate analysis of edges. We have a ww-dependency edge Ti −→ Tj when there SQL code), and the cases where a transaction must abort to is a data item X, such that Ti writes a version Xi, and Tj avoid non-serializable execution (two consecutive rw-conflict writes a version Xj that is later in the version order than edges among arbitrary transactions, rather than a single wr Xi. We have a wr-dependency edge Ti −→ Tj when there conflict edge among update transactions). is a data item X such that Tj contains the event Rj (Xi), so that Tj reads a version produced by Ti. Finally, we have rw 3. ENSURING 1-COPY SERIALIZABILITY an anti-dependency (also called rw-dependency) Ti −→ Tj , In the following, we describe a concurrency control al- when there is an item X such that Ti reads a version Xk gorithm that provides 1-copy serializable executions for a which is earlier in the version order than Xj written by Tj . replicated system where each node runs at SI. In particular, That is, any serialization must place Ti ahead of Tj , since we developed a new theorem that gives a sufficient condition Ti did not see the effects of Tj on item X. for 1-copy serializable execution. We also give the high-level Within the category of anti-dependency edges, we give account of our algorithm, that uses the theorem as its foun- special attention to those which form what we call a de- dation. scending structure. This is formed in certain cases when there are three transactions connected by successive anti- 3.1 Notation dependency edges. For simplicity, we assume that all events in the distributed system occur in a total order; effectively, we assume that Definition 3.1 (Descending structure). Consider some logical clock is operating. Thus the history of the three transactions Tp, Tf , and Tt. We say that these form system can be given as a sequence of events. Since the indi- a descending structure, with Tp as peak-transaction, Tf as
785 follow-transaction, and Tt as trailing transaction, when there that Adya [1] already showed that the absence of consecu- are the following relationships: tive rw-edges is sufficient for serializability in systems based on SI (and the result holds immediately for GSI as well); • there are anti-dependency edges rw rw the novelty of condition no-descending-structure lies in the Tp −→ Tf , and Tf −→ Tt, extra aspects concerning the lsv timestamps. • lsv(Tf ) lsv(Tp), and 3.3 Concurrency Control Algorithm
• lsv(Tt) lsv(Tp) One can use Theorem 3.1 for a family of concurrency con- trol algorithms that each achieves 1-copy serializability from 3.2 Theory replicas that provide SI. We follow the standard model for Our concurrency control algorithm is based on enforcing ROWA-replication, with each transaction submitted at one a sufficient condition for 1-copy serializability, expressed in node, and performed locally there; then the modifications the following theorem. made are sent to all replicas, in a consistent order. At each replica, a certification check occurs, and if the trans- Theorem 3.1. Let h be a history over a set of transac- action can commit, its modifications are installed through tions, such that one can assign to each transaction Ti an a “remote transaction”. Certification is done using Theo- event snapshot i that precedes bi, obeying the following three rem 3.1: a transaction will be allowed to commit provided conditions. that first-committer-wins and no-descending-structure are both valid.1 To implement this idea, we need to make sure • Read-from-snapshot If h contains Ri(Xj ) then h that at certification, each node has the information it needs. contains c preceding snapshot , and whenever h con- j i For checking first-committer-wins, the node needs informa- tains Wk(Xk) for some committed transaction Tk 6= tion on the writeset of the current transaction, and the write- Tj , then either ck precedes cj , or else ck follows snap- sets of the previously committed ones. This is the same shot (Ti). check as done for replication that gives GSI as the isolation level (as in [27, 13]), and the information is naturally avail- • First-committer-wins If h contains both Wi(Xi) and able since the writesets need to be propagated anyway for Wj (Xj ), then it is not possible that ci occurs between installation at remote nodes. snapshot(Tj ) and cj . The core challenge, in finding an efficient implementation • No-descending-structure The serialization graph of an algorithm in the family based on Theorem 3.1, is the SG(h) does not contain a descending structure among management of readset information and lsv-timestamps, so transactions that all commit. that certification can check for a descending structure. We have modified the kernel to track the set of ids of items that Then h is 1-copy serializable. are read by each transaction.2 A naive approach would cap- ture the readset and propagate it in the broadcast comple- For a proof of Theorem 3.1, readers are referred to Ap- tion message (just like the writeset) but this is unrealistic in pendix A, but we remark on how Theorem 3.1 relates to practice, since readsets may be extremely large, and the cost other results. of sending them around would be excessive. For example, The read-from-snapshot condition deals with wr-edges. consider a transaction which calculates an aggregate over a it holds automatically in any system that runs SI at each large table; the readset is every row in the table. Instead, replica, sends each transaction to be performed at one replica, the algorithm that we present in this paper keeps readset in- and then propagates the updates to other replicas in commit- formation only at the local node where a transaction runs. order (whether this order comes from a total-order broad- As each node has readsets for the transactions that are lo- cast, or by having all updates at a single primary site). This cal at a node, it can determine rw-edges involving any other is because we can choose the event snapshot(T ) to immedi- i transaction when learning of the completion of that other ately follow the commit of the last transaction whose com- transaction. It is this set of edges that we will share with mit was received at the appropriate node before T begins its i each node for use in certification. This means that we do not execution. That is, the timestamp begin(T ) is the highest i simply use a black-box total-order broadcast to propagate a commit-timestamp of any transaction that commits before transaction completion message; instead, we integrate shar- the event snapshot . i ing the rw-edges with the ordering and broadcast, so that The first-committer-wins condition deals with ww-edges. as part of the communication, the rw-edges can be collected The isolation level GSI (generalized snapshot isolation, [13]) from all nodes. Fortunately, the LCR broadcast protocol is defined exactly as the combination of condition read- [16] visits each node around a ring structure, when infor- from-snapshot and first-committer-wins; thus systems like mation can be collected, before visiting each node again to Postgres-R(SI) or Tashkent, which offer GSI, do so by en- deliver the message in order. Thus we have designed our forcing the first-committer-wins check in a certification step, system with a component GDCP, described in detail be- based on knowledge of each transaction’s writeset. To ensure serializability, one must also worry about read- 1We remark that when a descending structure is found, the sets and how they lead to rw-edges. The recent work [6] transaction that aborts will always be whichever completes uses a condition called “no read impact” that constrains last among the three; this could be the peak, the follow each rw-edge. It states that if h contains R (X ), then it or the tail transaction. This contrasts with the centralized j i algorithm of Cahill et al. [7] which tries to abort the middle is not possible that ci occurs between snapshot(Tj ) and cj . “pivot” of the three. The power of our no-descending-structure condition is that 2In our implementation, we only track records that are actu- it looks for pairs of consecutive rw-edges, which should be ally read; to prevent phantoms, one will also need to include much less frequent than single rw-edges. In this regard, note next-keys when scanning.
786 low, that follows LCR and collects rw-edge information (as we collect incoming anti-dependency information from other well as lsv information). When the certification check occurs replicas by distributing Dt,c and Dt,p. each node has the information needed, and so each node can In a protocol message, carriers for incoming dependency test for a descending structure, and each will reach the same information have different data types (i.e., a pair Dt,c and a decision to commit (or to abort) a transaction. set Dt,p). In principle, the latest lsv of a transaction having a rw-edge to Ti is vital to our SSI theory, so a pair type 4. GLOBAL DEPENDENCY CHECKING of Dt,c satisfies the principle by carrying one pair from all committed transactions. By contrast, pending transactions One of challenges in implementing RSSI lies in the con- are not confirmed yet to be delivered (some are aborted by sistent retrieval of global dependency information among our SSI concurrency control), and this requires us to keep all concurrent transactions. We approach this problem from a candidate pairs in a set D until their delivery decision is different perspective and propose a global dependency check- t,p made. Any transaction belonging to above sets (or the pair) ing protocol (GDCP). A standalone process that carries out must have an earlier commit timestamp (or vector clock) RSSI-related work is deployed, called a replication manager. than the current transaction (line 1-5 of gdcpBroadcast). GDCP has two notable features; it does not need to prop- Once the protocol message is created, we check presence of a agate an entire readset and it also reduces the amount of descending structure. If the current transaction is involved comparisons done at each node in dependency checking. in a non-serializable case with committed transactions, then The basic principle is, for a local transaction T , a replica- i it should be aborted immediately. Otherwise, a replication tion manager sends a writeset of T , along with a list of trans- i manager starts forwarding the protocol message to its suc- action IDs that have an incoming anti-dependency from T , i cessor (line 6-10 of gdcpBroadcast). instead of sending the entire readset. A replication manager A replication manager, upon receiving a protocol message checks only the presence of an outgoing anti-dependency for a transaction T from its predecessor, executes different from its local readsets to writesets of other transactions, i routines based on the contents of the received protocol mes- and it piggybacks that information on writesets and for- sage. For convenience, we put a pair of special characters wards writesets to other replicas. This rather simple tech- (⊥, ⊥) in D whenever we detect a cycle. nique not only makes all replicas retrieve complete depen- t,c dency information, but also is pivotal in reducing redundant • Remote writeset without a cycle. If a protocol mes- dependency checking. sage has a remote writeset without a cycle mark (i.e., To ensure data consistency, replicated databases deploy Dt,c 6=(⊥,⊥)), a replication manager first checks whether uniform total order broadcasting. Our choice is to imple- Wi has either an anti-dependency from the readsets of ment the LCR protocol [16] inside a replication manager. local committed transactions, or any cycle with them. LCR is especially targeted for replicated databases and can If a replication manager finds an anti-dependency from be better for this than generic group communication sys- local committed transactions, then it updates that in- tems, Spread [3] and JGroup [4]. formation in Dt,c only if the lsv of its local committed transaction is later than the existing one, because the lat- 4.1 GDCP est lsv of Tin is the most important criterion in detecting GDCP implements LCR straightforwardly and builds the presence of a descending structure. Besides checking de- dependency checking protocol on top of LCR. Algorithm 1 pendency, if a replication manager detects a cycle, then describes the pseudo-code of GDCP, and implicit in this it marks that information on Dt,c with (⊥,⊥) and stops pseudo-code is the manipulation of vector clocks for the checking (line 3-8 of the first Rreceive). When checking uniform total order delivery (refer to [16] for details). The an anti-dependency from local pending transactions, we important invariant that GDCP preserves, is to acquire all only consider transactions that have an earlier commit anti-dependency information created from transactions with order (vector clock) than that of Ti as by the invari- an earlier total order than a transaction checked. As a ant enacted. Whenever a replication manager detects step toward such an invariant, GDCP requires four types an anti-dependency, it appends that information to Dt,p of transactional logs, two of which are subsets of the other (line 9 of the first Rreceive). sets. As shown in Algorithm 1, Tc is a transactional log for After completing dependency checking, a replication man- all committed transactions including both local and remote ager reflects all incoming and outgoing dependency in- transactions, while Tp is a log for all pending transactions. formation of Ti (Df,c, Df,p, Dt,c and Dt,p) to its local Tl,c and Tl,p are subsets of Tc and Tp, respectively. Both pending log Tp, updating its own global dependency in- point to local committed/pending transactions. For discus- formation. Then, it forwards the protocol message to its sion on dealing with garbage logs, readers are referred to successor. Consequently, the protocol message conveys Appendix C.2.3. two types of anti-dependency information, one for the The first stage of GDCP starts when a replication man- readset and the other for the writeset (line 10-11 of the ager receives a readset (Ri) and a writeset (Wi) from a local first Rreceive). database. Using these sets, a replication manager creates a • Remote writeset with a cycle. If a protocol mes- protocol message that should convey the following contents: sage has a remote writeset with a cycle mark, then a a writeset, outgoing anti-dependency information from Ri to replication manager forwards the protocol message to its the writesets of committed/pending transactions (i.e., Df,c successor immediately (line 12-13 of the first Rreceive). and Df,p from Tc and Tp) and incoming anti-dependency information from the readsets of local committed/pending • Local writeset. When a replication manager receives its own protocol message, it starts the second circulation transactions to Wi (i.e., Dt,c and Dt,p from Tl,c and Tl,p). of the protocol message with “Decide” to let other repli- Circulating Df,c and Df,p in a ring topology informs other cas make a decision for Ti based on the obtained depen- replicas of Ti’s outgoing anti-dependency information, while
787 Algorithm 1 Pseudo-code of GDCP S V Wi and Ri: a writeset and a readset of transaction Ti. 9: Dt,p ← Dt,p {(j, lsvj )|(cj ≺t ci) (Dt,c.lsv ≺t lsvj ) rw Tc: a log for committed transactions. V V (Tj 99K Ti) Tj ∈ Tl,p}; Tp: a log for pending transactions. 10: update Tp with Df,c, Df,p, Dt,c and Dt,p; Tl,c: a log for local committed transactions (Tl,c ⊂ Tc). 11: Rsend
dency information. If the protocol message contains a 5. SYSTEM ARCHITECTURE cycle mark, a replication manager sends an “abort” mes- We implemented a research prototype that integrates RSSI sage to its local database immediately. After sending the into Postgres-R, a replicated PostgreSQL server that uses SI protocol message to a successor, a replication manager as concurrency control, inspired by Wu et al. [27], and that calls gdcpDeliver to check whether the first writeset in it is maintained as an open source project. Tp has the “Decide” mark, so a replication manager can Our system design is following two design principles: firstly, decide for the writeset. The remaining replicas, upon we strive for high overall system performance (in terms of receiving an explicit “Decide” message, can make a de- throughput) by an efficient implementation of RSSI. This re- cision for remote or local writesets based on aggregated quired some non-trivial engineering efforts to minimize the dependency information. We will see how gdcpDeliver overhead of RSSI compared to traditional SI-based replica- makes a decision (line 1-6 of the second Rreceive). tion. Secondly, we are interested in a modular system design In gdcpDeliver, we consider a writeset in Tp, having the that would allow for smooth adaptation of RSSI into exist- earliest total order. If the first writeset is not marked with ing database systems. We hence implement all SSI-related “Decide”, gdcpDeliver should return without processing data structures and algorithms inside a dedicated replication other writesets queued in Tp due to the total order delivery. manager that is separated from the DBMS. This minimizes In contrast, if the writeset has the “Decide” mark, we now the changes needed to the underlying database engine while can decide whether to deliver or discard the writeset based being readily portable to any SI-based database system. Ap- on the global dependency information conveyed in the pro- pendix C gives a detailed discussion of our Postgres-RSSI tocol message. The case we abort the current transaction implementation. is when the current transaction is involved in one of non- serializable cases and other involved transactions could be 6. PERFORMANCE EVALUATION (or were already) delivered earlier than the current trans- In the following, we report on the results of an experimen- action. If the decision is made for a local writeset, a repli- tal performance analysis of RSSI, in which we quantify its cation manager sends a “commit” or “abort” message back overhead and scalability behavior as compared to state-of- to a local database. Otherwise, we send the writeset to a the-art SI-based replication approaches from the literature. local database for updating. gdcpDeliver keeps processing queued writesets having the “Decide” mark likewise until it 6.1 Evaluation Setup encounters a writeset without the mark. This ensures the The experiments were conducted on a database server uniform total order delivery (line 1-14 of gdcpDeliver). cluster with eight nodes, each with a 3.2GHz Intel Pentium For additional details of GDCP, which explains an execu- IV CPU and 2GB RAM under RedHat Enterprise Linux tion of the protocol with an example transaction schedule, version 4 (Linux kernel 2.6.9-11), and interconnected with readers are referred to Appendix B. a Gigabit Ethernet network. We compare our design with
788 10 10 WW(RSSI) two others, which use the same overall system architecture 9 10 WW(CP‐ROO) 8 (kernel-based replication and log-based writeset handling, 8 DS(RSSI) 7 8 taken from the open-source implementation of Postgres-R, RW(CP‐ROO) (x1000) (x1000)
6 6
in 6 (x1000)
with GDCP instead of Spread for total-ordered broadcast) (x1000) 5 m
n i 4 but have different conflict management algorithms. The 4 3
baseline we call RSI, using ww-conflict management for 1- Aborts/min 2 2 Txns/min copy SI [27]. The other systems give serializability: RSSI is 1 RSSI CP‐ROO RSI our system, and CP-ROO aborts every detected rw-conflict 0 0 80 160 240 320 400 480 560 640 80 160 240 320 400 480 560 640 but does not globally certify read-only transactions (this is MPL MPL the certification approach used by Bornea et al. [6]). All (a) 0%RO-100%W databases are configured with default settings. 12 10 10 WW(RSSI) 10 WW(CP‐ROO) 6.2 TPC-C Workload 8 8 DS(RSSI) 8 (x1000) As a starting point, we ran a series of experiments based (x1000)
6 RW(CP‐ROO)
3 in 6 (x1000) on the TPC-C++ benchmark in order to evaluate the ef- (x1000) 6
m n i fectiveness and scalability of the different certification rules. 4 4
As it turns out, such a realistic workload has so few SI Aborts/min 2 Txns/min 2 anomalies so that all three approaches achieve the same RSSI CP‐ROO RSI performance and scalability with no measurable difference, 0 0 80 160 240 320 400 480 560 640 80 160 240 320 400 480 560 640 regardless whether they provided 1-copy SI or 1-copy seri- MPL MPL alizability (as does our RSSI). We have placed these results (b) 25%RO-75%W in Appendix D.2. 18 7 WW(RSSI) 16 7 6 WW(CP‐ROO) 6.3 Synthetic Workload 14 6 DS(RSSI) 12 5 (x1000) Next, we want to focus on the performance impact of our (x1000) 5 RW(CP‐ROO)
4
10 in (x1000)
RSSI concurrency control, especially comparing the over- (x1000) 4 m
n
i 8 3 head of RSSI and CP-ROO approaches. We hence created 6 2 a simple synthetic micro-benchmark called ssibench with 4 Aborts/min Txns/min numerous read-write conflicts between a query transaction 2 RSSI CP‐ROO RSI 1 and an update transaction, or among update transactions. 0 0 80 160 240 320 400 480 560 640 80 160 240 320 400 480 560 640 Appendix D.1.1 gives detailed explanation of ssibench. MPL MPL In the following, we use a fixed number of 8 replicas and (c) 50%RO-50%W we vary the multiprogramming level from 80 to 640, with 30 5 all clients trying to execute transactions as fast as possible WW(RSSI) 5 25 WW(CP‐ROO) without think time in between. All plotted points in Fig- 4 DS(RSSI) ure 1 are the average of 5 runs, each run consists of 1 minute 20 4 (x1000) (x1000)
3 RW(CP‐ROO)
in 3 (x1000)
ramp-up period and 1 minute measurement period. We vary (x1000) 15 m
n i the portion of read-only transactions from 0% up to 100% to 10 2
explore performance characteristics for RSSI and CP-ROO. Aborts/min 1
Txns/min 5 In RSSI, all transactions can be involved in a cycle so that RSSI CP‐ROO RSI any of them can be a victim when non-serializable cases 0 0 80 160 240 320 400 480 560 640 80 160 240 320 400 480 560 640 are detected. Meanwhile, CP-ROO is only concerned with MPL MPL rw-edges among update and insert transactions. Obviously, (d) 75%RO-25%W RSSI has an advantage of detecting non-serializable cases in a finer-grained way while it has the overhead of check- 120 ing upon all types of transactions. In contrast, CP-ROO 100 has a merit of allowing all read-only transactions to com- mit without certification, but it has a shortcoming of seeing 80 (x1000)
(x1000) only a single rw-edge, instead of multiple edges. Our exper- 60 imental results show a performance spectrum that captures n representative characteristics of the two concurrency control 40 algorithms. Note that in all experiments, except the 100%
Txns/min 20 read-only workload, RSI allows executions that are not se- RSSI CP‐ROO RSI rializable. 0 Figure 1-(a) shows results under 100% write-intensive work- 80 160 240 320 400 480 560 640 load, where all transactions in all concurrency control al- MPL gorithms have to pass through GDCP for certification. In (e) 100%RO-0%W this scenario, we could measure serializable concurrency con- Figure 1: Throughput and abort rates of ssibench trol overhead and compare false positive abort rate between with three tables, each has 100K items, at RSI, RSSI and CP-ROO. Throughput graphs of RSI and RSSI RSSI and CP-ROO. Abort causes are denoted as slowly increase as MPL grows, whereas CP-ROO is already WW (ww-conflict), DS (descending structure) and RW 3TPC-C++ has the same schema as TPC-C but adds a new (read impact). RO and W stand for read-only and transaction type Credit Check to induce SI-anomalies [7]. write transactions.
789 saturated and decreased slowly. When MPL is 80, the abort serializability for replicated databases with no performance rate of CP-ROO is three times higher than RSSI; most degradation. aborts are due to rw-conflicts (i.e., read impact in [6]). As MPL increases up to 640, we can observe the widening per- 8. REFERENCES formance gap between RSSI and CP-ROO; CP-ROO still [1] A. Adya. Weak consistency: a generalized theory and has very high abort rate at MPL=640, while RSSI could optimistic implementations for distributed transactions. PhD thesis, MIT Lab for Computer Science, 1999. lessen the effect of false positive abort since it checks pres- [2] M. Alomari, A. Fekete, and U. R¨ohm.A robust technique to ence of the descending structure. In a write-intensive work- ensure serializable executions with snapshot isolation DBMS. load, which has cycles and all types of conflicts, the finer- In Proccedings of ICDE 2009, pages 341–352, 2009. grained dependency checking of RSSI could be much better [3] Y. Amir, C. Danilov, M. Miskin-Amir, J. Schultz, and J. Stanton. The Spread Toolkit: Architecture and Performance. Technical than the coarser-grained checking of CP-ROO. Report CNDS-2004-1, Johns Hopkins University, April 2004. In Figure 1-(b), (c) and (d), we run experiments under [4] B. Ban. JGroup – a Toolkit for Reliable Multicast workloads that have varying proportions of read-only trans- Communication. http://www.jgroup.org, 2007. actions. When read-only transactions are involved, we can [5] H. Berenson, P. Bernstein, J. Gray, J. Melton, E. O’Neil, and P. O’Neil. A critique of ANSI SQL isolation levels. In see the trade-off of merits and shortcomings of two meth- Proceedings of SIGMOD 1995, pages 1–10, 1995. ods; in particular, as the portion of read-only transactions [6] M. A. Bornea, O. Hodson, S. Elnikety, and A. Fekete. One-copy increases, we could see a cross-over point where CP-ROO serializability with snapshot isolation under the hood. In starts to outperform RSSI. With 25% read-only, Figure 1- Proceedings of ICDE 2011, pages 625–636, 2011. [7] M. J. Cahill, U. R¨ohm,and A. Fekete. Serializable isolation for (b) shows that CP-ROO narrows the performance gap with snapshot databases. ACM TODS, 34(4):1–42, 2009. RSSI by committing read-only transactions immediately. The [8] M. Castro and B. Liskov. Practical Byzantine fault tolerance throughput of CP-ROO also scales slowly as MPL grows. and proactive recovery. ACM TOCS, 20(4):398–461, 2002. As we increase the portion of read-only transactions up to [9] E. Cecchet, G. Candea, and A. Ailamaki. Middleware-based database replication: the gaps between theory and practice. In 50% (i.e., Figure 1-(c)), both RSSI and CP-ROO deliver Proceedings of SIGMOD 2008, pages 739–752, 2008. almost the same throughput although CP-ROO still has a [10] K. Daudjee and K. Salem. Lazy database replication with higher abort rate than RSSI. Here the performance gain ob- snapshot isolation. In Proc. of VLDB06, pages 715–726, 2006. tained by the advantage of RSSI balances the overhead of [11] S. Elnikety, S. Dropsho, and F. Pedone. Tashkent: uniting durability with transaction ordering for high-performance the algorithm. Figure 1-(d) shows that the optimization scalable database replication. In EuroSys, pages 117–130, 2006. of CP-ROO, that avoids checks for read-only transactions, [12] S. Elnikety, S. Dropsho, and W. Zwaenepoel. Tashkent+: is beneficial when the workload is dominated by read-only memory-aware load balancing and update filtering in replicated transactions; here CP-ROO outperforms RSSI whose certi- databases. In Proc. of EuroSys 2007, pages 399–412, 2007. [13] S. Elnikety, W. Zwaenepoel, and F. Pedone. Database fication overhead for read-only transactions is evident. We replication using generalized snapshot isolation. In Proceedings note that CP-ROO still has a higher abort rate for write of SRDS 2005, pages 73–84, 2005. transactions than RSSI. [14] A. Fekete, D. Liarokapis, E. O’Neil, P. O’Neil, and D. Shasha. Making snapshot isolation serializable. ACM TODS, In Figure 1-(e), we measured the throughput when the 30(2):492–528, 2005. workload consists of only read-only transactions, where all [15] J. Gray, P. Helland, P. O’Neil, and D. Shasha. The dangers of algorithms are serializable under this scenario. In this ex- replication and a solution. In SIGMOD, pages 173–182, 1996. periments, RSI and CP-ROO show almost the same perfor- [16] R. Guerraoui, R. R. Levy, B. Pochon, and V. Qu´ema. Throughput optimal total order broadcast for cluster mance since no certification is needed, but RSSI still has environments. ACM TOCS, 28(2):1–32, 2010. the certification overhead and shows the overhead in the fig- [17] B. Kemme and G. Alonso. Don’t be lazy, be consistent: ure accordingly. The large performance gap when MPL is Postgres-R, a new way to implement database replication. In small (i.e., 80) is because queries and readsets are not fully Proceedings of VLDB 2000, pages 134–143, 2000. [18] B. Kemme and G. Alonso. A new approach to developing and pipelined yet in this stage. As MPL grows, readsets fully implementing eager database replication protocols. ACM saturate the GDCP network and this narrows the perfor- TODS, 25(3):333–379, 2000. mance gap with other two algorithms. [19] B. Kemme and G. Alonso. Database replication: a tale of Appendix D.1.2 provides the breakdown of GDCP over- research across communities. PVLDB, 3(1):5–12, 2010. [20] B. Kemme, R. Jim´enez-Peris, and M. Pati˜noMart´ınez. head under the same synthetic workloads. Database Replication. Morgan and Claypool, 2010. [21] R. Kotla, L. Alvisi, M. Dahlin, A. Clement, and E. Wong. Zyzzyva: Speculative Byzantine fault tolerance. ACM TOCS, 7. CONCLUSION 27(4):1–39, 2009. This paper presented a new approach to global serializable [22] L. Lamport, R. Shostak, and M. Pease. The Byzantine generals execution over replicated databases using SI. We introduced problem. ACM TOPLAS, 4(3):382–401, 1982. (and proved) a clear criterion that is sufficient for 1-copy se- [23] Y. Lin, B. Kemme, M. Pati˜noMart´ınez,and R. Jim´enez-Peris. Middleware based data replication providing snapshot isolation. rializability, based on absence of ww-conflicts and on absence In Proceedings of SIGMOD 2005, pages 419–430, 2005. of certain patterns of two consecutive rw-conflicts. Based on [24] M. Pati˜noMart´ınez,R. Jim´enez-Peris, B. Kemme, and this theorem, we developed our RSSI algorithm that allows G. Alonso. MIDDLE-R: Consistent database replication at the to efficiently check for the absence of those transactional middleware level. ACM TOCS, 23(4):375–423, 2005. [25] C. Plattner, G. Alonso, and M. T. Ozsu.¨ Extending DBMSs patterns in replicated databases. We incorporated our ap- with satellite databases. VLDB Journal, 17(4):657–682, 2008. proach into the Postgres-R(SI) system. RSSI was shown [26] G. Weikum and G. Vossen. Transactional Information to perform very well in a variety of cases with high update Systems: Theory, Algorithms, and the Practice of rates, by avoiding unnecessary aborts from false positives Concurrency Control and Recovery. Morgan Kaufmann, 2001. [27] S. Wu and B. Kemme. Postgres-R(SI): Combining replica as found in the CP-ROO approach that prevents single rw- control with concurrency control based on snapshot isolation. edges. For a TPC-C workload, RSSI can achieve 1-copy In Proceedings of ICDE 2005, pages 422–433, 2005.
790 Decide c1 < c2 < c3 T1 rw 1 T2 rw lsv3 < lsv2 < lsv1 1 l,p 1 - 1 T1 T2 l,p 1 - l,p 1 - C r(1) 3 2 R1 T 3 2 3 2 1 r,p r,p rw r,p w 2 1 T1 T2 rw r 2 1 T1 T2 R2 r(2) <2,-,-,->3 T2 R1 <1,-,-,->2 <-,-,-,-> w(1) R1 1 R1 w r <-,-,-,-> <2,-,-,-> R3 R2 1 R2 3 <1,-,(1,lsv1),->2 R3 T3 R3 Decide R2 R3 Decide w(2) - 2 2 T2 rw T3 - 3 - rw l,p l,p 2 Tl,p 3 l,p 2 l,p 3 rw D 2 T2 T3 Time 2 1 2 1 2 T2 T3 l,p 2 D <1,-,-,->2 1 T1 rw T2 2 1 2 1 <-,-,-,-> 1 - 1 - 1 2 1 <2,-, (2,lsv2),->3 rw Example schedule r,p r,p Tr,p T2 rw 1 - T1 T2 1 - r,p 3 2 T2 rw T3 r,p T3 1 - r,p 3 2 T2rw T3 (a) Round 1 (b) Round 2 (c) Round 3 Decide Decide T1 rw T2 1 T1 rw T2 1 1 l,p 1 - C l,p - C Legend: 3 2 T2 rw T3 D 3 2 T2 rw T3 A (local) r,p r,p T2 rw T3 T2 rw T3 RS 2 1 2 1 rw C T1 rwT2 T1 T2 TID WS (remote) <2,-,(2,lsv2),``Decide >3 <1,-,(1,lsv ),``Decide > TID WS R1 1 2 R1 <2,-,(2,lsv2),``Decide >3 (GDCP msg) <-,-,-,``Decide >1 R3 R2 Decide R3 R2
Figure 2: Global dependency checking protocol (GDCP) with an example schedule.
APPENDIX written by Tj . That is, Ti does not see the version Xj . By condition read-from-snapshot, we see that snapshot i pre- A. PROOF OF THEOREM 3.1 cedes cj . That is, in terms of timestamps, begin(Ti) ≺ In order to prove Theorem 3.1, we first establish a lemma commit(Tj ). about the relationships of the various timestamps along the three types of edges. Proof Of Theorem 3.1. For contradiction, suppose that h is not 1-copy serializable. Thus there must exist a cy- Lemma A.1. Let h be a history over a set of transac- cle among the committed transactions in the serialization tions, such that one can assign to each transaction Ti an graph SG(h). Consider in this cycle, the transaction Tp event snapshot i preceding bi and obeying the two conditions with the highest value for lsv(Tp). In traversing the cy- reads-from-snapshot and first-committer-wins. Suppose that cle, consider the two transactions visited immediately after h contains both ci and cj , and that SG(h) contains an edge Tp, call these Tf and Tt. That is, the cycle contains edges Ti −→ Tj . Then we have Tp −→ Tf −→ Tt. wr Now, the choice of T as having the largest lsv means that • If Ti −→ Tj , then commit(Ti) lsv(Tj ) p ww lsv(Tf ) lsv(Tp), and lsv(Tt) lsv(Tp). We claim that • If Ti −→ Tj , then commit(Ti) lsv(Tj ) each of the two edges following from Tp is a rw-edge (and rw • If Ti −→ Tj , then begin(Ti) ≺ commit(Tj ). thus the pair of edges forms a descending structure, con- wr tradicting the hypothesis that the graph has no descending Proof. If Ti −→ Tj , then h contains the event Rj (Xi) structure). for some item X. That is, Xi is one of the versions read All that remains is to prove the claims. To prove that by Tj , and so Ti is among the transactions whose commit- rw Tp −→ Tf , we show that the edge cannot be ww nor wr. If timestamps are considered in forming lsv(Tj ). Since lsv is chosen as the largest timestamp among those considered, we either were the case, Lemma A.1 would give commit(Tp) lsv(Tf ), and since lsv(Tp) ≺ commit(Tp), we would deduce have commit(Ti) lsv(Tj ). ww lsv(Tp) ≺ lsv(Tf ) contradicting the choice of Tp. Now, If Ti −→ Tj , then there is an item X such that h contains rw knowing that Tp −→ Tf , we use the third part of Lemma A.1 Wi(Xi) and Wj (Xj ), and the version order places Xi before to see that begin(Tp) ≺ commit(Tf ), and therefore lsv(Tp) ≺ Xj . That is, ci precedes cj . By first-committer-wins, ci rw commit(Tf ). To prove that Tf −→ Tt, we show that the edge precedes snapshot j . Thus Xi is one of the versions accessed by Tj , and so Ti is among the transactions whose commit- cannot be ww nor wr. If either were the case, Lemma A.1 timestamps are considered in forming lsv(Tj ). Since lsv is would give commit(Tf ) lsv(Tt), and we could combine chosen as the largest timestamp among those considered, we this with the above fact lsv(Tp) ≺ commit(Tf ), to show have commit(Ti) lsv(Tj ). lsv(Tp) ≺ lsv(Tt), contradicting the choice of Tp to have rw maximal lsv in the cycle. This completes the proof of the If Ti −→ Tj , then there is an item X such that Ti reads claims, and thus the proof of the theorem. a version Xk which is earlier in the version order than Xj
791 B. GDCP DETAILS deliver) writesets either for consistent database update or for global dependency checking. The replication manager B.1 Execution of GDCP exchanges messages with the local database instance in two In order to better understand the design of GDCP let us cases: first, when the local database is ready to commit, it explain the execution of GDCP with an example schedule transfers both a readset and a writeset of a transaction to the shown in Figure 2. Suppose that there are three replicas replication manager. Upon the receipt of these tuple sets, R1, R2 and R3 with their own local transactions T1, T2 and the replication manager sends the writeset using GDCP to T3, where lsv3 ≺t lsv2 ≺t lsv1 and c1 ≺t c2 ≺t c3. The the other replicas while keeping the readset for dependency three transactions are concurrent and ready to commit. To checking. The second case is when the replication manager simplify the example, we assume the absence of committed delivers to the local database a serializable writeset for up- transactions in all replicas, so our focus is on four fields in dating. The existing PostgreSQL components do not need a protocol message;
792 resenting transaction writesets as a collection of tuple logs of Insert 2->3 PostgresSQL. This approach has several advantages over the Update statement-based writeset handling: firstly, log-based write- Update set handling is typically faster than statement-level writesets 1->2 2->3 Read Read as it bypasses any SQL handling routines; but in particular, 2 3 it guarantees deterministic updating while statement-based methods fail in cases where the original query contains an Read update operation with non-deterministic value [9]. Insert 1 1->2 Insert 3->1 C.2.2 Resolving Conflicts An important issue in replicated databases is resolving Update conflicts either between local and remote transactions, or 3->1 solely among remote writesets. Note that this issue is in- dependent of the method for handling writesets. When a conflict occurs between remote and local transactions, RSSI Figure 5: Static dependency graph for ssibench, gives higher priority to remote writesets over local ones be- with rw-conflicts shown as dashed edges and ww- cause they passed already global dependency checking; if a conflicts shown in bold. There is a wr-edge not local transaction already has an exclusive lock on a tuple shown that revereses each rw-edge. Read i means that a remote writeset needs to update, the remote transac- that a read-only transaction reads items from tion forces a local transaction to immediately abort. When ssibench-i. Update/insert transactions are denoted conflicts occur among remote writesets, our solution is to as name i → j: a name transaction reads from serialize the execution for remote writesets by their total ssibench-i and updates/inserts rows in ssibench-j. order, so no conflicting writesets can be executed concur- rently. We only allow remote writesets to run in parallel if the replication manager receives the dependency informa- their writesets do not overlap with each other. Although tion conveyed in Df,c and Df,p of remote transaction’s proto- parallel updates allow different update ordering for remote col message, or when the local replication manager updates writesets, appearing a violation of the uniform total order the log by its own local transaction during GDCP. guarantee, a mapping table from a local TID to a global The replication manager also maintains a tuple lookup ta- commit order preserves the correctness. ble to efficiently detect anti-dependencies from/to a given readset/writeset. This lookup table is organised as an in- C.2.3 Collecting Garbage Entries verted index: each tuple entry points to a linked list of pairs Since the replication manager’s data structures reside in of transaction IDs and the corresponding snapshot version memory for efficiency, we must garbage-collect entries in (lsv), which read or updated that tuple. To further improve both the transaction log and the tuple lookup table. For the lookup performance, the TID list is sorted by lsv, and cleaning up obsolete entries, we need a staleness criterion. the tuple lookup table itself is organised as a hash-based The staleness criterion indicates the entry point before which index table. The entry format of the tuple lookup table is we could recycle all preceding log entries. In a single sys- shown in Figure 4-(b). tem, the log entry for a transaction Ti can be safely removed when there is no pending transaction that starts before Ti C.1.2 Failure and Recovery commits. In distributed systems, all replication managers In this paper, we concentrate on giving an overview how have different staleness values. RSSI can recover from simple stop-failures. We expect that We avoid running consensus on a staleness point. Each more complex failure models such as Byzantine Fault Tol- replica has a bookkeeping variable to trace a staleness value, erance (BFT) [22] can be dealt with by adapting one of the so a replica sends its staleness value to all members when- existing solutions [8, 21]. If a stop-failure happens at any ever the staleness value is updated. The staleness value is replica, RSSI first shuts down the group and then recovers an earliest begin timestamp of pending transactions. Upon the replicated databases from any data inconsistency. Inside receiving a new staleness value from others, a replication a single database instance, we rely on the recovery mecha- manager takes the oldest of all those received from replicas, nism of the underlying database for data consistency, while and allows clean up of all preceding log entries before this consistent recovery across replicas is guaranteed by copying oldest point. When removing log entries, the replication data from a master replica that has the most up-to-date manager can efficiently clean up table entries by checking database among all replicas. After setting up a new ring the readset and writeset arrays of a log entry. topology, the replicas then initiate operation again. C.3 Limitation C.2 Implementation Details Although our theory can be applied to any system, the In the following, we take a closer look at some important architecture presented here may not perform well if replicas implementation details of our Postgres-RSSI prototype. are far apart (e.g., on a WAN). The high cost of propagating updates in long delay networks is the main performance bot- C.2.1 Writeset Handling tleneck. Such an embarrassingly distributed database might There are two common ways to represent and handle write- need completely different architectures and algorithms. In sets: trigger-based replication systems capture logical SQL future we will explore a practical solution for data manage- statements, while log-sniffer based approaches ship some ment services in cloud platforms, where consistency across form of log entries. RSSI follows the later approach by rep- data centers is a concern.
793 RSSI(1S) CPͲROO(1S) RSI(1S) RSSI(2S) RSSI(1S) CPͲROO(1S) RSI(1S) RSSI(2S) CPͲROO(2S) RSI(2S) RSSI(4S) CPͲROO(4S) CPͲROO(2S) RSI(2S) RSSI(4S) CPͲROO(4S) RSI(4S) RSSI(8S) CPͲROO(8S) RSI(8S) RSI(4S) RSSI(8S) CPͲROO(8S) RSI(8S) 9 25 8
7 20
6 (x100) (x100) (x1000) (x1000) 15 in
n 5 m 4 10 Txns/min
3 Aborts/min
2 5 1
0 0 5 10 20 30 40 50 60 70 80 100 120 140 160 200 240 280 320 400 480 560 640 5 10 20 30 40 50 60 70 80 100 120 140160 200 240280 320 400480 560 640
MPL MPL (a) Throughput (b) Abort rate
Figure 6: Throughput and abort rates of TPC-C++ with varying number of replicas and MPL for RSI, RSSI and CP-ROO.
D. EXPERIMENTAL DETAILS Txn Type Read Only Read-Insert Read-Update Workload Txn Time GDCP Time Txn Time GDCP TimeTxn Time GDCP Time 408 ms 514 ms N/A N/A 2397 ms 3853 ms D.1 Synthetic Workload R:W=0:100 (17%) (13%) 718 ms 709 ms 807 ms R:W=25:75 2335 ms 2619 ms 3915 ms (30%) (27%) (20%) D.1.1 Synthetic Micro-benchmark 555 ms 611 ms 650 ms R:W=50:50 2151 ms 2397 ms 3740 ms The synthetic micro-benchmark uses three tables called (25%) (25%) (17%) 407 ms 411 ms 466 ms ssibench-{1, 2, 3} with one non-null integer and ten vari- R:W=75:25 1791 ms 2028 ms 2019 ms (22%) (20%) (23%) able sized character columns; the integer value is a primary 379 ms R:W=100:0 500 ms N/A N/A N/A N/A key. Each of the three tables is populated with randomly (75%) chosen 100K items. To measure the serializable concurrency Table 1: The breakdown of transaction execution control overhead, we use three types of queries: a query time measured under synthetic workloads. Numbers transaction (read-only), an update transaction and an in- in parenthesis give the relative ratio of time spent sert transaction. All three transactions read a predefined during GDCP operations to the entire lifetime of a number of consecutive rows starting at a randomly selected transaction. row. An update(or insert) transaction updates (or inserts) a predefined number of rows chosen from a uniform random distribution of the identifiers. To create cycles, we config- D.2 The TPC-C++ Benchmark ured update/insert transactions to have the following access We also ran a series of experiments based on a more tra- rules; if a transaction reads items from ssibench-i, then ditional database benchmark. We are interested in the scal- update/insert rows from/to ssibench-j, where j=i + 1 mod ability of our RSSI approach with increasing cluster size. 3. The static dependency graph for ssibench is depicted in Since Fekete et al. [14] formally proved that TPC-C is se- Figure 5. rializable when run at snapshot isolation, Cahill et al. [7] D.1.2 GDCP Overhead proposed TPC-C++ in order to evaluate the effectiveness and overhead of avoiding SI anomalies. We use TPC-C++ To provide insight on the GDCP behavior in the RSSI and conduct experiments. We run experiments from 1 server architecture, we measured average elapsed time spent dur- to 8 servers by varying MPL from 5 to 640. Plotted points ing the execution of a successful transaction, compared to are also the average of 5 runs with 1 minute ramp-up period the sum of time spent in the GDCP code at the differ- and 1 minute measurement period. ent nodes. Due to space limitation, we present the results Figure 6 shows the throughput and abort rate for three with MPL=560 (high load condition) and 8 replicas under algorithms. Note that TPC-C++ is not designed to create ssibench. Table 1 shows this breakdown. The relative ratio numerous cycles as we have in ssibench. The breakdown of of GDCP overhead to the lifetime of a transaction is between aborts is not presented in this paper, but is similar to what 13% and 30%, except the case where a workload consists of is found in [7]. We observe the performance scalability un- 100% read-only transactions. The interesting result is that if der an update-intensive workload with very low probability a workload contains write transactions, the lifetime of read- of having non-serializable cases. As the number of servers only transactions is significantly elongated, as the portion increases, the throughput behavior of all three algorithms is of write transactions increases. We conjecture this may be the same as we could expect. As we can see in Figure 6(b), due to an increased amount of dependency information for most of aborts are ww-conflicts, not non-serializable errors. read-only transactions, but more investigation is needed.
794