Diamond: Automating Data Management and Storage for Wide-Area, Reactive Applications Irene Zhang, Niel Lebeck, Pedro Fonseca, Brandon Holt, Raymond Cheng, Ariadna Norberg, Arvind Krishnamurthy, and Henry M. Levy, University of Washington https://www.usenix.org/conference/osdi16/technical-sessions/presentation/zhang-irene
This paper is included in the Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16). November 2–4, 2016 • Savannah, GA, USA ISBN 978-1-931971-33-1
Open access to the Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation is sponsored by USENIX. Diamond: Automating Data Management and Storage for Wide-area, Reactive Applications
Irene Zhang Niel Lebeck Pedro Fonseca Brandon Holt Raymond Cheng Ariadna Norberg Arvind Krishnamurthy Henry M. Levy
University of Washington
Abstract sistent manner. Users of today’s popular wide-area apps (e.g., Twitter, This paper presents Diamond, the first reactive data Google Docs, and Words with Friends) must no longer management service (RDS) for wide-area applications save and reload when updating shared data; instead, these that continuously synchronizes shared application data applications are reactive, providing the illusion of con- across distributed processes. Specifically, Diamond per- tinuous synchronization across mobile devices and the forms the following functions on behalf of an application: cloud. Achieving this illusion poses a complex distributed (1) it ensures that updates to shared data are consistent data management problem for programmers. This pa- and durable, (2) it reliably coordinates and synchronizes per presents the first reactive data management service, shared data updates across processes, and (3) it automati- called Diamond, which provides persistent cloud storage, cally triggers reactive code when shared data changes so reliable synchronization between storage and mobile de- that processes can perform appropriate tasks. For example, vices, and automated execution of application code in when a user updates data on one device (e.g., a move in a response to shared data updates. We demonstrate that multi-player game), Diamond persists the update, reliably Diamond greatly simplifies the design of reactive appli- propagates it to other users’ devices, and transparently cations, strengthens distributed data sharing guarantees, triggers application code on those devices to react to the and supports automated reactivity with low performance changes. overhead. Reactive data management in the wide-area context re- quires a balanced consideration of performance trade-offs 1 Introduction and reasoning about complex correctness requirements in The modern world’s ubiquitous mobile devices, infinite the face of concurrency. Diamond implements the difficult cloud storage, and nearly constant network connectivity mechanisms required by these applications (such as log- are changing applications. Led by social networks (e.g., ging and concurrency control), letting programmers focus Twitter), social games (e.g., Words with Friends) and col- on high-level data-sharing requirements (e.g., atomicity, laboration tools (e.g., Google Docs), today’s popular ap- concurrency, and data layout). Diamond introduces three plications are reactive [41]: they provide users with the new concepts: illusion of continuous synchronization across their de- 1. Reactive Data Map (rmap), a primitive that lets ap- vices without requiring them to explicitly save, reload, plications create reactive data types – shared, per- and exchange shared data. This trend, not limited merely sistent data structures – and map them into the Dia- to mobile apps, includes the latest distributed versions mond data management service so it can automati- of traditional desktop apps on both Windows [13] and cally synchronize them across distributed processes OSX [4]. and persistent storage. Maintaining this illusion presents a challenging dis- 2. Reactive Transactions, an interactive transaction tributed data management problem for application pro- type that automatically re-executes in response to grammers. Modern reactive applications consist of widely shared data updates. These “live” transactions run distributed processes sharing data across mobile devices, application code to make local, application-specific desktops, and cloud servers. These processes make concur- updates (e.g., UI changes). rent data updates, can stop or fail at any time, and may be 3. Data-type Optimistic Concurrency Control connected by slow or unreliable links. While distributed (DOCC), a mechanism that leverages data-type storage systems [17, 77, 15, 23, 20] provide persistence semantics to concurrently commit transactions and availability, programmers still face the formidable executing commutative operations (e.g., writes to challenge of synchronizing updates between application different list elements, increments to a counter). Our processes and distributed storage in a fault-tolerant, con- experiments show that DOCC copes with wide-area
USENIX Association 12th USENIX Symposium on Operating Systems Design and Implementation 723 latencies very effectively, reducing abort rates by up Sum: 10 Client Device Client Device Sum: 10 Your turn: 5 Your turn: to 5x. myname Alice myname Bob myturn? true myturn? false We designed and implemented a Diamond prototype in curplay Alice 2 3 curplay Alice 10
Move(5) C++ with language bindings for C++, Python, and Java on players [Alice,Bob] players [Alice,Bob] turn 2 GetMove() turn 2 1 9 both x86 and Android platforms. We evaluate Diamond sum 10 7 sum 10 by building and measuring both Diamond and custom ver- Notify(Bob) Put(turn,3) Cloud Server Cloud Server Put(sum,15) 4 sions (using explicit data management) of four reactive 5 Notify(Bob) apps. Our experiments show that Diamond significantly Distributed Storage Get(turn) 6 players [Alice,Bob] 8 Get(sum)
Notification Service Cloud reduces the complexity and size of reactive applications, turn 2 notifications=[] provides strong transactional guarantees that eliminate sum 10 data races, and supports automatic reactivity with perfor- mance close to that of custom-written reactive apps. Figure 1: The 100 game. Each box is a separate address space. players, turn and sum are shared across address spaces and the 2 Traditional Data Management Tech- storage system; myturn? and curplay are derived from shared niques for Reactive Apps data. When shared values change, the app manually updates distributed storage, other processes with the shared data, and any Reactive applications require synchronized access to dis- data in those processes derived from shared data, as shown by tributed shared data, similar to shared virtual memory sys- the numbered steps needed to propagate Alice’s move to Bob. tems [46, 10]. For practical performance in the wide-area environment, apps must be able to control: (1) what data shows a typical three-tiered architecture used by these in each process is shared, (2) how often it is synchronized, apps (e.g., PlayFish uses it to serve over 50 million and (3) when concurrency control is needed. Existing users/month [34]). Processes on client devices access applications use one of several approaches to achieve syn- stateless cloud servers, which store persistent game chronization with control. This section demonstrates that state in a distributed storage system and use a reliable these approaches are all complex, error-prone, and make notification service (e.g., Thialfi [3]) to trigger changes it difficult to reason about application data consistency. in other processes for reactivity. While all application As an example, we analyze a simple social game based processes can fail, we assume strong guarantees – such on the 100 game [1]. Such games are played by mil- as durability and linearizability – for the storage system lions [78], and their popularity changes constantly; there- and notification service. Although such apps could rely fore, game developers want to build them quickly and on a single server to run the game, this would create a focus on game logic rather than data management. Be- centralized failure point. Clients cache game data to give cause game play increasingly uses real money (almost $2 users a responsive experience and to reduce load on the billion last year [24]), their design parallels other reactive cloud servers [34]. applications where correctness is crucial (e.g., apps for The numbers in Figure1 show the data management first responders [52] and payment apps [81, 72]). steps that the application must explicitly perform for Al- In the 100 game, players alternately add a number be- ice’s move (adding 5 to the sum). Alice’s client: (1) up- tween 1 and 10 to the current sum, and the first to reach dates turn and sum locally, (2) calculates new values for 100 wins. Players make moves and can join or leave the myturn? and curplay, and (3) sends the move to a cloud game at different times; application processes can fail at server. The server: (4) writes turn and sum to distributed any time. Thus, for safety, the game must maintain tradi- storage, and (5) sends a notification to Bob. The notifica- tional ACID guarantees – atomicity, consistency, isolation tion service: (6) delivers the notification to Bob’s client, and durability – as well as reactivity for data updates. We which (7) contacts a cloud server to get the latest move. call this combination of properties ACID+R. While a stor- The server: (8) reads from distributed storage and returns age system provides ACID guarantees for its own data, the latest turn and sum. Bob’s client: (9) updates turn and those guarantees do not extend to application processes. sum locally, and (10) re-calculates myturn? and curplay. In particular, pushing updates to storage on mobile devices Note that such data management must be customized is insufficient for reactivity because application processes to such games, making it difficult to implement a must re-compute local data derived from shared data to general-purpose solution. For example, only the appli- make changes visible to users and other components. cation knows that: (1) clients share turn and sum (but not myname), (2) it needs to synchronize turn and sum after 2.1 Roll-your-own Data Management each turn (but not players), and (3) it does not need con- Many current reactive apps “roll-their-own” currency control because turn already coordinates moves. application-specific synchronization across distributed Correctly managing this application data demands that processes on top of general-purpose distributed stor- the programmer reason about failures and data races at age (e.g., Spanner [17], Dropbox [23]). Figure1 every step. For example, the cloud server could fail in the
724 12th USENIX Symposium on Operating Systems Design and Implementation USENIX Association middle of step 4, violating atomicity. It could also fail Sum: 10 Client Device Client Device Sum: 10 Your turn: 5 Your turn: between steps 4 and 5, making the application appear as myname Alice myname Bob if it is no longer reactive. myturn? true myturn? false curplay Alice curplay Alice A new player, Charlie, could join the game while Bob players [Alice,Bob] players [Alice,Bob] makes his move, leading to a race; if Alice receives Bob’s turn 2 turn 2 sum notification first, but Charlie writes to storage first, then sum 10 10 libDiamond libDiamond both Alice and Charlie would think that it was their turn, Diamond Cloud Storage violating isolation. players [Alice,Bob] turn 2
Finally, even if the programmer were to correctly han- Diamond dle every failure and data race and write bug-free code, sum 10 reasoning about the consistency of application data would Figure 2: Diamond 100 game data model. The app rmaps prove difficult. Enforcing a single global ordering of join, players, turn and sum, updates them in read-write transactions leave and move operations requires application processes and computes myturn? and curplay in a reactive transaction. to either forgo caching shared data (or data derived from shared data) altogether or invalidate all cached copies and nization mechanisms. Further, they offer no distributed update the storage system atomically on every operation. concurrency control, leaving application programmers to The first option is not realistic in a wide-area environ- contend with race conditions; for example, they can lead ment, while the second is not possible when clients may to the race condition described in Section 2.1. be unreachable. 3 Diamond’s System and Programming 2.2 Wide-area Storage Systems Model A simple, alternative way to manage data manually is Diamond is a new programming platform designed to sim- to store shared application data in a wide-area storage plify the development of wide-area reactive applications. system (e.g., Dropbox [23]). That is, rather than calling This section specifies its data and transaction models and move in step 3, the application stores and updates turn and system call API. sum in a wide-area storage system. Though simple, this design can be very expensive. Distributed file systems are 3.1 System Model not designed to frequently synchronize small pieces of Diamond applications consist of processes running on data, so their coarse granularity can lead to moving more mobile devices and cloud servers. Processes can commu- data than necessary and false sharing. nicate through Diamond or over a network, which can Further, while this solution synchronizes Alice’s up- vary from local IPC to the Internet. Every application pro- dates with the cloud, it does not ensure that Bob receives cess is linked to a client-side library, called LIBDIAMOND, Alice’s updates. To simulate reactive behavior and en- which provides access to the shared Diamond cloud – a sure that Bob sees Alice’s updates, Alice must still use a highly available, fault-tolerant, durable storage system. wide-area notification system (e.g., Apple Push Notifica- Diamond subsumes some applications’ server-side func- tions [6]) to notify Bob’s client after her update. Unfor- tionality, but our goal is not to eliminate such code. We tunately, this introduces a race condition: if Bob’s client expect cloud servers to continue providing reliable and receives the notification before the wide-area storage sys- efficient access to computation and datacenter services tem synchronizes Alice’s update, then Bob will not see (e.g., data mining) while accessing shared data needed for Alice’s changes. Worse, Bob will never check the storage these tasks through Diamond. system again, so he will never see Alice’s update, leaving Figure2 shows the 100 game data model using Dia- him unable to make progress. Thus, this solution retains mond. Compared to Figure1, the application can directly all of the race conditions described in Section 2.1 and read and write to shared data in memory, and Diamond introduces some new ones. ensures updates are propagated to cloud storage and other 2.3 Reactive Programming Frameworks processes. Further, Diamond’s strong transactional guar- antees eliminate the need for programmers to reason about Several programming frameworks (e.g., Firebase [26], failures and concurrency. Parse [60] with React [64], Meteor [51]) have recently 3.2 Data Model been commercially developed for reactive applications. These frameworks combine storage and notification sys- Diamond supports reactive data types for fine-grained tems and automate data management and synchronization synchronization, efficient concurrency control, and persis- across systems. However, they do not provide a clear con- tence. As with popular data structure stores [19], such as sistency model, making it difficult for programmers to Redis [67] and Riak [68], we found that simple data types reason about the guarantees provided by their synchro- are general enough to support a wide range of applications
USENIX Association 12th USENIX Symposium on Operating Systems Design and Implementation 725 Table 1: Reactive data types. Table 2: Diamond system calls. Type Operations Description System call Description
Boolean Get(), Put(bool) Primitive boolean create(table, [isolation]) Create table Long Get(), Put(long) Primitive number status = rmap(var, table, key) Bind var to key String Get(), Put(str) Primitive string id = execute txn(func, cb) Start read-write transaction Counter Get(), Put(long) Long Counter id = register reactxn(func) Start reactive transaction Increment(long) reactxn stop(txn id) Stop re-executing Decrement(long) commit txn(), abort txn() Commit/Abort and exit IDGen GetUID() Unique ID generator LongSet Get(idx), Contains(long) Ordered number set 3.3.2 Transaction model Insert(long) LongList Get(idx), Set(idx, long) Number list Application processes use Diamond transactions to read Append(long) and write rmapped variables. Diamond transactions are StringSet Get(idx), Contains(str) Ordered string set interactive [73], i.e., they let applications interleave ap- Insert(str) plication code with accesses to reactive data types. We StringList Get(idx), Set(idx, str) String list support both standard read-write transactions and new Append(str) reactive transactions. Applications cannot execute trans- HashTable Get(key), Set(key, val) Unordered map actions across rmapped variables from different tables, while operations executed outside transactions are treated and provide the necessary semantics to enable commuta- as single-op transactions. tivity and avoid false sharing. Table1 lists the supported persistent data types and their operations. In addition to Read-write transactions. Diamond’s read-write trans- primitive data types, like String, Diamond supports sim- actions let programmers safely and easily access shared re- ple Conflict-free Replicated Data-types (CRDTs) [69] active data types despite failures and concurrency. Appli- (e.g., Counter) and collection types (e.g., LongSet) with cations invoke read-write transactions using execute txn. efficient type-specific interfaces. Using the most specific The application passes closures for both the transaction type possible provides the best performance (e.g., using a and a completion callback. Within the transaction closure, Counter for records that are frequently incremented). the application can read or write rmapped variables and A single Diamond instance provides a set of tables; variables in the closure, but it cannot modify program each table is a key-to-data-type map, where each entry, variables outside the closure. This limitation ensures: (1) or record, has a single persistent data type. Applications the transaction can access all needed variables when it access Diamond through language bindings; however, ap- executes asynchronously (and they have not changed), plications need not be written in a single language. We and (2) the application is not affected by the side effects currently support C++, Python and Java on both x86 and of aborted transactions. Writes to rmapped variables are Android but could easily add support for other languages buffered locally until commit, while reads go to the client- (e.g., Swift [5]). side cache or to cloud storage. 3.3 System Calls Before execute txn returns, Diamond logs the trans- action, with its read and write sets, to persistent storage. While apps interact with Diamond largely through reactive This step guarantees that the transaction will eventually data types, we provide a minimal system call interface, execute and that the completion callback will eventually shown in Table2, to support transactions and rmap. execute even if the client crashes and restarts. This guar- 3.3.1 The rmap Primitive antee lets applications buffer transactions if the network is unavailable and easily implement custom retry functional- rmap is Diamond’s key abstraction for providing shared ity in the completion callback. If the callback reports that memory that is flexible, persistent, and reactive across the transaction successfully committed, then Diamond wide-area application processes. Applications call rmap guarantees ACID+R semantics for all accesses to rmapped with an application variable and a key to the Diamond records; we discuss these in more detail in Section 3.4. record, giving them control over what data in their address On abort, Diamond rolls back all local modifications to space is shared and how it is organized. In this way, dif- rmapped variables. ferent application processes (e.g., an iOS and an Android client) and different application versions (e.g., a new and Reactive transactions. Reactive transactions help ap- current code release) can effectively share data. When plication processes automatically propagate changes rmapping records to variables, the data types must match. made to reactive data types. Each time a read-write trans- Diamond’s system call library checks at runtime and re- action modifies an rmapped variable in a reactive transac- turns an error from the rmap call if a mismatch occurs. tion’s read set, the reactive transaction re-executes, prop-
726 12th USENIX Symposium on Operating Systems Design and Implementation USENIX Association agating changes to derived local variables. As a result, Table 3: Diamond’s isolation levels. Isolation levels for read- reactive transactions provide a “live” view that gives the write transactions and associated ones for reactive transactions. illusion of reactivity while maintaining an imperative Stronger Read-write Isolation Level Reactive Isolation Level programming style comfortable to application program- Guarantees Strict Serializability Serializable Snapshot mers. Further, because they read a consistent snapshot of Snapshot Isolation rmapped data, reactive transactions avoid application-level Fewer Serializable Snapshot Aborts bugs common to reactive programming models [48]. Read Committed Read Committed Applications do not explicitly invoke reactive transac- tions; instead, they register them by passing a closure to register reactxn, which returns a txn id that can be used paradigm, this mechanism is transparent to applications to unregister the transaction with reactxn stop. Within because the ACID+R guarantees ensure that Diamond the reactive transaction closure, the application can read automatically re-executes the appropriate reactive trans- but not write rmapped records, preventing potential data actions when read-write transactions commit. flow cycles. Since reactive transactions are designed to Table3 lists Diamond’s isolation levels, which can be propagate changes to local variables, the application can set per table. Diamond’s default is strict serializability read and write to local variables at any time and trigger because it eliminates the need for application program- side-effects (i.e., print-outs, updating the UI). Diamond mers to deal with inconsistencies caused by data races guarantees that reactive transactions never abort because and failures. Lowering the isolation level leads to fewer it commits read-only transactions locally at the client. aborts and more concurrency; however, more anomalies Section4 details the protocol for reactive transactions. arise, so applications should either expect few conflicts, re- Reactive transactions run in a background thread, con- quire offline access, or tolerate inaccuracy (e.g., Twitter’s currently with application threads. Diamond transactions most popular hash tag statistics). Section 5.1 describes do not protect accesses to local variables, so the program- how DOCC increases concurrency and reduces aborts for mer must synchronize with locks or other mechanisms. transactions even at the highest isolation levels. The read set of a reactive transaction can change on every 3.5 A Simple Code Example execution; Diamond tracks the read set from the latest execution. Section 6.2 explains how to use reactive trans- To demonstrate the power of Diamond to simplify reactive actions to build general-purpose, reactive UI elements. applications, Figure3 shows code to implement the 100 3.4 Reactive Data Management Guarantees game from Section2 in Diamond. This implementation provides persistence, atomicity, isolation and reactivity Diamond’s guarantees were designed to meet the require- for every join and move operation in only 34 lines of code. ments of reactive applications specified in Section2, elim- We use three reactive data types for shared game data, inating the need for each application to implement its own declared on line 2 and rmapped in lines 7-9. It is important complex data management. To do so, Diamond enforces to ensure a strict ordering of updates, so we create a table ACID+R guarantees for reactive data types: in strict serializable mode on line 6. On line 12, we de- • Atomicity: All or no updates to shared records in a fine a general-purpose transaction callback for potential read-write transaction succeed. transaction failures. On line 16, we execute a read-write • Consistency: Accesses in all transactions reflect a transaction to add the player to the game, passing myname 1 consistent view of shared records. by value into the transaction closure. Using DOCC allows • Isolation: Accesses in all transactions reflect a Diamond to commit two concurrent executions of this global ordering of committed read-write transac- transaction while guaranteeing strict serializability. tions. Line 20 registers a reactive transaction to print out the • Durability: Updates to shared records in committed score and current turn. Diamond’s ACID+R guarantees read-write transactions are never lost. ensure that the transaction re-executes if players, turn • Reactivity: Accesses to modified records in regis- or sum change, so the user always has a consistent, up-to- tered reactive transactions will eventually re-execute. date view. Note that we can print to stdout because the These guarantees create a producer-consumer relation- reactive transaction will not abort, and the printouts reflect ship: Diamond’s read-write transactions produce updates a serializable snapshot, avoiding reactive glitches [48]. On to reactive data types, while reactive transactions con- line 32, we wait for user input in the while loop and use a sume those updates and propagate them to locally derived read-write transaction to commit the entered move. data. However, unlike the traditional producer-consumer Diamond’s strong guarantees eliminate the need for programmers to reason about data races or failures. Tak- 1The C in ACID is not well defined outside a database context. Diamond simply guarantees that each transaction reads a consistent ing our examples from Section2, Diamond ensures that snapshot. when the game commits Alice’s move, the move is never
USENIX Association 12th USENIX Symposium on Operating Systems Design and Implementation 727 1 int main (int argc , char ** argv ) { App Client App Server App Client 2 DStringSet players ; DCounter sum , turn ; 3 string myname = string ( argv [1]); libDiamond libDiamond libDiamond 4 register notify 5 // Map game state 6 create("100game", STRICT_SERIALIZABLE); Diamond front-end servers 7 rmap(players, "100game", "players"); get 8 rmap(sum, "100game", "sum"); prepare subscribe publish 9 rmap(turn, "100game", "turn"); commit 10 11 // General−purpose callback, exit if txn failed store store store store 12 auto cb = [] (txn_func_t txn, int status ) { 13 if (status == REPLY_FAIL) exit(1); }; 14 15 // Add user to the game 16 execute_txn([myname] () { 17 players.Insert(myname); }, cb); Figure 4: Diamond architecture. Distributed processes share a 18 single instance of the Diamond storage system. 19 // Set up our print outs 20 register_reactxn([myname] () { 21 string curplay = ing retried. For applications with higher contention, Dia- 22 players[turn % players.size()]; mond’s read committed mode enables commits locally at 23 bool myturn = myname == curplay; 24 cout << "Sum: " << sum << "\n"; the client while offline, and any modifications eventually 25 if (sum >= 100) converge to a consistent state for Diamond’s CRDTs. 26 cout <
728 12th USENIX Symposium on Operating Systems Design and Implementation USENIX Association them to authenticate with many back-end servers or track Alice libDiamond Front-end Back-end Front-end libDiamond Bob register the partitioning scheme. read(a) <1,[11,13)> read(b,14) 4.2 rmap and Language Bindings write(b,1) <0,[11,15)> sub reg(14,[b])
execute
(b,14) Register Diamond language bindings implement the library of reac- commit prepare gettimestamp validation tive data types for apps to use as rmap variables. Diamond ok ok pub commit notify interposes on every operation to an rmapped variable. Dur- commit (b,1,16) callback (b,1,16) re-exec ing a transaction, LIBDIAMOND collects an operation set read(b) OCC for D to later check for conflicts. Reads may hit the Notify Read-Write Transaction Reactive Transaction LIBDIAMOND client-side cache or require a wide-area ac- cess to the Diamond cloud, while writes (and increments, Figure 5: Diamond transaction coordination. Left: Alice exe- appends, etc.) are buffered in the cache until commit. cutes a read-write transaction that reads A and writes B. Right: Bob registers a reactive transaction that reads B (we omit the 4.3 Transaction Coordination Overview txn id). When Alice commits her transaction, the back-end Figure5 shows the coordination needed across LIBDIA- server publishes the update to the front-end, which pushes the MOND clients, front-end servers and back-end storage for notification and the update to Bob’s LIBDIAMOND, which can both read-write and reactive transactions. This section then re-execute the reactive transaction locally. briefly describes the transaction protocols. 2. Each participant runs a DOCC validation check (de- timestamp ordering Diamond uses to enforce isolation scribed in Section5); if DOCC validation succeeds, across LIBDIAMOND clients and back-end storage; it as- the participant adds the transaction to a prepared list signs every read-write transaction a unique commit times- and returns true; otherwise, it returns false. tamp that is provided by a replicated timestamp service 3. As an optimization, the front-end server concurrently (tss) (not shown in Figure4). Commit timestamps reflect retrieves a commit timestamp from the tss. the transaction commit order, e.g., in strict serializability 4. If all participants respond true, the front-end sends mode, they reflect a single linearizable ordering of com- Commits to the participants with the commit times- mitted, read-write transactions. Both Diamond’s client- tamp; otherwise, it sends Aborts. Then, it returns the side cache and back-end storage are multi-versioned using transaction outcome to the LIBDIAMOND client. these commit timestamps. When the client receives the response, it logs the transac- 4.3.1 Running Distributed Transactions tion outcome and invokes the transaction callback. Read-write and reactive transactions execute similarly; 4.3.2 Managing Reactive Transactions however, as Section5 relates, reactive transactions can As shown in Figure5 (right), when an application registers commit locally and often avoid wide-area accesses alto- a reactive transaction, the LIBDIAMOND client: (1) gives gether. We lack the space to cover Diamond’s transaction the reactive transaction a txn id, (2) executes the reactive protocol in depth; however, it is similar to Spanner’s [17] transaction at its latest known timestamp, and (3) sends with two key differences: (1) Diamond uses DOCC for the txn id, the timestamp, and the read set in a Register concurrency control rather than a locking mechanism, and request to the front-end server. For each key in the read (2) Diamond uses commit timestamps from the timestamp set, the front-end server creates a Subscribe request and service (tss) rather than TrueTime [17]. sends those requests, along with the timestamp, to each As shown in Figure5 (left), transactions progress key’s back-end partition. through two phases, execution and commit. During the For efficiency, LIBDIAMOND tracks read set changes execution phase, LIBDIAMOND runs the application code between executions and re-registers. We expect each reac- in the transaction closure passed into txn execute. It runs tive transaction’s read set to change infrequently, reducing the code locally on the LIBDIAMOND client node (i.e., not the overhead of registrations; if it changes often, we can on a storage node like a stored procedure). use other techniques (e.g., map objectrange described in The execution phase completes when the application Section 6.2) to improve performance. exits the transaction closure or calls txn commit explic- When read-write transactions commit, Diamond exe- itly. Reactive transactions commit locally; for read-write cutes the following steps for each updated record: transactions, LIBDIAMOND sends the operation sets to the 1. The leader in the partition sends a Publish request front-end server, which acts as the coordinator for a two- with the transaction’s commit timestamp to each phase commit (2PC) protocol, as follows: front-end subscribed to the updated record. 1. It sends Prepare to all participants (i.e., partitions 2. For each Publish, the front-end server looks up the of the Diamond back-end that hold records in the reactive transactions that have the updated record in operation sets), which replicate it via VR. their read sets and checks if the commit timestamp
USENIX Association 12th USENIX Symposium on Operating Systems Design and Implementation 729 is bigger than the last notification sent to that client. Table 4: DOCC validation matrix. Matrix shows whether the 3. If so, the front-end server sends a Notify request to committing transaction can commit (C) or must abort (A) on the client with the commit timestamp and the reactive conflicts. Each column is further divided by the isolation level (RC=read committed, SI=snapshot isolation, SS=strict serializ- transaction id. ability). Commutative CRDT operations have the same outcome. 4. The client logs the notification on receipt, updates its Prepared Isolation Level read write CRDT op latest known timestamp, and re-executes the reactive Commiting Op transaction at the commit timestamp. Op RC SI SS RC SI SS RC SI SS For keys that are updated frequently, back- and front-end read C C C C C A C C A servers batch updates. Application clients can bound the write C C A C A A C A A batching latency (e.g., to 5 seconds), ensuring that reactive transactions refresh at least once per batching latency CRDT op C C A C A A C C C when clients are connected. 4.3.3 Handling Failures types with commutative operations, such as counters and While both the back-end storage and tss are replicated us- ordered sets. As noted in Section 4.3.1, LIBDIAMOND col- lects an operation set for every data type operation during ing VR, Diamond can suffer failures of the LIBDIAMOND the transaction’s execution phase. For each operation, it clients or front-end servers. On client failure, LIBDIA- collects the key and table. It also collects the read version MOND runs a client recovery protocol using its transaction log to ensure that read-write transactions eventually com- for every Get, the written value for every Put, the index mit. For each completed but unprocessed transaction (i.e., (e.g., list index or hash table key) for every collection op- eration, and the diff (e.g., the increment value or the insert in the log but with no outcome), LIBDIAMOND retries the commit. If the cloud store has a record of the transaction, or append element) for every commutative CRDT oper- it returns the outcome; otherwise, it re-runs 2PC. For each ation. We show in Section6 that although fine-grained reactive transaction, the application re-registers on recov- tracking slightly increases DOCC overhead, it improves overall performance. ery. LIBDIAMOND uses its log to find the last timestamp at which it ran the transaction. Using operation sets, DOCC runs a validation proce- Although front-end servers are stateless, LIBDIAMOND dure that checks every committing transaction for poten- clients must set up a new front-end server connection tial violations of isolation guarantees. A conflicting access when they fail. They use the client recovery protocol to occurs for an operation if the table, key, and index (for do this and re-register each reactive transaction with its collection types) match an operation in a prepared trans- latest notification timestamp. Front-end servers also act action. For a read, a conflict also occurs if the latest write as coordinators for 2PC, so back-end storage servers use version (or commutative CRDT operation) to the table, the cooperative termination protocol [11] if they do not key, and index is bigger than the read version. For each, receive Commit requests after some timeout. DOCC makes an abort decision, as noted in Table4. Since transactions that contain only commutative oper- 5 Wide-area Optimizations ations can concurrently commit, DOCC can allow many This section discusses Diamond’s optimizations to reduce concurrent transactions that modify the same keys. This wide-area overhead. property is important for workloads with high write con- tention, e.g., the Twitter “like” counter for popular celebri- 5.1 Data-type Optimistic Concurrency Control ties [36]. Further, because Diamond runs read-only and Diamond uses an optimistic concurrency control (OCC) reactive transactions in serializable snapshot mode, they mechanism to avoid locking across wide-area clients. Un- do not conflict with read-write transactions with commu- fortunately, OCC can perform poorly across the wide area tative CRDT operations. due to the higher latency between a transaction’s read 5.2 Client Caching with Bounded Validity Intervals of a record and its subsequent commit. This raises the likelihood that a concurrent write will invalidate the read, Some clients in the wide-area setting may occasionally be thereby causing a transaction abort. For example, to in- unavailable, making it impossible to atomically invalidate crement a counter, the transaction reads the current value, all cache entries on every write to enforce strong order- increments it, and then commits the updated value; if an- ing. Diamond therefore uses multi-versioning in both the other transaction attempts the same operation at the same client-side cache and back-end storage to enforce a global time, an abort occurs. ordering of transactions. To do this, it tags each version DOCC tackles this issue in two ways. First, it uses fine- with a validity interval [62], which begins at the start grained concurrency control based on the semantics of timestamp and is terminated by the end timestamp. In reactive data types, e.g., allowing concurrent updates to Diamond’s back-end storage, a version’s start timestamp different list elements. Second, it uses conflict-free data is the commit timestamp of the transaction that wrote the
730 12th USENIX Symposium on Operating Systems Design and Implementation USENIX Association Table 5: Application comparison. Diamond both reduces code A Diamond B size and adds to the application’s ACID+R guarantees. Client Cache C Added Application LoC w/o LoC w/ LoC 10 11 12 13 14 15 16 Diamond Diamond Saved ACIDR Diamond A Cloud B 100 Game 46 34 26% DDD Storage C Chat Room 355 225 33% DDD D 10 11 12 13 14 15 16 PyScrabble 8729 7603 13% DD DDD Figure 6: Diamond versioned cache. Every Diamond client Twitter clone 14278 12554 13% has a cache of the versions of records stored by the Diamond cloud storage system. The bottom half shows versions for three quests. Further, since the reads were done at the commit keys (A, B and C), and the top half shows cached versions of timestamp, LIBDIAMOND knows that the transaction can those same keys. Note that the cache is missing some versions, be serialized at that timestamp and committed locally, and all of the validity intervals in the cache are bounded. eliminating all wide-area communication. version. The end timestamp is either the commit times- 6 Experience and Evaluation tamp of the transaction writing the next version (making This section evaluates Diamond with respect to both pro- that version out-of-date) or unbounded for the latest ver- gramming ease and performance. Overall, our results sion. Figure6 shows an example of back-end storage with demonstrate that Diamond simplifies the design of re- three keys. active applications, provides stronger guarantees than ex- On reads, the Diamond cloud tags the returned value isting custom solutions, and supports automated reactivity with a validity interval for the LIBDIAMOND client-side with low performance overhead. cache. These validity intervals are conservative; back-end storage guarantees that the returned version is valid at 6.1 Prototype Implementation least within the validity interval, although it may be valid We implemented a Diamond prototype in 11,795 lines beyond. If the version is the latest, back-end storage will of C++, including support for C++, Python and Java lan- bound the validity interval by setting the end timestamp to guage bindings on both x86 and ARM. The Java bindings the latest commit timestamp of a transaction that accessed (939 LoC) use javacpp [39], and the Python bindings that record. For example, in Figure6, the validity interval (115 LoC) use Boost [2]. We cross-compiled Diamond of the latest version of B and C are capped at timestamp 16 and its dependencies for Android using the NDK stan- in the cache, while they are unbounded in storage. Most dalone toolchain [29]. We implemented most Diamond importantly, bounded validity intervals eliminate the need data types, but not all are supported by DOCC. Our cur- for cache invalidations because the version is always valid rent prototype does not include client-side persistence and within the validity interval. Diamond eventually garbage relies on in-memory replication for the back-end store; collects cached versions as they become too outdated to however, we expect disk latency on SSDs to have a low use. performance impact compared to wide-area network la- tency, with NVRAM reducing storage latency even further 5.3 Data Push Notifications in the future. Reactive transactions require many round-trips to syn- 6.2 Programming Experience chronously fetch each update; these can be expensive in a wide-area network. Fortunately, unlike stand-alone no- This section evaluates our experience in building new tifications services (e.g., Thialfi), Diamond has insight Diamond apps, porting existing apps to Diamond, and cre- into what data the application is likely to access when ating libraries to support the needs of reactive programs. the reactive transaction re-executes. Thus, Diamond uses 6.2.1 Simplifying Reactive Applications data push notifications to batch updates along with notifi- cations, reducing wide-area round trips. To evaluate Diamond’s programming benefits, we imple- mented applications both with and without Diamond. Ta- When front-end servers receive Publish requests from ble5 shows the lines of code for both cases. For all of the back-end storage, they perform a snapshot read of every apps, Diamond simultaneously decreased program size key in the reactive transaction’s last read set at the up- and added important reliability or correctness properties. dating transaction’s commit timestamp, then piggyback We briefly describe the programs and results below. the results with the Notify request to the LIBDIAMOND client. LIBDIAMOND re-executes the reactive transaction 100 Game. Our non-Diamond version of the 100 game at the commit timestamp; therefore, if its read set has is based on the design in Figure1. For simplicity, we used not changed, then it requires no additional wide-area re- Redis [67] for both storage and notifications. We found
USENIX Association 12th USENIX Symposium on Operating Systems Design and Implementation 731 several data races between storage updates and notifica- cache and back-end storage. Diamond directly plugged tions when running experiments for Figure9, forcing us into UI elements and published updates with read-write to include updates in the notifications to ensure clients did transactions. As a result, it simplified the design, elim- not read stale data from the store. The Diamond version inated 1700 lines (13%) from the 14K-line application, eliminated these bugs and the complexities described in transparently provided stronger atomicity and isolation Section2 and guaranteed correctness with atomicity and guarantees, and eliminated inconsistent behaviors (e.g., a isolation; in addition, it reduced the code size by 26%. user seeing a retweet before the original tweet). Chat Room. As another simple but representative ex- 6.2.2 Simplifying Reactive Libraries ample of a reactive app, we implemented two versions of a chat room. Our version with explicit data management In addition to simplifying the design and programming used Redis for storage and the Jetty [40] web server to of reactive apps, we found that Diamond facilitates the implement a REST [25] API. It used POST requests to send creation of general-purpose reactive libraries. As one ex- messages and polled using GET requests for displaying the ample, Diamond transactions naturally lend themselves log. This design is similar to that used by Twitter [80, 35] to managing UI elements. For instance, a check box usu- to manage its reactive data (e.g., Twitter has POST and ally rmaps a Boolean, re-draws a UI element in a reac- GET requests for tweets, timelines, etc.). The Diamond tive transaction, and writes to the Boolean in a read-write version used a StringList for the chat log, a read-write transaction when the user checks/unchecks the box. We transaction to append messages, and a reactive transac- implemented a general library of Android UI elements, in- tion to display the log. In comparison, Diamond not only cluding a text box and check box. Each element required eliminated the need for a server or storage system, it also under 50 lines of code yet provided strong ACID+R guar- provided atomicity (the Redis version has no failure guar- antees. Note that these elements tie the user’s UI to shared antees), isolation (the Redis version could not guarantee data, making it impossible to update the UI only locally; that all clients saw a consistent view of the chat log), and for example, if a user wants to preview a message before reactivity (the Redis version polled for new messages). sharing it with others, the app must update the UI in some Diamond also shrunk the 355-line app by 130 lines, or other way. 33%. For generality, Diamond makes no assumptions about an app’s data model, but we can build libraries using PyScrabble and Diamond Scrabble. To evaluate the rmap for common data models. For example, we imple- impact of reactive data management in an existing appli- mented object-relational mapping for Java objects whose cation, we built a Diamond version of PyScrabble [16], fields were Diamond data types. Using Java reflection, an open-source, multiplayer Scrabble game. The original rmap object maps each Diamond data type inside an ob- PyScrabble does not implement persistence (i.e., it has no ject to a key derived from a base key and the field’s name. storage system) and uses a centralized server to process We also support rmap for subsets of Diamond collections, moves and notify players. The centralized server enforces e.g., rmap range for Diamond’s primitive list types, which isolation and consistency only if there are no failures. We binds a subset of the list to an array, and rmap objectrange, made some changes to add persistence and accommodate which maps a list of objects using rmap object. Diamond’s transaction model. We chose to directly rmap These library functions were easy to build (under 75 the Scrabble board to reactive data types and update the lines of code) and greatly simplified several applications; UI in a reactive transaction, so our implementation had to for example, our Diamond Twitter implementation stores commit and share every update to make it visible to the a user’s timeline as a LongList of tweet ids and uses user; thus, other users could see the player lay down tiles map objectrange to directly bind the tail of the user’s in real-time rather than at the end of the move, as in the timeline into a custom Android adapter, which then plugs original design. Overall, our port of PyScrabble to Dia- into the Twimight Android client and automatically man- mond removed over 1000 lines of code from the 8700-line ages reactivity. In addition to reducing application com- app (13%) while transparently simplifying the structure plexity, these abstractions also provide valuable hints for (removing the server), adding fault tolerance (persistence) prefetching and for how reactive transaction read sets and atomicity, and retaining strong isolation. might change. Overall, we found Diamond’s program- Twimight and Diamond Dove. As another modern re- ming model to be extremely flexible, powerful, and easy active application, we implemented a subset of Twitter us- to generalize into widely useful libraries. ing an open-source Android Twitter client (Twimight [79]) 6.3 Performance Evaluation and a custom back-end. The Diamond version eliminated much of the data management in the Twimight version, Our performance measurements demonstrate that Dia- i.e., pushing and retrying updates to the server and main- mond’s automated data management and strong consis- taining consistency between a client-side SQLite [71] tency impose a low performance cost relative to custom-
732 12th USENIX Symposium on Operating Systems Design and Implementation USENIX Association Strong Operation Ordering in Figure7 shows the Diamond read-committed (RC) Strong Transaction Ordering version, whose performance (30.5K trans./sec.) is nearly Linearizable Txns identical. Unlike the Redis-based implementation, how- ever, the Diamond benchmark provides strong consistency based on VR, i.e., it enforces a single global order of op- erations but not transactions. The Diamond version also Figure 7: Peak throughput for explicit data management vs provides all of its reactivity support features. Diamond Diamond. We compare an implementation using Redis and Jetty therefore provides better consistency properties and sim- to Diamond at different isolation levels with and without DOCC. plifies programming at little additional cost. We label the ordering guarantees provided by each configuration. As we add stronger isolation through transactions, In all cases, the back-end servers were the bottleneck. throughput declines because two-phase commit requires written applications. Using transactions with strong isola- each back-end server to process an extra message per tion properties lowers throughput, as one would expect. transaction. As the graph shows, snapshot isolation (SI) We also show that Diamond’s DOCC improves perfor- and strict serializability (SS) reduce throughput by nearly mance of transactional guarantees, and that data push no- 50% from RC. The graph also shows SI and SS both with tifications reduce the latency of wide-area transactions. and without DOCC; eliminating DOCC hurts SS more Finally, our experiments prove that Diamond has low over- than SI (27% vs. 13%) because SI lets transactions with head on mobile devices and can recover quickly from read-write conflicts commit (leading to write skew). failures. From this experiment, we conclude that Diamond’s general-purpose data management imposes almost no 6.3.1 Experimental Setup throughput overhead. Also, achieving strong transactional We ran experiments on Google Compute Engine [30] us- isolation guarantees does impose a cost due to the more ing 16 front-end servers and 5 back-end partitions, each complex message protocol required. Depending on the with 3 replicas placed in different availability zones in application, programmers can choose to offset the cost the same geographic region (US-Central). Our replica- by allocating more servers or tolerate inconsistencies that tion protocol used adaptive batching with a batch size result from weaker transactional guarantees. of 64. We placed clients in a different geographic re- 6.3.3 Benefit of DOCC gion in the same country (US-East). The latency between zones was ≈1 ms, while the latency between regions was DOCC’s benefit depends on both contention and trans- ≈36 ms. For our mobile device experiments, we used action duration. To evaluate this effect, we measured the Google Nexus 7 LRX22G tablets connected via Wi-Fi throughput improvement of DOCC for each type of Retwis and, for desktop experiments, we used a Dell workstation transaction with at least one CRDT operation (Figure8). with an Intel Xeon E5-1650 CPU and 16 GB RAM. The add user and like transactions are short and thus We used a benchmark based on Retwis [45], a Redis- unlikely to abort, but they still see close to a 2x improve- based Twitter clone previously used to benchmark transac- ment. add follower gets a larger benefit (4x) because it is tional storage systems [84]. The benchmark was designed a longer transaction with more commutative operations. to be a representative, although not realistic, reflection of Even get timeline, a read-only transaction, gets a tiny im- a Twitter-like workload that provides control over con- provement (2.5%) due to reduced load on the servers from tention. It ran a mix of five transactions that range from aborting transactions. Further, because get timeline runs 4-21 operations, including: loading a user’s home time- in serializable snapshot mode, post tweet transactions line (50%), posting a tweet (20%), following a user (5%), can commit concurrently with get timeline transactions. creating a new user (1%), and “like”-ing a tweet (24%). The post tweet transaction appends a user’s new tweet To increase contention, we used 100K keys and a Zipf to his timeline and his followers’ home timelines (each distribution with a co-efficient of 0.8. user has between 5 and 20 followers). If a user follows a large number of people that tweet frequently, conven- 6.3.2 Overhead of Automated Data Management tional OCC makes it highly likely that a conflicting Append For comparison, we built an implementation of the Retwis would cause the entire transaction to fail. With DOCC, all benchmark that explicitly manages reactive data using Appends to a user’s home timeline can commute, avoid- Jetty [40] and Redis [67]. The Redis WAIT command offers ing these aborts. As a result, we saw a 5x improvement synchronous in-memory replication, which matches Dia- in abort rate with DOCC over conventional OCC for mond’s fault-tolerance guarantees but provides no opera- post tweet, leading to a 25x improvement in throughput. tion or transaction ordering [66]. The leftmost bar in Fig- Overall, these results show that Diamond’s support for ure7 shows the peak Retwis throughput of 31K trans./sec. data types in its API and concurrency control mechanism for the Redis-based implementation, while the second bar is crucial to reducing the cost of transactional guarantees.
USENIX Association 12th USENIX Symposium on Operating Systems Design and Implementation 733 0 aewrla n ildabc-n evrduring server back-end a killed and workload game 100 emaueteltnyfo n lyrscin o each for client player’s one from latency the measure We sarsl,Daodhs3%lwrltnyadstronger and latency lower 30% has Diamond result, a As im- management data automated Diamond’s Although 734 12th USENIXSymposium onOperating Systems Designand Implementation US-East. in remained clients while geo- US-Central Europe, Asia, and we across overhead, servers recovery back-end the the replicated increase To game. the same the used we impact, read-write this and measure To reactive both transactions. of latency Failures the Server affect Failures Storage Wide-area of Impact 6.3.5 implementation. Redis our than guarantees transactional game. the in transactions read-write and for reactive reads the wide-area both eliminating by overall 50% the almost reduce by notifications latency push data one that shows for 9 or ure turn a take to player notifications. push data of version manual a as updates clients carry to notifications imple- where an Redis, design using also We mentation read transaction. the read-write with the overlaps of reactive it set the and of change, set not read does transaction the This because time). ideal “think” zero is soon (i.e., experiment as finishes move two player a other benchmark: make the a players as and into game, 3 each Figure join from players game 100 implementation the our of turned We cache. popu- client-side to the notifications late transaction reactive with updates one. only needs Redis while commit, to to one one transaction, and per read trips round two because requires implementation for Diamond Redis high as our twice to is relative transaction Diamond Retwis exam- a For of cloud. latency Diamond the the ple, to due trips latency round hurt wide-area can to it overhead, throughput low a poses Notifications Push Data of Benefit 6.3.4 updates. and batched without Redis and using with notifications notifications Diamond and implemented We management client. 1 data from explicit measured is latency players; 2 of notifications. push data 9: Figure type. transaction Retwis 8: Figure
aaps oictosrdc hsltnyb batching by latency this reduce notifications push Data Latency Throughput
0.25 s 0.75 s Improvement 0.5 s 0X 1X 2X 3X 4X 5X 0 s 1 s aec oprsnfr10gm onswith rounds game 100 for comparison Latency hogptipoeetwt D with improvement Throughput add_user No DataPush 2X add_follower 4X ahrudcnito oeb each by move 1 of consist round Each post_tweet Data Push 25X 25x round get_timeline ftegm.Fig- game. the of 1X OCC Redis o each for like 2X ekl h edri SCnrl wthn tt Europe. to it switching US-Central, in leader the kill we rgt hw prtosfrteCa omo nAndroid an on Room Chat the for operations shows (right) h aec fec on nrae oams seconds 4 almost to increases round each of latency The oeaut imn’ mato h sreprec,we experience, user the on impact Diamond’s evaluate To eso ssaJv TPcin.Oeal efudthe found we Redis Overall, client. the HTTP while Java C++, a native uses in version runs it because version h edri uoeicesst 0 s n h latency the and ms, 100 to increases Europe and in servers leader front-end the the between latency the afterwards: game, the into seconds 15 up About and contact. ms, to 100 ms, least a 150 at from to take response which a replicas, for of wait to quorum experiment has previous leader the VR in the that because than higher is latency the move. region, each geographic on another latency messaging Af- to overall seconds. moves increasing 15 leader about the after recovery, partition killed ter storage and the move a of make leader to the players both for latency the measured 10: Figure hrdmmr ytm n oicto systems. notification and systems memory shared frameworks, programming distributed reac- programming, databases, tive and systems sys- storage storage transactional wide-area tems, from inspiration takes Diamond Work Related 7 users. to perceivable not were non-Diamond operations and Diamond between differences latency Redis the than faster percent few The a back. is it commits version and Diamond message, a appends log, chat the 11 tablet. Figure respectively). 16%, and is (9% latency higher Diamond’s slightly PyScrabble, original Com- the visible. to it make pared to transaction reactive and tification and move, MakeMove were cloud Diamond the to ≈ runs times app ping Room The Chat Android. our on while application, PyScrabble desktop Diamond. a without is from and apps with two built in 6.2 operations Section user of latency the measure Latency Application End-user seconds. 3 6.3.6 than the less detect in leader can the Diamond replace and that takes failure failure meaning the seconds, during 7 round the only increases this, Asia Despite in ms. replica 250 remaining to the to leader the from 8m ntedstpand desktop the on ms 38 sosteltnyo ahrud oethat Note round. each of latency the shows 10 Figure eea omrilpafrs[ platforms commercial Several PyScrabble: for operations two shows (left) 11 Figure Latency 0 s 2 s 4 s 6 s 8 s ReadLog 6 omt rnato htudtsteuser’s the updates that transaction a commits aec f10gm onsdrn failure. during rounds game 100 of Latency DisplayMove 7 estefl htlg and log, chat full the gets 8 Number ofRoundsPassed 9 nldsMkMv lsteno- the plus MakeMove includes 10 ≈ 11 6m nteAdodtablet. Android the on ms 46 12 51 USENIX Association 13 , PostMessage 26 14 , 60 15 provide ] 16 gets 17 We state. Diamond’s design draws on Thialfi [3]; however, Thialfi cannot efficiently support data push notifications without insight into the application’s access patterns. DOCC is similar to Herlihy [32, 31] and Weihl’s [83] work on concurrency control for abstract data types. How- Figure 11: End-user operation latency for PyScrabble and ever, Diamond applies their techniques to CRDTs [69] Chat Room on Diamond and non-Diamond implementa- over a range of isolation levels in the wide area. DOCC tions. is also related to MDCC [43] and Egalitarian Paxos [54]; however, DOCC uses commutativity for transactional con- an early form of reactive data management without dis- currency control rather than Paxos ordering and supports tributed transactions. Other open source projects [38, 55, more data types. DOCC extends recent work on software 21, 59, 70] have replicated the success of their commercial transactional objects [33] for single-node databases to the counterparts. Combined, they comprise a mobile back-end wide area; integrating the two would let programmers market of $1.32 billion dollars [49]. implement custom data types in Diamond. However, these products do not meet the requirements Diamond does not strive to support a fully reactive, of reactive applications, still requiring programmers to ad- data-flow-based programming model, like functional re- dress failures and race conditions. Meteor [51] lets client- active or constraint-based programming [82, 7]; however, side code directly access the database interface. How- reactive transactions are based on the idea of change ever, because it uses existing databases (MongoDB [53], propagation. Recent interest in reactive programming and most recently, Postgres [63]) that do not support dis- for web client UIs has resulted in Facebook’s popular tributed transactions and offer weak consistency guar- React.js [64], the ReactiveX projects [65], and Google’s antees by default, programmers must still reason about Agera[28]. DREAM [48], a recently proposed, distributed race conditions and consistency bugs. Parse [60] and reactive platform, lacks transactional guarantees. Sap- Firebase [26] similarly enable clients to read, write, and phire [85], another recent programming platform for mo- subscribe to objects that are automatically synchronized bile/could applications, does not support reactivity, dis- across mobile devices; however, these systems offer no tributed transactions, or general-purpose data manage- concurrency control or transactions. As demonstrated by ment. these Stack Overflow questions [56, 50], programmers find this to be a significant issue with these systems. Di- 8 Conclusion amond addresses this clear developer need by providing ACID+R guarantees for reactive applications. This paper described Diamond, the first data management service for wide-area reactive applications. Diamond in- There has been significant work in wide-area storage troduced three new concepts: the rmap primitive, reactive systems for distributed and mobile applications, including transactions, and DOCC. Our evaluation demonstrated numerous traditional instantiations [77, 42, 57] as well that: (1) Diamond’s programming model greatly simpli- as more recent work [18, 9, 74, 61, 75]. Many mobile ap- fies reactive applications, (2) Diamond’s strong transac- plications today use commercial storage services such as tional guarantees eliminate data race bugs, and (3) Dia- Dropbox and others [23, 22, 37], while users can also em- mond’s low performance overhead has no impact on the ploy revision-based storage (e.g., git [27]). Applications end-user. often combine distributed storage with notifications [3, 6]. As discussed, these systems help with data management, 9 Acknowledgements but none offers a complete solution. Diamond shares a data-type-based storage model with We thank the UW systems lab for their comments through- data structure stores [67, 68]. Document stores (e.g., Mon- out the project. This work was supported by Google, Na- goDB [53]) support application objects; this prevents tional Science Foundation grant CNS-1518702 and NSF them from leveraging semantics for better performance. GRFP, and MSR Ph.D. fellowships. We also thank our These datastores, along with more traditional key-value anonymous reviewers and our shepherd, Bryan Ford, for and relational storage systems [15, 8, 44, 76], were not their feedback. designed for wide-area use although they could support References reactive applications with additional work. Reactive transactions in Diamond are similar to [1] Nim, Feb 2016. https://en.wikipedia.org/wiki/Nim#The database triggers [47], events [14], and materialized 100 game. views [12]. They differ from these mechanisms because [2] D. Abrahams and S. Seefeld. Boost C++ libraries, they modify local application state and execute applica- 2015. http://www.boost.org/doc/libs/1 60 0/libs/python/ tion code rather than database queries that update storage doc/html/index.html.
USENIX Association 12th USENIX Symposium on Operating Systems Design and Implementation 735 [3] A. Adya, G. Cooper, D. Myers, and M. Piatek. Thialfi: a [19] DB-engine’s ranking of key-value stores, 10 2015. http: client notification service for internet-scale applications. //db-engines.com/en/ranking/key-value+store. In Proc. of SOSP, 2011. [20] G. DeCandia, D. Hastorun, M. Jampani, G. Kakulapati, [4] S. Alvos-Bock. The convergence of iOS and OSX A. Lakshman, A. Pilchin, S. Sivasubramanian, P. Vosshall, user interface design, July 2015. http://www.solstice- and W. Vogels. Dynamo: Amazon’s highly available key- mobile.com/blog/the-convergence-of-ios-and-os-x-user- value store. In Proc. of SOSP, 2007. interface-design. [21] deepstream. deepstream.io: a scalable server for realtime [5] Apple. The Swift programming language, 2016. web apps. https://deepstream.io/. https://developer.apple.com/library/ios/documentation/ [22] Google Drive, 2016. http://drive.google.com. Swift/Conceptual/Swift Programming Language/#/ [23] Dropbox, 2015. http://www.dropbox.com. /apple ref/doc/uid/TP40014097-CH3-ID0. [24] eMarketer. Mobile game revenues to grow 16.5% [6] Apple push notification service, 2015. https: in 2015, surpassing $3 billion, Feb 2015. http: //developer.apple.com/library/ios/documentation/ //www.emarketer.com/Article/Mobile-Game-Revenues- NetworkingInternet/Conceptual/RemoteNotificationsPG/ Grow-165-2015-Surpassing-3-Billion/1012063. Chapters/ApplePushService.html. [25] R. T. Fielding. Architectural Styles and the Design of [7] K. Apt. Principles of Constraint Programming. Cam- Network-based Software Architectures. PhD thesis, Uni- bridge University Press, 2003. versity of California, Irvine, 2000. [8] J. Baker, C. Bond, J. C. Corbett, J. Furman, A. Khorlin, [26] Firebase, 2015. https://www.firebase.com/. J. Larson, J.-M. Leon, Y. Li, A. Lloyd, and V. Yushprakh. Megastore: Providing scalable, highly available storage [27] Git, 2015. https://git-scm.com/. for interactive services. In Proc. of CIDR, 2011. [28] Google. Agera. https://github.com/google/agera. [9] N. M. Belaramani, J. Zheng, A. Nayate, R. Soule,´ [29] Google. Android standalone toolchain, M. Dahlin, and R. Grimm. PADS: A policy architecture 2016. http://developer.android.com/ndk/guides/ for distributed storage systems. In Proc. of NSDI, 2009. standalone toolchain.html. [10] J. K. Bennett, J. B. Carter, and W. Zwaenepoel. Munin: [30] Google Compute Engine. https://cloud.google.com/ Distributed shared memory based on type-specific memory products/compute-engine/. coherence. In Proc. of PPOPP, 1990. [31] M. Herlihy. Optimistic concurrency control for abstract [11] P. A. Bernstein, V. Hadzilacos, and N. Goodman. Concur- data types. In Proc. of PODC. ACM, 1986. rency Control and Recovery in Database Systems. Addison [32] M. Herlihy. Apologizing versus asking permission: Opti- Wesley, 1987. mistic concurrency control for abstract data types. ACM [12] J. A. Blakeley, P.-A. Larson, and F. W. Tompa. Efficiently Transactions on Database Systems, 1990. updating materialized views. In Proc. of SIGMOD, 1986. [33] N. Herman, J. P. Inala, Y. Huang, L. Tsai, E. Kohler, [13] J. Callaham. Yes, windows 10 is the next ver- B. Liskov, and L. Shrira. Type-aware transactions for sion of windows phone. Windows Central, Sept faster concurrent code. In eurosys, 2016. 2014. http://www.windowscentral.com/yes-windows-10- [34] T. Hoff. Playfish’s social gaming architecture - 50 million next-version-windows-phone. monthly users and growing. High Scalability, Sept 2010. [14] S. Chakravarthy. Sentinel: an object-oriented DBMS with highscalability.com/blog/2010/9/21/playfishs-social- event-based rules. In Proc. of SIGMOD, 1997. gaming-architecture-50-million-monthly-user.html. [15] F. Chang, J. Dean, S. Ghemawat, W. C. Hsieh, D. A. Wal- [35] T. Hoff. The architecture that Twitter uses to deal with lach, M. Burrows, T. Chandra, A. Fikes, and R. E. Gruber. 150m active users, 300k qps, a 22 mb/s firehose, and Bigtable: A distributed storage system for structured data. send tweets in under 5 seconds. High Scalability, July ACM Transactions on Computer Systems, 2008. 2013. http://highscalability.com/blog/2013/7/8/the- architecture-twitter-uses-to-deal-with-150m-active- [16] K. Conaway. Pyscrabble. http://pyscrabble.sourceforge. users.html. net/. [36] M. Humphries. Ellen DeGeneres crashes Twitter with [17] J. C. Corbett, J. Dean, M. Epstein, A. Fikes, C. Frost, Oscar selfie, 2014. http://www.geek.com/mobile/ellen- J. J. Furman, S. Ghemawat, A. Gubarev, C. Heiser, degeneres-crashes-twitter-with-an-oscars-selfie- P. Hochschild, W. Hsieh, S. Kanthak, E. Kogan, H. Li, 1586464/. A. Lloyd, S. Melnik, D. Mwaura, D. Nagle, S. Quinlan, R. Rao, L. Rolig, Y. Saito, M. Szymaniak, C. Taylor, [37] Apple iCloud, 2016. https://www.icloud.com/. R. Wang, and D. Woodford. Spanner: Google’s globally- [38] A. Incubator. Apache Usergrid. http://usergrid.apache. distributed database. In Proc. of OSDI, 2012. org/. [18] M. Dahlin, L. Gao, A. Nayate, A. Venkataramana, P. Yala- [39] JavaCPP: The missing bridge between Java and native gandula, and J. Zheng. PRACTI replication. In Proc. of C++. github, Mar 2016. https://github.com/bytedeco/ NSDI, 2006. javacpp.
736 12th USENIX Symposium on Operating Systems Design and Implementation USENIX Association [40] Jetty web server. http://www.eclipse.org/jetty/. [59] openio. openio.io: object storage grid for apps. http: [41] R. K. Jonas Boner, Dave Farley and M. Thompson. The re- //openio.io/. active manifesto, Sept 2014. http://www.reactivemanifesto. [60] Parse, 2015. http://www.parse.com. org/. [61] D. Perkins, N. Agrawal, A. Aranya, C. Yu, Y. Go, H. V. [42] J. J. Kistler and M. Satyanarayanan. Disconnected op- Madhyastha, and C. Ungureanu. Simba: tunable end-to- eration in the coda file system. ACM Transactions on end data consistency for mobile apps. In Proc. of EuroSys, Computer Systems, 1992. 2015. [43] T. Kraska, G. Pang, M. J. Franklin, S. Madden, and [62] D. R. K. Ports, A. T. Clements, I. Zhang, S. Madden, and A. Fekete. MDCC: multi-data center consistency. In B. Liskov. Transactional consistency and automatic man- Proc. of EuroSys, 2013. agement in an application data cache. In Proc. of OSDI, [44] A. Lakshman and P. Malik. Cassandra: A decentralized 2010. structured storage system. ACM SIGOPS Operating Sys- [63] PostgreSQL, 2013. http://www.postgresql.org/. tems Review, 2010. [64] React: A JavaScript library for building user interfaces. [45] C. Leau. Spring Data Redis - Retwis-J, Github, 2016. https://facebook.github.io/react/. 2013. http://docs.spring.io/spring-data/data- [65] ReactiveX: An api for asynchronous programming with keyvalue/examples/retwisj/current/. observable streams, 2016. http://reactivex.io/. [46] K. Li and P. Hudak. Memory coherence in shared vir- [66] Redis. Wait numslaves timeout. http://redis.io/commands/ tual memory systems. ACM Transactions on Computer WAIT. Systems, 1989. [67] Redis: Open source data structure server, 2013. http:// [47] B. F. Lieuwen, N. Gehani, and R. Arlein. The Ode active redis.io/. database: trigger semantics and implementation. In Proc. of ICDE, Feb 1996. [68] Riak, 2015. http://basho.com/products/riak-kv/. [48] A. Margara and G. Salvaneschi. We have a DREAM: [69] M. Shapiro, N. Preguic¸a, C. Baquero, and M. Zawirski. Distributed reactive programming with consistency guar- Conflict-free replicated data types. In Proc. of SSS, 2011. antees. In Proc. of DEBS. ACM, 2014. [70] socketcluster.io. socketcluster.io: a scalable framework for [49] Markets and Markets. Backend as a service (BaaS) realtime apps and microservices. http://socketcluster.io/#! market worth 28.10 billion USD by 2020. http://www. /. marketsandmarkets.com/PressReleases/baas.asp. [71] Sqlite home page, 2015. https://www.sqlite.org/. [50] martypdx. Firebase data consistency across [72] Square cash. https://cash.me/. multiple nodes. Stack Overflow, Apr 2015. http://stackoverflow.com/questions/29947898/firebase- [73] M. Stonebraker and J. M. Hellerstein. Readings in data-consistency-across-multiple-nodes. Database Systems. Morgan Kaufmann San Francisco, 1998. [51] Meteor, 2015. http://www.meteor.com. [74] J. Strauss, J. M. Paluska, C. Lesniewski-Laas, B. Ford, [52] M. Mode. New mobile apps revolutionize how orga- R. Morris, and M. F. Kaashoek. Eyo: Device-transparent nizations respond to crises and operations issues, Aug personal storage. In Proc. of USENIX ATC, 2011. 2014. http://www.missionmode.com/new-mobile-apps- revolutionize-organizations-respond-crises-operations- [75] J. Stribling, Y. Sovran, I. Zhang, X. Pretzer, J. Li, M. F. issues/. Kaashoek, and R. Morris. Flexible, wide-area storage for distributed systems with WheelFS. In Proc. of NSDI, 2009. [53] MongoDB, 2015. https://www.mongodb.org. [76] R. Sumbaly, J. Kreps, L. Gao, A. Feinberg, C. Soman, and [54] I. Moraru, D. G. Andersen, and M. Kaminsky. There is S. Shah. Serving large-scale batch computed data with more consensus in egalitarian parliaments. In Proc. of Project Voldemort. In Proc. of FAST, 2012. SOSP, 2013. [77] D. B. Terry, M. M. Theimer, K. Petersen, A. J. Demers, [55] Mozilla. Kinto. http://kinto.readthedocs.org/en/latest/. M. J. Spreitzer, and C. H. Hauser. Managing update con- [56] R. Mulia. Firebase - maintain/guarantee con- flicts in bayou, a weakly connected replicated storage sys- sistency. Stack Overflow, Jan 2016. http: tem. In Proc. of SOSP, 1995. //stackoverflow.com/questions/34678083/firebase- [78] Global social gaming market to reach US$17.4 bn maintain-guarantee-data-consistency. by 2019 propelled by rising popularity of fun games. [57] A. Muthitacharoen, B. Chen, and D. Mazieres.` A low- Transparency Market Research Press Release, Sept bandwidth network file system. In Proc. of SOSP, 2001. 2015. http://www.transparencymarketresearch.com/ pressrelease/social-gaming-market.htm. [58] B. M. Oki and B. H. Liskov. Viewstamped replication: A new primary copy method to support highly-available [79] Twimight open-source Twitter client for Android, 2013. distributed systems. In Proc. of PODC, 1988. http://code.google.com/p/twimight/.
USENIX Association 12th USENIX Symposium on Operating Systems Design and Implementation 737 [80] Twitter. Twitter developer API, 2014. https://dev.twitter. com/overview/api. [81] Venmo. https://venmo.com/. [82] Z. Wan and P. Hudak. Functional reactive programming from first principles. In Proc. of PLDI, 2000. [83] W. E. Weihl. Local atomicity properties: modular concur- rency control for abstract data types. ACM Trans. Prog. Lang. Syst., 1989. [84] I. Zhang, N. K. Sharma, A. Szekeres, A. Krishnamurhty, and D. R. K. Ports. Building consistent transactions with inconsistent replication. In Proc. of SOSP, 2015. [85] I. Zhang, A. Szekeres, D. V. Aken, I. Ackerman, S. D. Gribble, A. Krishnamurthy, and H. M. Levy. Customizable and extensible deployment for mobile/cloud applications. In Proc. of OSDI, 2014.
738 12th USENIX Symposium on Operating Systems Design and Implementation USENIX Association