Proof-of-Execution: Reaching Consensus through Fault-Tolerant Speculation Suyash Gupta Jelle Hellings Sajjad Rahnama Mohammad Sadoghi Exploratory Systems Lab Department of Computer Science University of California, Davis ABSTRACT a common order for these requests [9, 21, 29, 38, 51]. Furthermore, Multi-party data management and blockchain systems require bft consensus comes with the added benefit of democracy, as data sharing among participants. To provide resilient and consis- bft consensus gives all replicas an equal vote in all agreement tent data sharing, transactions engines rely on Byzantine Fault- decisions, while the resilience of bft can aid in dealing with the Tolerant consensus (bft), which enables operations during fail- billions of dollars losses associated with prevalent attacks on data ures and malicious behavior. Unfortunately, existing bft proto- management systems [44]. cols are unsuitable for high-throughput applications due to their Akin to commit protocols, the majority of bft consensus pro- high computational costs, high communication costs, high client tocols use a primary-backup model in which one replica is des- latencies, and/or reliance on twin-paths and non-faulty clients. ignated the primary that coordinates agreement, while the re- In this paper, we present the Proof-of-Execution consensus pro- maining replicas act as backups and follow the protocol [46]. tocol (PoE) that alleviates these challenges. At the core of PoE are This primary-backup bft consensus was first popularized by the out-of-order processing and speculative execution, which allow influential Pbft consensus protocol of Castro and Liskov [9]. The PoE to execute transactions before consensus is reached among design of Pbft requires at least 3f ¸1 replicas to deal with up-to-f the replicas. With these techniques, PoE manages to reduce the malicious replicas and operates in three communication phases, costs of bft in normal cases, while guaranteeing reliable con- two of which necessitate quadratic communication complexity. sensus for clients in all cases. We envision the use of PoE in As such, Pbft is considered costly when compared to commit or high-throughput multi-party data-management and blockchain replication protocols, which has negatively impacted the usage systems. To validate this vision, we implement PoE in our effi- of bft consensus in large-scale data management systems [8]. cient ResilientDB fabric and extensively evaluate PoE against The recent interest in blockchain technology has revived in- several state-of-the-art bft protocols. Our evaluation showcases terest in bft consensus, has led to several new resilient data that PoE achieves up-to-80% higher throughputs than existing management systems (e.g., [3, 18, 29, 43]), and has led to the bft protocols in the presence of failures. development of new bft consensus protocols that promise effi- ciency at the cost of flexibility (e.g., [21, 28, 38, 51]). Despite the existence of these modern bft consensus protocols, the majority 1 INTRODUCTION of bft-fueled systems [3, 18, 29, 43] still employ the classical In federate data management a single common database is man- time-tested, flexible, and safe design of Pbft, however. aged by many independent stakeholders (e.g., an industry con- In this paper, we explore different design principles that can sortium). In doing so, federated data management can ease data enable implementing a scalable and reliable agreement protocol sharing and improve data quality [17, 32, 48]. At the core of fed- that shields against malicious attacks. We use these design princi- erated data management is reaching agreement on any updates ples to introduce Proof-of-Execution (PoE), a novel bft protocol on the common database in an efficient manner, this to enable that achieves resilient agreement in just three linear phases. To fast query processing, data retrieval, and data modifications. concoct PoE’s scalable and resilient design, we start with Pbft One can achieve federated data management by replicating and successively add four design elements: the common database among all participant, this by replicat- (I1) Non-Divergent Speculative Execution. In Pbft, when ing the sequence of transactions that affect the database to all the primary replica receives a client request, it forwards that stakeholders. One can do so using commit protocols designed request to the backups. Each backup on receiving a request from for distributed databases such as two-phase [22] and three-phase the primary agrees to support by broadcasting a prepare message. commit [49], or by using crash-resilient replication protocols When a replica receives prepare message from the majority of arXiv:1911.00838v3 [cs.DB] 23 Feb 2021 such as Paxos [39] and Raft [45]. other replicas, it marks itself as prepared and broadcasts a commit These solutions are error-prone in a federated decentralized message. Each replica that has prepared, and receives commit environment in which each stakeholder manages its own replicas messages from a majority of other replicas, executes the request. and replicas of each stakeholder can fail (e.g., due to software, Evidently, Pbft requires two phases of all-to-all communica- hardware, or network failure) or act malicious: commit protocols tion. Our first ingredient towards faster consensus is speculative and replication protocols can only deal with crashes. Conse- execution. In Pbft terminology, PoE replicas execute requests quently, recent federated designs propose the usage of Byzantine after they get prepared, that is, they do not broadcast commit mes- Fault-Tolerant (bft) consensus protocols. bft consensus aims at sages. This speculative execution is non-divergent as each replica ordering client requests among a set of replicas, some of which could has a partial guarantee–it has prepared–prior to execution. be Byzantine, such that all non-faulty replicas reach agreement on (I2) Safe Rollbacks and Robustness under Failures. Due to speculative execution, a malicious primary in PoE can en- © 2021 Copyright held by the owner/author(s). Published in Proceedings of the 24th International Conference on Extending Database Technology (EDBT), March sure that only a subset of replicas prepare and execute a request. 23-26, 2021, ISBN 978-3-89318-084-4 on OpenProceedings.org. Hence, a client may or may not receive a sufficient number of Distribution of this paper is permitted under the terms of the Creative Commons matching responses. PoE ensures that if a client receives a full license CC-by-nc-nd 4.0. Protocol Phases Messages Resilience Requirements proof-of-execution, consisting of responses from a majority of the Zyzzyva 1 O¹nº 0 Reliable clients and unsafe non-faulty replicas, then such a request persists in time. Other- PoE (our paper) 3 O¹3nº f Sign. agnostic 2 wise, PoE permits replicas to rollback their state if necessary. This Pbft 3 O¹n ¸ 2n º f HotStuff 8 O¹8nº f Sequential Consensus proof-of-execution is the cornerstone of the correctness of PoE. SBFT 5 O¹5nº 0 Optimistic path (I3) Agnostic Signatures and Linear Communication. bft protocols are run among distrusting parties. To provide secu- Figure 1: Comparison of bft consensus protocols in a sys- rity, these protocols employ cryptographic primitives for signing tem with n replicas of which f are faulty. The costs given the messages and generating message digests. Prior works have are for the normal-case behavior. shown that the choice of cryptographic signature scheme can impact the performance of the underlying system [9, 30]. Hence, we allow replicas to either employ message authentication codes Hyperledger Fabric [4]; in multi-primary meta-protocols such as (MACs) or threshold signatures (TSs) for signing [36]. When few RCC [26, 28]; and in sharding protocols such as AHL [15]. replicas are participating in consensus (up to 16), then a single phase of all-to-all communication is inexpensive and using MACs 2 ANALYSIS OF DESIGN PRINCIPLES for such setups can make computations cheap. For larger setups, To arrive at an optimal design for PoE, we studied practices fol- TS TS we employ s to achieve linear communication complexity. s lowed by state-of-the-art distributed data management systems permit us to split a phase of all-to-all communication into two and applied their principles to the design of PoE where possi- [21, 51]. linear phases ble. In Figure 1, we present a comparison of PoE against four (I4) Avoid Response Aggregation. SBFT [21], a recently- well-known resilient consensus protocols. proposed bft protocol, suggests the use of a single replica (desig- To illustrate the merits of PoE’s design, we first briefly look at nated as the executor) to act as a response aggregator. In specific, Pbft. The last phase of Pbft ensures that non-faulty replicas only all replicas execute each client request and send their response to execute requests and inform clients when there is a guarantee the executor. It is the duty of the executor to reply to the client that such a transaction will be recovered after any failures. Hence, and send a proof that a majority of the replicas not only executed clients need to wait for only f ¸ 1 identical responses, of which this request, but also outputted the same result. In PoE, we avoid at-least one is from a non-faulty replica, to ensure guaranteed this additional communication between the replicas by allowing execution. By eliminating this last phase, replicas speculatively each replica to respond directly to the client. execute requests before obtaining recovery guarantees. This im- In specific, we make the following contributions: pacts Pbft-style consensus in two ways: (1) We introduce PoE, a novel Byzantine fault-tolerant con- (1) First, clients need a way to determine proof-of-execution sensus protocol that uses speculative execution to reach after which they have a guarantee that their requests are agreement among replicas.