FastFabric: Scaling Hyperledger Fabric to 20,000 Transactions per Second Christian Gorenflo Stephen Lee Lukasz Golab, S. Keshav University of Waterloo University of Massachusetts University of Waterloo Waterloo, Canada Amherst, USA Waterloo, Canada Email: cgorenfl[email protected] Email: [email protected] Email: [lgolab, keshav]@uwaterloo.ca Abstract—Blockchain technologies are expected to make a work [3], [4], motivating us to look beyond consensus to significant impact on a variety of industries. However, one identify further performance improvements. issue holding them back is their limited transaction throughput, In this paper, we critically examine the design of Hyper- especially compared to established solutions such as distributed database systems. In this paper, we re-architect a modern ledger Fabric 1.2 since it is reported to be the fastest available permissioned blockchain system, Hyperledger Fabric, to in- open-source permissioned blockchain [5]. While there has crease transaction throughput from 3,000 to 20,000 transactions been some work on optimizing Hyperledger Fabric, e.g., using per second. We focus on performance bottlenecks beyond the aggressive caching [6], we are not aware of any prior work consensus mechanism, and we propose architectural changes on re-architecting the system as a whole2. We hence design that reduce computation and I/O overhead during transaction ordering and validation to greatly improve throughput. Notably, and implement several architectural optimizations based on our optimizations are fully plug-and-play and do not require any common system design techniques that together improve the interface changes to Hyperledger Fabric. end-to-end transaction throughput by a factor of almost 7, from 3,000 to 20,000 transactions per second, while decreasing I. INTRODUCTION block latency. Our specific contributions are as follows: Distributed ledger technologies such as blockchains offer a 1) Separating metadata from data: the consensus layer in way to conduct transactions in a secure and verifiable manner Fabric receives whole transactions as input, but only the without the need for a trusted third party. As such, it is widely transaction IDs are required to decide the transaction believed that blockchains will significantly impact industries order. We redesign Fabric’s transaction ordering service ranging from finance and real estate to public administration, to work with only the transaction IDs, resulting in energy and transportation [1]. However, in order to be viable greatly increased throughput. in practice, blockchains must support transaction rates com- 2) Parallelism and caching: some aspects of transaction parable to those supported by existing database management validation can be parallelized while others can benefit systems, which can provide some of the same transactional from caching transaction data. We redesign Fabric’s guarantees. transaction validation service by aggressively caching In contrast to permissionless blockchains, which do not unmarshaled blocks at the comitters and by parallelizing restrict network membership, we focus on permissioned as many validation steps as possible, including endorse- blockchains, in which the identities of all participating ment policy validation and syntactic verification. nodes are known. Permissioned blockchains are suitable for 3) Exploiting the memory hierarchy for fast data access on many application domains including finance; e.g., the Ripple1 the critical path: Fabric’s key-value store that maintains blockchain aims to provide a payment network for currency world state can be replaced with light-weight in-memory arXiv:1901.00910v2 [cs.DC] 4 Mar 2019 exchange and cross-bank transactions akin to the current data structures whose lack of durability guarantees can SWIFT system. be compensated by the blockchain itself. We redesign From a technical standpoint, we observe an important dis- Fabric’s data management layer around a light-weight tinction between these two types of blockchains. The trustless hash table that provides faster access to the data on the nature of permissionless blockchains requires either proofs critical transaction-validation path, deferring storage of of work or stake, or expensive global-scale Byzantine-fault- immutable blocks to a write-optimized storage cluster. tolerant consensus mechanisms [2]. Much recent work has 4) Resource separation: the peer roles of committer and focused on making these more efficient. On the other hand, endorser vie for resources. We introduce an architecture permissioned blockchains typically delegate consensus and that moves these roles to separate hardware. transaction validation to a selected group of nodes, reducing Importantly, our optimizations do not violate any APIs or the burden on consensus algorithms. While even in this setting modularity boundaries of Fabric, and therefore they can be consensus remains a bottleneck, it is being addressed in recent incorporated into the planned release of Fabric version 2.0 [7]. 1https://ripple.com 2Please refer to Section V for a detailed survey of related work. We also outline several directions for future work, which, C. Implementation details together with the optimization we propose, have the potential To set the stage for a discussion of the improvements in to reach 50,000 transactions per second as required by a credit Section III, we now take a closer look at the orderer and peer card company such as Visa [2]. architecture. In the remainder of this paper, Section II gives a brief 1) Orderer: After receiving responses from endorsing overview of Hyperledger Fabric, Section III presents our peers, a client creates a transaction proposal containing a improved design, Section IV discusses experimental results, header and a payload. The header includes the Transaction Section V places our contributions in the context of prior work, ID and Channel ID4. The payload includes the read-write sets and Section VI concludes with directions for future work. and the corresponding version numbers, and endorsing peers’ signatures. The transaction proposal is signed using the client’s II. FABRIC ARCHITECTURE credentials and sent to the ordering service. A component of the open-source Hyperledger project hosted The two goals of the ordering service are (a) to achieve con- by the Linux Foundation, Fabric is one of the most actively sensus on the transaction order and (b) to deliver blocks con- developed permissioned blockchain systems [8]. Since An- taining ordered transactions to the committer peers. Fabric cur- droulaki et al [9] describe the transaction flow in detail, we rently uses Apache Kafka, which is based on ZooKeeper [10], present only a short synopsis, focusing on those parts of the for achieving crash-fault-tolerant consensus. system where we propose improvements in Section III. When an orderer receives a transaction proposal, it checks To avoid pitfalls with smart-contract determinism and to if the client is authorized to submit the transaction. If so, the allow plug-and-play replacement of system components, Fab- orderer publishes the transaction proposal to a Kafka cluster, ric is structured differently than other common blockchain where each Fabric channel is mapped to a Kafka topic to systems. Transactions follow an execute-order-commit flow create a corresponding immutable serial order of transactions. pattern instead of the common order-execute-commit pattern. Each orderer then assembles the transactions received from Client transactions are first executed in a sandbox to determine Kafka into blocks, based either on the maximum number their read-write sets, i.e., the set of keys read by and written of transactions allowed per block or a block timeout period. to by the transaction. Transactions are then ordered by an Blocks are signed using the orderer’s credentials and delivered ordering service, and finally validated and committed to the to peers using gRPC [11]. blockchain. This workflow is implemented by nodes that are 2) Peer: On receiving a message from the ordering service assigned specific roles, as discussed next. a peer first unmarshals the header and metadata of the block A. Node types and checks its syntactic structure. It then verifies that the signatures of the orderers that created this block conform to Clients originate transactions, i.e., reads and writes to the the specified policy. A block that fails any of these tests is 3 blockchain, that are sent to Fabric nodes . Nodes are either immediately discarded. peers or orderers; some peers are also endorsers. All peers After this initial verification, the block is pushed into a commit blocks to a local copy of the blockchain and apply queue, guaranteeing its addition to the blockchain. However, the corresponding changes to a state database that maintains before that happens, blocks go sequentially through two vali- a snapshot of the current world state. Endorser peers are dation steps and a final commit step. permitted to certify that a transaction is valid according to During the first validation step, all transactions in the block business rules captured in chaincode, Fabric’s version of are unpacked, their syntax is checked and their endorsements smart contracts. Orderers are responsible solely for deciding are validated. Transactions that fail this test are flagged as transaction order, not correctness or validity. invalid, but are left in the block. At this point, only transactions B. Transaction flow that were created in good faith are still valid. In the second validation step,