Efficient and Egalitarian Consensus by Ling Ren B.Eng., Tsinghua University (2012) S.M., Massachusetts Institute of Technology (2014) Submitted to the Department of Electrical Engineering and Computer Science in partial fulfillment of the requirements for the degree of Doctor of Philosophy at the MASSACHUSETTS INSTITUTE OF TECHNOLOGY September 2018 ○c Massachusetts Institute of Technology 2018. All rights reserved. Author................................................................................... Department of Electrical Engineering and Computer Science August 10, 2018 Certified by. Srinivas Devadas Professor of Electrical Engineering and Computer Science Thesis Supervisor Accepted by.............................................................................. Leslie A. Kolodziejski Professor of Electrical Engineering and Computer Science Chair, Committee on Graduate Students 2 Efficient and Egalitarian Consensus by Ling Ren Submitted to the Department of Electrical Engineering and Computer Science on August 10, 2018, in partial fulfillment of the requirements for the degree of Doctor of Philosophy Abstract Consensus is a classic problem in distributed computing. Research on consensus has traditionally focused on the permissioned setting where participants are fixed and know each other beforehand. Recently, the digital currency Bitcoin has popularized a new line of research on consensus in a permissionless environment, where participants may join or leave at any time and need not know how many other participants exist or who they are. Bitcoin’s solution, now known as the Nakamoto consensus, is to build a proof-of-work chain and treat the longest proof-of-work chain as consensus decisions. However, this elegant solution does have limitations. First, it has long latency: under current parameters, it can take hours for a Bitcoin transaction to go through. Second, its use of hash-based proof-of-work has raised concerns about fairness and energy consumption. This thesis presents distributed algorithms and cryptographic primitives to address these limitations. I will first describe Solida, a permissionless consensus protocol with low latency. It hasbeen observed that traditional Byzantine consensus protocols have much lower latency than Nakamoto consensus. Following this observation, Solida adapts traditional Byzantine consensus protocols from the permissioned setting to the permissionless setting by combining them with proof-of-work. I also design improved protocols for permissioned synchronous Byzantine consensus. I then turn to potential replacements for hash-based proof-of-work. I construct a proof-of-space protocol with tight security bounds as an energy-efficient alternative. Finally, I revisit the concept of memory-hard functions, the standard approach to improve fairness in proof-of-work. I argue that the memory-hardness approach overlooks energy efficiency fairness and suggest bandwidth-hard functions as egalitarian alternatives. 3 Thesis Supervisor: Srinivas Devadas Title: Professor of Electrical Engineering and Computer Science 4 Acknowledgments I am grateful to many people who have helped me during my PhD. First and foremost, I would like to thank my advisor Srini Devadas. I have enjoyed unlimited academic freedom at MIT thanks to Srini. I remember distinctively a meeting with him before my PhD. In that meeting, he said “work on something that interests you here; if you don’t, I will force you to.” His passion and approach to research and have had a great impact on me. I also very much appreciate his guidance during my academic job search and his advice on many other aspects of life. Next, I would like to thank Marten van Dijk, Elaine Shi, Dahlia Malkhi and Ittai Abraham. It was Ittai and Dahlia who led me into consensus research. A lot of work in this thesis started while I was an intern at VMware Research under their mentorship. Marten and Elaine have effectively served as my unofficial advisors and have been sources of inspiration on many research projects. This thesis is based on joint works with Ittai Abraham, Srinivas Devadas, Dahlia Malkhi, Kartik Nayak and Alexander Spiegelman. This thesis would not have been possible without their contributions. Many others have provided valuable insights and discussions. Vinod Vaikuntanathan, Bryan Ford, and Eleftherios Kokoris-Kogias helped clarify comparisons with their works and shared other inspiring ideas. Alin Tomescu pointed me to efficient libraries for pairing-based cryptography. Joël Alwen, Jeremiah Blocki, Henry Corrigan-Gibbs, Ilia Lebedev, Quanquan Liu, Krzysztof Pietrzak, Daniel Sanchez, Daniel Wichs, and Samson Zhou provided helpful feedback and discussions on various aspects of proof-of-space, bandwidth-hard functions, and graph pebbling. I also had the pleasure to work with many wonderful collaborators on topics outside this thesis: Christopher Fletcher, Xiangyao Yu, Albert Kwon, Emil Stefanov, Charles Herder, Meng-Day Yu, Daniel Wichs, Hubert Chan, Omer Khan, Benny Pinkas, and Hanshen Xiao. Especially to Chris, my closest collaborator, you wrote in your thesis acknowledgment that we would collaborate again in the future and now we are to become colleagues. It is great pleasure to work with David Wentzlaff’s group on a chip tape-out. Special thanks to Mike McKeown, who has spent a considerable amount of time helping us test the chip. I have been fortunate to teach 6.046 Design and Analysis of Algorithms with Erik Demaine, Srini Devadas, Nancy Lynch, Debayan Gutpa, Virginia Williams and other amazing staff members. I learned a lot from them missed those fun staff meetings. Thanks to Nancy Lynch for her trust. I am deeply grateful to my family and friends. The friends I met at MIT have brought so much fun and laughter to the PhD journey. Finally, as always, I am thankful to my parents for their continuous support. 5 6 Contents 1 Introduction 9 1.1 Overview of Contributions . 10 2 Background 13 2.1 Byzantine Broadcast and Byzantine Agreement . 13 2.2 Byzantine Fault Tolerant State Machine Replication . 15 2.3 Nakamoto Consensus . 17 2.4 Cryptographic Primitives . 19 2.4.1 Aggregate Signature . 19 2.4.2 Verifiable Random Function . 20 3 Solida: Permissionless Consensus from Reconfigurable Byzantine Consensus 21 3.1 Introduction . 21 3.1.1 Overview of the Protocol . 23 3.2 Model . 25 3.3 The Solida Protocol . 27 3.3.1 Structure of Views and Leaders . 28 3.3.2 Steady State . 29 3.3.3 View Change . 30 3.3.4 Reconfiguration . 32 3.3.5 Improvement with Aggregate Signatures . 33 3.3.6 Puzzles and Defense against Selfish Mining . 34 3.4 Proof of Safety and Liveness . 34 3.5 Implementation and Evaluation . 38 3.6 Related Work . 41 4 Efficient Synchronous Byzantine Consensus 43 4.1 Introduction . 43 4.2 A Synchronous Byzantine Synod Protocol . 45 4.2.1 Model and Overview . 45 4.2.2 Detailed Protocol . 46 4.2.3 Safety and Liveness . 50 4.3 Byzantine Broadcast and Agreement . 53 4.4 Byzantine Fault Tolerant State Machine Replication . 56 4.4.1 Warm-up: A Basic Protocol . 56 7 4.4.2 Towards Stable Leaders . 58 4.4.3 Common Case . 59 4.4.4 View Change . 61 4.4.5 Safety and Liveness . 63 4.4.6 Message Complexity . 64 4.4.7 Comparison with Prior Work . 65 5 Proof-of-Space from Stacked Expanders 67 5.1 Introduction . 67 5.2 Preliminaries . 69 5.2.1 Proof-of-Space . 69 5.2.2 Graph Labeling and Pebbling . 70 5.2.3 Bipartite and Stacked Expanders . 71 5.3 Pebble Games on Stacked Expanders . 73 5.3.1 Localization of Expanders . 74 5.4 Proof-of-Space from Stacked Expanders . 77 6 Bandwidth-Hard Functions 81 6.1 Introduction . 81 6.2 Bandwidth-Hard Functions: Definition and Limit . 84 6.2.1 Model . 84 6.2.2 Definition: Energy Fairness . 85 6.2.3 Red-Blue Pebble Games . 86 6.2.4 Limit of Energy Fairness . 87 6.3 Bandwidth-Hardness of Candidate Constructions . 89 6.3.1 Scrypt . 90 6.3.2 Bit-Reversal Graphs . 91 6.3.3 Stacked Expanders . 93 6.3.4 Stacked Butterfly Graphs Are Not Bandwidth-Hard . 95 6.3.5 Summary . 96 6.4 Related Work . 97 7 Conclusion 99 8 Chapter 1 Introduction Consensus is a classic problem in distributed computing. Informally, the problem is to ensure that a set of parties agree on a common value despite some parties being faulty. The consensus problem has been studied under various combinations of models and assumptions on communication channels, timings, faults, etc.. This thesis considers consensus in the presence of Byzantine faults [126, 100], meaning that faulty parties may deviate from the protocol arbitrarily in an adversarial and coordinated manner. Studies on Byzantine consensus in academia have traditionally focused on the permissioned setting where participants are fixed and know each other a priori. Recently, the digital currency Bitcoin [114] explored and popularized a new setting of consensus. Bitcoin is a payment system that does not rely on a single authority (like a central bank) but instead runs on a decentralized peer-to-peer network. While consensus has no obvious connection to a centrally controlled fiat currency, it is at the heart of a decentralized digital currency. A key insight of Bitcoin is that, if all non-faulty parties in a peer-to-peer network reach consensus on every transaction (hence on every account’s balance), then we have obtained a payment system or a currency. Bitcoin achieves consensus in a permissionless setting. Participants may join or leave the Bitcoin protocol at will and need not know how many other participants exist or who they are. There is supposed to be no identity verification or barrier to entry. Because there is no identity verification, permissionless consensus faces a threat called the Sybil attack [58]: the attacker can easily create a large number of pseudonymous identities to participate in the protocol. Conventional permissioned consensus protocols usually rely on some form of majority 9 voting, so they will fall victim to the Sybil attack.