The Book of Swarm Storage and Communication Infrastructure for Self-Sovereign Digital Society Back-End Stack for the Decentralised Web
Total Page：16
File Type：pdf, Size：1020Kb
the book of Swarm storage and communication infrastructure for self-sovereign digital society back-end stack for the decentralised web Viktor Trón v1.0 pre-release 7 - worked on November 17, 2020 the swarm is headed toward us Satoshi Nakamoto ii CONTENTS Prolegomena xi Acknowledgments xii i prelude 1 the evolution2 1.1 Historical context 2 1.1.1 Web 1.02 1.1.2 Web 2.03 1.1.3 Peer-to-peer networks 6 1.1.4 The economics of BitTorrent and its limits 7 1.1.5 Towards Web 3.08 1.2 Fair data economy 12 1.2.1 The current state of the data economy 12 1.2.2 The current state and issues of data sovereignty 13 1.2.3 Towards self-sovereign data 15 1.2.4 Artificial intelligence and self-sovereign data 16 1.2.5 Collective information 17 1.3 The vision 18 1.3.1 Values 18 1.3.2 Design principles 19 1.3.3 Objectives 19 1.3.4 Impact areas 20 1.3.5 The future 21 ii design and architecture 2 network 25 2.1 Topology and routing 25 2.1.1 Requirements for underlay network 25 2.1.2 Overlay addressing 26 2.1.3 Kademlia routing 27 2.1.4 Bootstrapping and maintaining Kademlia topology 32 2.2 Swarm storage 35 2.2.1 Distributed immutable store for chunks 35 2.2.2 Content addressed chunks 38 2.2.3 Single-owner chunks 41 2.2.4 Chunk encryption 42 2.2.5 Redundancy by replication 43 2.3 Push and pull: chunk retrieval and syncing 47 iii 2.3.1 Retrieval 47 2.3.2 Push syncing 51 2.3.3 Pull syncing 53 2.3.4 Light nodes 55 3 incentives 57 3.1 Sharing bandwidth 58 3.1.1 Incentives for serving and relaying 58 3.1.2 Pricing protocol for chunk retrieval 59 3.1.3 Incentivising push-syncing 64 3.2 Swap: accounting and settlement 66 3.2.1 Peer to peer accounting 66 3.2.2 Cheques as off-chain commitments to pay 68 3.2.3 Waivers 70 3.2.4 Best effort settlement strategy 71 3.2.5 Zero cash entry 73 3.2.6 Cashing out and risk of insolvency 74 3.2.7 Sanctions and blacklisting 74 3.3 Storage incentives 75 3.3.1 Spam protection with postage stamps 76 3.3.2 Postage lottery: positive incentives for storage 81 3.3.3 Price signalling capacity pressure 86 3.3.4 Insurance: negative incentives 89 3.4 Summary 94 4 high-level functionality 96 4.1 Data structures 96 4.1.1 Files and the Swarm hash 96 4.1.2 Collections and manifests 100 4.1.3 URL-based addressing and name resolution 102 4.1.4 Maps and key–value stores 103 4.2 Access control 103 4.2.1 Encryption 104 4.2.2 Managing access 105 4.2.3 Selective access to multiple parties 106 4.2.4 Access hierarchy 108 4.3 Swarm feeds and mutable resource updates 109 4.3.1 Feed chunks 109 4.3.2 Indexing schemes 111 4.3.3 Integrity 114 4.3.4 Epoch-based indexing 115 4.3.5 Real-time data exchange 117 4.4 Pss: direct push messaging with mailboxing 121 4.4.1 Trojan chunks 122 iv 4.4.2 Initial contact for key exchange 125 4.4.3 Addressed envelopes 128 4.4.4 Notification requests 131 5 persistence 138 5.1 Redundancy, latency and repair 138 5.1.1 Error correcting codes 138 5.1.2 Redundancy by erasure codes 139 5.1.3 Per-level erasure coding in the Swarm chunk tree 139 5.2 Pinning, reupload and missing chunk notifications 142 5.2.1 Local pinning 142 5.2.2 Global pinning 143 5.2.3 Recovery 144 5.3 Insurance 148 5.3.1 Insurance pool 148 5.3.2 Buying insurance on a chunk 148 5.3.3 Uploading with insurance 148 5.3.4 Accessing receipts 148 6 user experience 150 6.1 Configuring and tracking uploads 150 6.1.1 Upload options 150 6.1.2 Upload tags and progress bar 151 6.1.3 Postage 152 6.1.4 Additional upload features 154 6.2 Storage 155 6.2.1 Uploading files 155 6.2.2 Collections and manifests 156 6.2.3 Access control 157 6.2.4 Download 158 6.3 Communication 160 6.3.1 Feeds 160 6.3.2 Pss 160 6.4 Swarm as backend for web3 161 6.4.1 Primitives 161 6.4.2 Scripting 161 6.4.3 Built-in composite APIs 161 6.4.4 Standard library 161 iii specifications List of definitions 165 7 data types and algorithms 168 7.1 Built-in primitives 168 7.1.1 Crypto 168 v 7.1.2 State store 174 7.1.3 Local context and configuration 174 7.2 Bzz address 175 7.2.1 Overlay address 175 7.2.2 Underlay address 175 7.2.3 BZZ address 176 7.3 Chunks, encryption and addressing 177 7.3.1 Content addressed chunks 177 7.3.2 Single owner chunk 179 7.3.3 Binary Merkle Tree Hash 181 7.3.4 Encryption 182 7.4 Files, manifests and data structures 184 7.4.1 Files and the Swarm hash 184 7.4.2 Manifests 187 7.4.3 Resolver 191 7.4.4 Pinning 191 7.4.5 Tags 192 7.4.6 Storage 193 7.5 Access Control 193 7.6 PSS 196 7.6.1 PSS message 196 7.6.2 Direct pss message with trojan chunk 196 7.6.3 Envelopes 200 7.6.4 Update notifications 200 7.6.5 Chunk recovery 200 7.7 Postage stamps 201 7.7.1 Stamps for upload 201 7.7.2 Postage subscriptions 203 7.7.3 Postage lottery race 203 8 protocols 204 8.1 Introduction 204 8.1.1 Underlay network 204 8.1.2 Protocols and streams 204 8.1.3 Data exchange sequences 205 8.1.4 Stream headers 206 8.1.5 Encapsulation of context for tracing 206 8.2 Bzz handshake protocol 207 8.3 Hive discovery 208 8.3.1 Streams and messages 208 8.4 Retrieval 209 8.5 Push-syncing 210 8.6 Pull-syncing 210 vi 8.7 SWAP settlement protocol 210 9 strategies 211 9.1 Connection 211 9.1.1 Connectivity and its constraints 212 9.1.2 Light node connection strategy 212 9.2 Forwarding 212 9.3 Pricing 213 9.3.1 Passive strategy 213 9.3.2 Behavior of a node 215 9.4 Accounting and settlement 216 9.5 Push-syncing 216 9.6 Pull-syncing 216 9.7 Garbage collection 216 10 api-s 218 10.1 External API requirements 219 10.1.1 Signer 219 10.1.2 Blockchain 219 10.1.3 User input 220 10.2 Storage API 221 10.2.1 Chunks 221 10.2.2 File 223 10.2.3 Manifest 225 10.2.4 High level storage API 227 10.2.5 Tags 230 10.2.6 Pinning 232 10.2.7 Swap and chequebook 233 10.2.8 Postage stamps 233 10.2.9 Access Control 236 10.3 Communications 237 10.3.1 PSS 237 10.3.2 Feeds 239 Bibliography 239 iv appendix a formalisations and proofs 245 a.1 Logarithmic distance and proximity order 245 a.2 Proximity orders in graph topology 246 a.3 Constructing kademlia topology 247 a.4 Complexity of filling up a postage stamp batch 247 a.5 Average hop count for chunk retrieval 249 a.6 Distribution of requests 249 a.7 Epoch-based feeds 249 vii v indexes Glossary 255 Index 266 List of acronyms and abbreviations 270 viii LISTOFFIGURES Figure 1 Swarm’s layered design 23 Figure 2 From overlay address space to Kademlia table 28 Figure 3 Nearest neighbours 29 Figure 4 Iterative and Forwarding Kademlia routing 30 Figure 5 Bin density 33 Figure 6 Distributed hash tables (DHTs) 36 Figure 7 Swarm DISC: Distributed Immutable Store for Chunks 37 Figure 8 Content addressed chunk 38 Figure 9 BMT: Binary Merkle Tree hash used as chunk hash in Swarm 39 Figure 10 Compact segment inclusion proofs for chunks 40 Figure 11 Single-owner chunk 41 Figure 12 Chunk encryption in Swarm 43 Figure 13 Nearest neighbours 45 Figure 14 Alternative ways to deliver chunks: direct, routed and backward 48 Figure 15 Backwarding: a pattern for anonymous request-response round- trips in forwarding Kademlia 49 Figure 16 Retrieval 50 Figure 17 Push syncing 52 Figure 18 Pull syncing 54 Figure 19 Incentive design 57 Figure 20 Incentivising retrieval 59 Figure 21 Uniform chunk price across proximities would allow a DoS 61 Figure 22 Price arbitrage 62 Figure 23 Incentives for push-sync protocol 65 Figure 24 Swap balance and swap thresholds 67 Figure 25 Cheque swap 68 Figure 26 The basic interaction sequence for swap chequebooks 69 Figure 27 Example sequence of mixed cheques and waivers exchange 72 Figure 28 Zero cash entry 73 Figure 29 Postage stamps 77 Figure 30 Postage stamp 78 Figure 31 Limiting the size of a postage stamp 80 Figure 32 Timeline of events in a raffle round 83 ix Figure 33 Batch proof of custody 84 Figure 34 Garbage collection 88 Figure 35 Swarm hash 97 Figure 36 Intermediate chunk 97 Figure 37 Random access at arbitrary offset with Swarm hash 99 Figure 38 Compact inclusion proofs for files 99 Figure 39 Manifest structure 101 Figure 40 Manifest entry 101 Figure 41 Access key as session key for single party access 106 Figure 42 Credentials to derive session key 107 Figure 43 Access control for multiple grantees 108 Figure 44 Feed chunk 110 Figure 45 Feed aggregation 113 Figure 46 Epoch grid with epoch-based feed updates 116 Figure 47 Swarm feeds as outboxes 118 Figure 48 Advance requests for future updates 119 Figure 49 Future secrecy for update addresses 120 Figure 50 Trojan chunk or pss message 122 Figure 51 Trojan chunk 123 Figure 52 X3DH pre-key bundle feed update 126 Figure 53 X3DH initial message 127 Figure 54 X3DH secret key 127 Figure 55 Stamped addressed envelopes timeline of events 129 Figure 56 Stamped addressed envelopes 130 Figure 57 Direct notification request and response 132 Figure 58 Direct notification from publisher timeline of events 133 Figure 59 Neighbourhood notifications 134 Figure 60 Neighbourhood notification timeline of events 135 Figure 61 Targeted chunk deliveries 136 Figure 62 Targeted chunk delivery timeline of events 137 Figure 63 Swarm chunk 140 Figure 64 A generic node in the tree has 128 children 140 Figure 65 Swarm hash split 140 Figure 66 Swarm hash erasure 141 Figure 67 Missing chunk notification process 146 Figure 68 Recovery request 146 Figure 69 Recovery response 147 Figure 70 From retrieval to litigation 149 Figure 71 buzz apiary 162 Figure 72 Updates of epoch-based feed in the epoch grid 251 Figure 73 Latest update lookup algorithm for time based feeds 252 x PROLEGOMENA intended audience The primary aim of this book is to capture the wealth of output of the first phase of the Swarm project, and to serve as a compendium for teams and individuals participating in bringing Swarm to life in the forthcoming stages.