P2P Networking and Applications

John F. Buford

Heather Yu

Eng Keong Lua

AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO u' SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO MORGAN Morean Kaufmann Publishers is an imprint of Elsevier Table of Contents

Preface xvii About the Authors xxi

CHAPTER 1 Introduction 1 P2P Emerges as a Mainstream Application 1 The Rise of P2P File-Sharing Applications 1 Voice over P2P (VoP2P) 4 P2PTV 5 P2P Networking and the Internet 8 P2P Overlays and Network Services 8 Impact of P2P Traffic on the Internet 10 Motivation for P2P Applications 11 P2P from the End User's Perspective 11 Is P2P = Piracy? 12 P2P Strengths and Benefits 12 P2P Open Issues 14 P2P Economics 14 The P2P Value Proposition 14 Barrier to Entry 15 Revenue Models and Revenue Collection 15 P2P Application Critical Mass 16 Anatomy of Some P2P Business Models 17 VoP2P 17 File Sharing 19 Social Impact 21 Technology Trends Impacting P2P 22 Summary 23 Further Reading 23

CHAPTER 2 Peer-to-Peer Concepts 25 Operation of a P2P System 25 The User View 25 P2P Beyond the Desktop Computer 26 Overlay View 28 Principles of the P2P Paradigm 29 A Graph Theoretic Perspective 31 Overview 31 Overlay 31 Graph Properties 33 Object Storage and Lookup 34

vi Table of Contents

A Design Space Perspective 35 A Routing Performance Perspective 36 Routing Geometries and Resilience 36 Tradeoff Between Routing State and Path Distance 37 Churn and Maintaining the Overlay 37 Locality 39 An Implementation Perspective: OverlayWeaver 41 Summary 43 For Further Reading 44

CHAPTER 3 Unstructured Overlays 45 Connecting Peers on a Global Scale 45 Basic Routing in Unstructured Overlays 46 Flooding and Expanding Ring 46 Random Walk 49 Unstructured Topology Considerations 50 Types of Unstructured Graphs 50 Random Graphs 50 Power-Law Random Graphs 51 Scale-Free Graphs and Self-Similarity 52 Social Networks and the Small-World Phenomenon 54 Early Systems 55 Napster 55 Gnutella 55 FastTrack 55 Freenet 56 Improving on Flooding and Random Walk 61 Techniques 61 Metrics 62 Case Study: Gia 62 Social Overlays 64 Using Similar Interests Among Peers 64 Tribler 65 INGA 67 Key-based Routing in Unstructured Topologies 69 Overview 69 Local Minima Search 69 Unstructured Distributed Hash Table 70 Under the Hood: An Overlay Emulator 72 OverlayWeaver Routing Layer 72 Unstructured Overlays in OverlayWeaver 73 Summary 73 For Further Reading 73 Table of Contents ix

CHAPTER 4 Structured Overlays: Geometry and Routing 75 Structured Overlays 75 Motivation and Categories 75 Geometry and Routing 77 Roadmap for the Chapter 77 Logarithmic Degree with Prefix Routing 79 PRR 79 Tapestry 82 P-Grid 84 Pastry 85 Other Prefix-Routing Overlays 86 Ring with Embedded Logarithmic Degree Mesh 86 Chord 86 DKS(N,k,f) 88 Chord* 89 Constant Degree 91 Features of Constant Degree Graphs 91 Koorde 93 Ulysses 93 Cycloid 94 Other Distance Metrics 95 Content Addressable Network (CAN) 95 Kademlia 97 0(l)-Hop Routing 99 Multihop Versus One-Hop 99. Kelips 100 OneHop 100 EpiChord 101 Comparison and Evaluation 101 Analytical Performance Bounds 101 Measurement Through Simulation 103 Summary 104 For Further Reading 104 Surveys and Frameworks 104 History of Distributed Hash Tables 105 Other Structured Overlays 105 Routing and Geometry in Computer Networks 105

CHAPTER 5 Structured Overlays: Maintenance and Dynamics 107 Peer Churn 107 Approaches to Overlay Maintenance 109 Active Maintenance 109 Opportunistic Maintenance 109 x Table of Contents

Overlay Maintenance Algorithms 110 Logarithmic Degree with Prefix Routing 110 Ring with Embedded Logarithmic Degree Mesh 114 Constant Degree 116 O(l)-Hop Routing 119 Stochastic Modeling of Peer Churn 123 The Network Model 123 Stochastic Model for Long-range Connections 124 Maintenance of Short-range Connections 125 Maintenance of Long-range Connections 125 Comparison with Existing DHT Overlay 126 Federated Overlay Topologies 126 Universal Overlay 128 Hierarchical Overlays 129 Summary 133 For Further Reading 133

CHAPTER 6 Peer-to-Peer in Practice 135 P2P Building Blocks 135 Network Programming 138 Overlay Protocol Design 139 General Protocol Issues 139 Unstructured Overlay: Gnutella 140 BitTorrent 144 Structured Overlays 146 Network Address Translation and P2P Overlays 148 How NAT Effects P2P Connectivity 148 NAT Traversal 150 NAT Traversal with ICE 151 NAT Traversal in a P2P Overlay 153 Peer Capability Determination 153 Overview 153 Network Capacity 154 Peer Lifetime 155 Bootstrapping and Partitions 155 Finding a Rendezvous Peer 155 Merging Partitions 156 P2P Networking Support in Microsoft Windows 157 Peer Identity 157 Peer Name Resolution Protocol 158 Peer Overlay 158 Grouping 159 Identity Management 160 Search 161 Table of Contents xi

Summary l6l For Further Reading l6l

CHAPTER 7 Search 163 Overview 163 Centralized vs. Localized vs. Distributed Indexing 164 Centralized Indexing 164 Localized Indexing 166 Distributed Indexing 167 Hybrid Indexing 167 Hashing-Based Indexing and Lookups 168 Searching in a Flat DHT 168 Searching in Hierarchical DHTs 169 Discussion 169 Searching in Unstructured Overlays 170 Flooding-Based Search 170 Iterative Deepening 171 Random Walk-Based Search 171 Guided Search 171 Hybrid-Based Approaches 172 Keyword Search 173 Range Queries 173 Non-DHT-Based Approaches 174 Range Queries in DHTs 174 Skip Graphs 175 Semantic Queries 176 Semantic Search in Structured Overlays 176 Semantic Search in Unstructured Overlays 178 Advanced Topics 179 Distributed Pattern Matching System (DPMS) 179 DiffSearch 179 Content-Based Search 180 Summary 181 For Further Reading 181

CHAPTER 8 Peer-to-Peer Content Delivery 183 Content Delivery 183 Classification of P2P Content Delivery Schemes 184 Design Criteria 187 P2P Caching 187 Design Issues 191 Example P2P Caching Systems 193 Summary 195 xii Table of Contents

Content Pull and Content Push 195 Case Study 196 Hybrid CDN and P2P Architectures 198 Overview 199 Case Study 199 Summary 201 For Further Reading 202

CHAPTER 9 Peercasting and Overlay Multicasting 203 Introduction 203 The Television Paradigm Shift 203 Popular Peercasting Applications 203 Terminology 204 P2P Streaming 206 Multicast Applications and P2P Overlay Multicast 209 Multicast Applications 209 IP Multicast vs. Overlay Multicast 209 Hybrid Multicast 213 Proxy-Based Overlay Multicast 214 OM Design Considerations 215 Performance Metrics 215 OM Groups and OM Sessions 215 Group Management 216 Message Dissemination 218 Categorization of OM Systems 219 Improving OM performance 221 Summary 227 For Further Reading 228

CHAPTER 10 Measurement for P2P Overlays 229 Motivation 229 Network Embedding 234 Basic Properties of Network Embedding 234 Lipschitz Embedding 236 Numerical Optimization Embedding 237 Internet Coordinate Systems 237 Systems using Lipschitz Embedding and Matrix Factorization 241 Systems using Numerical Optimization 242 Meridian 246 Multiresolution Rings 246 Ring Membership Management 247 Table of Contents xiii

Gossip-based Node Discovery 248 Closest-Node Discovery 248 Accuracy and Overhead 249 Summary 252 For Further Reading 253

CHAPTER 11 Service Overlays 255 Service Orientation and P2P Networking 255 Service Overlay Concepts 257 Resource Virtualization 257 Service Orientation 257 Devices as Peers 258 Serving DNS Records From an Overlay 258 Domain Name Service 258 DDNS 259 Resilient Overlay Networks 262 Internet Routing and ISP Peering 262 Resilient Overlay Network 262 Bandwidth-Aware RON 264 QoS Aware Overlays 266 Overview 266 OverQoS 268 QRON 269 Service Orientation 270 Overview 270 Wide-Area Service Discovery 270 INS/Twine 271 Location-Based Service Discovery 271 Other P2P Approaches to Service Discovery 272 Replication and Load Balancing 272 Churn and Index Availability 272 Index Load and Object Popularity 274 Beehive 275 Service Composition 276 Summary 277 For Further Reading 278

CHAPTER 12 Voice Over Peer-to-Peer 279 From VoIP to VoP2P 279 VoP2P 280 VoIP Elements 280 Mapping VoIP Elements to a VoP2P Overlay 281 Application Relays 282 Types of Relays 282 xiv Table of Contents

Relay Selection and Discovery 283 Dynamic Path Switching 285 Call Processing 285 Overview 285 Assumptions 286 Dimensionality 286 Example Peer Features 287 Case Study: Skype 289 Case Study: Peer-to-peer SIP 291 Overview 291 Hip-Hop 292 Address Settlement by Peer-to-Peer 294 Reload 295 Summary 296 For Further Reading 297

CHAPTER 13 Mobility and Heterogeneity 299 Impact of Mobile Devices on P2P Overlays 299 P2P Overlay Issues Caused by Mobility 300 Roaming and Node Lifetime 300 Growing Mobile Peer Frequency 302 Mitigating Mobility Churn 303 Mobile IP Support 303 Stealth Nodes 304 Bristle 305 Warp 306 Multihomed Peers 308 Variable-Hop Overlays 309 Overview 309 Accordion 309 P2PandManets 311 Overview 311 Mobile Hash Table 312 MADPastry 314 Other P2P MANET Designs 315 Summary 317 For Further Reading 317

CHAPTER 14 Security 319 Introduction 319 Security Risks and Attacks 320 Classifications of Attacks 320 The P2P Security Gap 322 Sample Attacks and Threats 323 Overlay Layer Attacks 326 Table of Contents xv

Security Mechanisms 328 Cryptographic Solutions 328 DoS Countermeasures 329 Secure Routing in Structured P2P 330 Fairness in Resource Sharing 333 Trust and Privacy Issues 334 Architecture 335 Reputation 336 Privacy 337 Case Study: Groove 338 Case Study: Pollution in File-Sharing Systems 338 Summary 339 For Further Reading 340

CHAPTER 15 Managed Overlays 341 Introduction 341 Management of Overlays vs. Conventional Networks 343 Overlay Dimensions Impacting Manageability 343 Managed Overlay Model 345 Managed Overlays and Overlay Operators 346 Role of the Overlay Operator 346 Examples 348 Managing a Resilient Overlay Network 348 Managing a Distributed File Storage Service 350 Overlay Management Architecture 352 Integration with Peer State and Event Detection 352 Security Considerations 354 Generality for Various Types of Overlay 354 Overlay Messaging for Management Operations 354 Reaching All Peers 354 Aggregating Data Collection for Performance Management 355 Multicast 357 Managing the Impact of the Overlay Traffic on the ISP Network 357 P2P Traffic in the ISP Network 357 Approaches to Managing P2P Traffic 358 P4P 358 Summary 359 For Further Reading 359

Glossary Зб1 References 367 Index 399