Overview – Part 1 1 Scope and Relevance of P2P in Distributed Systems and Networking 1.1 Motivation Advanced Topics www.httc.de 1.2 Evolution of Internet Computing Paradigms 1.3 Success of P2P Networking in Distributed Systems 1.4 P2P Application & Service Domains 2 Specification of Peer-to-Peer Peer-to-Peer, Part 1 www.kom.tu-darmstadt.de 2.1 An Early Definition of P2P 2.2 Nine Characteristics of “Pure” P2P Systems 2.3 P2P Networks are Overlay Networks 2.4 Overlay Structures 3 P2P Applications and Systems 3.1 P2P Applications and Systems: 1st Generation 3.2 P2P Applications and Systems: 2nd Generation Prof. Dr.-Ing. Ralf Steinmetz, Dr.-Ing. Oliver Heckmann 3.3 Some 2nd Generation Applications and Systems Beyond File Sharing 3.4 Some Applications and Systems: 3rd Generation TU Darmstadt – Technical University of Darmstadt Dept. of Electrical Engineering and Information Technology, Dept. of Computer Science 4 Properties of P2P Network Graphs KOM - Multimedia Communications Lab 4.1 Some Metrics Merckstr. 25, D-64283 Darmstadt, Germany, 4.2 Clustering {steinmetz, heckmann}@KOM.tu-darmstadt.de Tel.+49-6151-16-5188, Fax. +49-6151-16-6152 4.3 Average Path Length 4.4 Small World Phenomenon 4.5 Power Law Phenomenon 1 2 Scope and Relevance of P2P 1 Overview – Part 2 in Distributed Systems and Networking 5 Mechanisms for Unstructured P2P Networks 5.1 Broadcast P2P (Peer-2-Peer): A distributed systems and a 5.2 Expanding Ring communications paradigm 5.3 Random Walk www.httc.de www.httc.de 5.4 Bloom filters 6 Mechanisms for Structured P2P Networks 6.1 DHT Distributed Hash Tables Distributed systems definition (very general) 6.2 DHT: Usage "A distributed system is www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de 6.3 Chord, a DHT Example 6.4 Pastry, a prefix-based DHT a collection of individual computing devices 6.5 Tapestry, a suffix-based DHT 6.6 Kademlia that can communicate with each other" 6.7 Content Addressable Network (CAN) 6.8 Semantics-based Search Techniques 6.9 Topology: a Summary Lecture focus: 7 Case Study: Omicron - a Hybrid Overlay Design • Systems with loosely-coupled, autonomous devices 7.1 Design Mechanisms: Overlay Structure 7.2 De Bruijn Networks • Devices have their own semi-independent agenda 7.3 Design Mechanisms: Clusters 7.4 Design Mechanisms: Roles • (At least) limited coordination and cooperation needed 8 Accounting for P2P Networks 8.1 Introduction and Overview 8.2 KOM Token-based Accounting System 9 GRID Computing 10 Research: Some Major Issues in P2P Networking 11 Annex: Some References 3 4 1.1 Motivation Motivation (2) www.httc.de www.httc.de www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de freenet One of the newest buzzwords in networking is Peer-to-Peer (P2P) Is it only a hype? • initially 40 million Napster users in 2 years • integrated into commercial systems, e.g., Microsoft P2P SDK • Advanced Networking Pack for Windows XP, http://www.microsoft.com/windowsxp/p2p • open source, e.g., JXTA (Sun) with Protocols & Services • strong presence at international networking conferences 5 6 Above logos copied from the respective web page Motivation (3) Dominant P2P Applications P2P traffic is the major traffic source, since at least 2003 Sandvine Study 2003 Overall Internet traffic is more than ~50% P2P traffic • in Europe (France, Germany, ..) www.httc.de www.httc.de • predominant EDonkey/EMule e.g. France Telecom • in USA • see N.B. Azzouna & F. Guillemin • predominant KaZaA/Fastrack www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de Analysis of ADSL traffic on an IP backbone link IEEE Globecom 2003 Today, • Bittorrent seems to be the most successful P2P • Results application HTTP: 14.6 % Edonkey: 37.5 % • KaZaA more and more irrelevant FTP: 2.1 % KaZaA: 7.8 % • eDonkey largely replaced by eMule (using an NNTP: 1.9 % Napster: 3.8 % extended but compatible protocol) Other: 31.8 % Gnutella: 0.3 % Sum P2P: 49.6% + large part of “Other” 7 8 1.2 Evolution of Internet Computing Paradigms Evolution of Internet Computing Paradigms (2) 1st generation (since the beginning of the Internet): 3rd generation (since 2000): • permanent IP addresses, always connected • more collaboration and personalized applications www.httc.de www.httc.de • static domain name system (DNS) mapping • powerful edge devices (peers), instant networking • limited specialized applications, protocols: Telnet, • protocols/applications: FTP, Gopher, .... • Napster, Gnutella www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de Ö World Wide Access • Emule/Edonkey/MLDonkey, Fasttrack (KaZaA), Freenet, .. •Chord, … 2nd generation (since 90ties): Ö World Wide Peering P P • WWW & graphical browsers • dynamic IP addresses / NAT / firewalls • heterogeneous applications, asymmetric server based services P P P • protocol: HTTP, .. Ö World Wide Web P P P 9 10 1.3 Success of P2P Networking Success of P2P Networking (2) Some reasons for the success of P2P applications: New services at the edge of the network • P2P overlay networks make it relatively easy to deploy new Filesharing: highly attractive and cheap content services www.httc.de www.httc.de • users share their content with other users • Ö attractive content Group collaboration superior for business processes • copyrights are usually not respected (problem!) • grow organically, non-uniform and highly dynamic www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de • Ö cheap content • largely manual, ad-hoc, iterative and document-intensive work • often distributed, not centralized Unused resources at the edges • no single person/organisation understands the entire process • assume e.g. a SME enterprise with 100 desktop computers: from beginning to end • storage space: 100 x 150 GB = 15 TB spare storage space • processing power: 100 x 2,5 GHz x 5 ops/cycle = 1,25 trillion Cost effectiveness ops/sec spare processing power • reduces centralized management resources • optimizes computing, storage and communication resources Publishing: exploding amount of data • rapid deployment • 2 x 10e+18 Bytes are produced per year • 3 x 10e+12 Bytes are published per year P2P applications/protocols tailored for user’s needs • search engines like Google only index 1.3x10e+8 websites • Napster’s success depended to a great amount on its ease of • see Gong: JXTA: A Network Programming Environment, use 11 IEEE Computing 2001 12 1.4 P2P Application & Service Domains P2P Application & Service Domains (2) File Sharing: music, video and other data Distributed Computing - GRID • Napster, Gnutella, FastTrack (KaZaA, ...), eDonkey, eMule, • P2P CPU cycle sharing www.httc.de www.httc.de BitTorrent, eXeem, etc. • GRID Computing, ..., distributed simulation • SETI@home: search for extraterrestrial intelligence Distributed Storage/Distributed Filesharing • Popular Power: former battle to the influenza virus www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de • (Anonymous) Publication • Freenet, PAST, OceanStore, etc. Security and Reliability • Resilient Overlay Network (RON) Collaboration • Secure Overlay Services (SOS) • P2P groupware • Groove Application Layer Multicast • P2P content generation, • Narada • Online Games • P2P instant messaging 13 14 2 Specification of Peer-to-Peer 2.1 An Early Definition of P2P www.httc.de www.httc.de www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de Definition of P2P Networking (C. Shirkey): • "Peer-to-peer (P2P) is a class of applications that takes advantage of resources - storage, cycles, human presence - available at the edges of the Internet. Because accessing these decentralized resources means operating in an environment of unstable connectivity and unpredictable IP addresses, peer- to peer nodes must operate outside the DNS and have significant or total autonomy from central servers” 15 16 An Early Definition of P2P (2) 2.2 Nine Characteristics of “Pure” P2P Systems www.httc.de www.httc.de www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de Litmus test for a P2P application: 1.does it treat variable connectivity as the norm? • e.g. does it support dial-up users with variable IP addresses? 2.does it give the nodes at the edges of the network significant autonomy? • e.g. is storage / processing done by autonomous end-systems ÖIf answer to both is yes then the application is P2P otherwise not. (see Andy Oram: Peer-To-Peer / Harnessing the Power of Disruptive 17 Technologies, O’Reilly 2001) 18 Detailed Characteristics (1) Detailed Characteristics (2) www.httc.de www.httc.de www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de Resources (location, sharing) Networking 1. relevant resources located at nodes (peers) at the edges of a 4. variable connectivity is the norm network • support of dial-up users with variable IP addresses • operating outside the domain name system (DNS) 2. peers share their resources • often operating behind firewalls or NAT gateways 3. resource locations • widely distributed 19 • most often largely replicated 20 Detailed Characteristics (3) Detailed Characteristics (4) www.httc.de www.httc.de www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de Interaction of Peers Management 5. combined client and server functionality 7. peers have significant automony and mostly • “SERVer + cliENT = SERVENT” similar rights • minimal demands of the underlying infrastructure • services provided by end systems 8. no central control or centralized usage/provisioning of a service 6. direct interaction (provision of services, e.g. file transfer) between 9. self-organizing system 21 peers (= “peer to peer”) 22 Peer-to-Peer: 9 Properties 2.3 P2P Networks are Overlay Networks 1. relevant resources located at nodes (“peers”) at the edges of a network P P Service B Service A Service C www.httc.de www.httc.de 2. peers share their resources 3. resource locations P P P • widely distributed www.kom.tu-darmstadt.de www.kom.tu-darmstadt.de P P P • most often largely replicated Peers identified by PeerID 4. variable connectivity is the norm Overlay Network 5. combined Client and Server functionality 6.