P2P Systems
IP networks course Giovanni Neglia
Outline
¢ P2P systems overview
¢ The ancestor: Napster
¢ Only file sharing?
¢ distributed computing, collaborative environment, anonimity...
¢ Why P2P?
¢ File sharing, file sharing, file sharing...
¢ witha note on structuredP2P networks and DHT
1 What is Peer-to-Peer (P2P)?
The Client-Server Model
¢ Contact a server and get the service. ¢ Server has all the resources and capabilities. ¢ No interaction among clients ¢ Common model in the Internet (e.g. www).
2 What is a peer?
¢ —an entity with capabilities similar to other entities in the system“
The P2P Model
¢ A peer‘s resources are similar to the resources of the other participants
¢ P2P œ peers communicating directly with other peers and sharing resources
¢ Peer = Servent = Server+Client
3 P2P Application Taxonomy
¢ Proposed for
¢ file sharing (Napster, Gnutella, Kazaa, BitTorrent)
¢ Distributed Computing (SETI@home)
¢ Collaboration (Jabber, Groove)
¢ generalpurpose platforms (JXTA)
¢ audio/videoconference -> Skype
¢ chensorshipresistance -> Infranet, Tangler
P2P File Sharing Software
¢ Allows a user to open up a directory in their file system
¢ Anyone can retrieve a file from directory
¢ Like a Web server ¢ Allows the user to copy files from other users‘ open directories:
¢ Like a Web client ¢ Allows users to search nodes for content based on keyword matches:
¢ Like Google
4 Outline
¢ P2P systems overview
¢ The ancestor: Napster
¢ Only file sharing?
¢ distributed computing, collaborative environment, anonimity...
¢ Why P2P?
¢ File sharing, file sharing, file sharing...
Napster: How Did It Work
¢ Application-level, client-server protocol over point-to-point TCP ¢ Centralized directory server ¢ Steps:
¢ Connect to Napster server
¢ Give server keywords to search the full list with.
¢ Select —best“ of correct answers.
¢ One approach is selecting based on the response time of pings.
¢ Shortest response time is chosen.
5 Napster: How Did It Work
napster.com centralized directory 1. File list and IP address is uploaded
Napster: How Did It Work
napster.com 2.U ser centralized directory requests search at server. Q uery and results
6 Napster: How Did It Work
napster.com 3. U ser pings centralized directory hosts that apparently have data.
Looks for pings best transfer pings rate.
Napster: How Did It Work
napster.com 4. U ser chooses centralized directory server
N apster’s centralized server farm had difficult tim e Retrieves keeping file up w ith traffic
7 Napster History
¢ 5/99: Shawn Fanning (freshman, NortheastenU.) founds Napster Online music service ¢ 12/99: first lawsuit ¢ 3/00: 25% UWisctraffic Napster ¢ 2/01: US Circuit Court of Appeals: Napster knew users violating copyright laws
Napster
8M
¢ Judge orders Napster to
6c M
pull plug in July ”01 e s
r
¢ Other file sharing apps e
4p M
take over! s t i b ¢ 7/01: # simultaneous 2M online users:
¢ Napster 160K, 0.0 ¢ Gnutella: 40K,
¢ Morpheus (KaZaA): 300K gnutella napster fastrack (KaZ aA )
8 Napster: Discussion
¢ Locates files quickly
¢ Vulnerable to censorship and technical failure
¢ Popular data become less accessible because of the load of the requests on a central server
Outline
¢ P2P systems overview
¢ The ancestor: Napster
¢ Only file sharing?
¢ distributed computing, collaborative environment, anonimity...
¢ Why P2P?
¢ File sharing, file sharing, file sharing...
9 SETI@home
¢ Search for Extraterrestrial Intelligence ¢ Search through massive amounts of radio telescope data to look for signals ¢ Build huge virtual computer by using idle cycles on Internet computer ¢ Conceived 1995, launched April 1999
SETI@HOME
¢ Distributes a screen saverœbased application to users
¢ Applies signal analysis algorithms different data sets to process radio-telescope data. 1.Install 3.S ET I client gets S creen S erver data from server and runs S ET I@ H om e M ain S erver 4.Client sends results back to server Radio-telescope Data
2.S ET I client (screen S aver) starts
10 Why don‘t we build a huge supercomputer?
¢ Supercomputers time is expensive
¢ Home PCs are commonly idle
Why don‘t we build a huge supercomputer?
¢ Top500 supercomputer list over time: k ¢ Zipfdistribution: Perf(rank) ≈ rank -0.68 Parameter 'k' evolution . 1. 0000
-0.70 ) .
e 2001 2000
l
) -0.72 a e c
l 1999 1998 s
a -0.74 2001 c
g 1000 1997 1996 s
-0.76 o l g ( 1995 -0.78 o l S (
P -0.80 S O P 100 -0.82 L O F -0.84 L G F . 3 2 1 0 9 8 7 6 5 f 0 0 0 0 9 9 9 9 9 G r 0 0 0 0 9 9 9 9 9
e 2 2 2 2 1 1 1 1 1 k p c 10 a k P c a n i P L n i
L 1 1 10 100 1000 Rank (log scale)
11 -0.68 Parameter 'k' evolution . -0.70 -0.72 -0.74 -0.76 -0.78 -0.80 -0.82 Trend -0.84 3 2 1 0 9 8 7 6 s 5 0 0 0 0 9 9 9 9 9 0 0 0 0 9 9 9 9 9 2 2 2 2 1 1 1 1 1
¢ Increasingly interest to aggregate the capabilities of the machines in the tail of this distribution.
¢ A virtual machine that aggregates the last 10 in Top500 would rank 32ndin ‘95 but 14thin ”03
¢ Grid computing is, in part, results of this trend: infrastructure enabling controlled, secure resource sharing (for a relatively small number of resources)
Lessons from SETI@home
¢ Can apply this technology to real problems ¢ Expected 100k participants, but have 3 million, 0.5M active, 226 countries ¢ 40 TB data recorded, processed ¢ 25 TeraFLOPs average over 2005 ¢ Almost 1 million years CPU time ¢ No ET signals yet, but other results ¢ Followers: ¢ Genome@home, Folding@home, CancerResearch Iniziative
12 Groove P2P Work environment
¢ Ideal fordynamicteam work
Integration Secure, faster & with existing cost effective system Functional online Extension interaction
Edit and Upload User Download/Update /Store contents components
Developer
Update/Upload contents Download components
Interact / Negotiate Share file / Coordinate events Centralized IS control for IT Sh are d Virtu al Space Better real-time Manager interaction Upload contents Network & Web Service Flexible and integral Platform Download contents P rovide Software/ Application T ech Partne r/Su pplie r/Family GROO VE
Outline
¢ P2P systems overview
¢ The ancestor: Napster
¢ Only file sharing?
¢ distributed computing, collaborative environment, anonimity...
¢ Why P2P?
¢ File sharing, file sharing, file sharing...
13 Why P2P?
¢ Distributed systems pros...
¢ Scalability, Reliability, Saving,... ¢ and cons
¢ complexity, management, security ¢ The Internet has three valuable fundamental assets...
¢ Information
¢ Computing resources
¢ Bandwidth ¢ ...all of which are vastly under utilized, partly due to the traditional client-server model
Why P2P?
¢ No single search engine can locate and catalog the ever-increasing amount of information on the Web in a timely way ¢ Moreover, a huge amount of information is transient and not subject to capture by techniques such as Web crawling 8 ¢ Google claims that it searches about 1.3x10web pages
¢ Finding useful information in real time is increasingly difficult!
14 Why P2P?
¢ Although miles of new fiber have been installed, the new bandwidth gets little use if everyone goes to Yahoo for content and to eBay
¢ Instead, hot spots just get hotter while cold pipes remain cold
¢ This is partly why most people still feel the congestion over the Internet while a single fiber‘s bandwidth has increased by a factor of 106 since 1975, doubling every 16 months
Why P2P?
¢ P2P potentially can eliminating the single- source bottleneck
¢ P2P can be used to distribute data and control and load-balance requests across the Net
¢ P2P potentially eliminates the risk of a single point of failure
¢ P2P infrastructure allows direct access and shared space, and this can enable remote maintenance capability
15 P2P impact today (1)
¢ Widespread adoption
¢ KaZaAœ 360 million downloads (1.3M/week) one of the most popular applications ever!
¢ leading to (almost) zero-cost content distribution:
¢ is forcing companies to change their business models and might impact copyright laws FastTrack 2,460,120 eDonkey 1,987,097 Overnet 1,261,568 iMesh 803,420 Sources: www.slyck.com, Warez 440,289 www.kazaa.com, July ‘04 Gnutella 389,678
P2P impact today (2)
¢ P2P applications market (Solomon Smith Barney)
¢ 5.800 milliondollars in 2003
¢ 36.500 milliondollars in 2004
¢ Resources saving
¢ Seti@Home $1.5M / year in additional power consumed
16 P2P impact today (3)
¢ P2P traffic in the network
¢ from 20% up to80%!!
¢ Driving adoption of consumer broadband
¢ Need for control accordingtonetwork providers and administrators
Gnutella
¢ The focus is on a decentralized method of searching for files
¢ Central directory server no longer the bottleneck
¢ More difficult to —pull plug“
¢ Each application instance serves to:
¢ Store selected files
¢ Route queries from and to its neighboring peers
¢ Respond to queries if file stored locally
¢ Serve files
17 Gnutella
¢ Gnutella history:
¢ 3/14/00: release by AOL, almost immediately withdrawn
¢ Became open source
¢ Many iterations to fix poor initial design (poor design turned many people off) ¢ Issues:
¢ How much traffic does one query generate?
¢ How many hosts can it support at once?
¢ What is the latency associated with querying?
¢ Is there a bottleneck?
Gnutella: Searching
¢ Searching by flooding: ¢ A Query packet might ask, "Do you have any content that matches the string ”Homer"?Ifa node does not have the requested file, then 7 (default set by Gnutella) of its neighbors are queried. ¢ If the neighbors do not have it, they contact 7 of their neighbors. ¢ Maximum hop count: 10 (this is called time-to- live TTL) ¢ Reverse path forwarding for responses (not files)
18 Gnutella: Searching
Gnutella: Searching
19 Gnutella: Searching
¢ Downloading ¢ Peers respond with a —QueryHit“ (contains contact info) ¢ File transfers use direct connection using HTTP protocol‘s GET method ¢ When there is a firewall a "Push" packet is used œ reroutes via Push path
Gnutella: arrival of a new Peer
20 KaZaA: Architecture
¢ Each peer is either a supernode or is assigned to a supernode ¢ 56 min avgconnect supernodes ¢ Each SN has about 100- 150 children ¢ Roughly 30,000 SNs ¢Each supernode has TCP connections with 30-50 supernodes ¢ 23 min avgconnect
Free Rider Problem
21 Free Rider Problem
¢ No individual is willing to contribute towards the cost of something (public goods) when he hopes that someone else will bear the cost instead ¢ In P2P, many people just download files contributed by others and never share any of their files ¢ Few servers become the hot spots: Anonymous?, Copyright?, Privacy? Scalability?, Is it P2P?
How to build a reliable P2P system
¢ Require: Contribution should be predictable
¢ Peers can be motivated using economic principle
¢ Monetary payment (one pays to consume resources and paid to contribute resource)
¢ Differential service (peers that contributes more get better quality of service)
22 Real P2P systems
¢ Differential service (peers that contributes more get better quality of service)
¢ KaZaA
¢ participation level: upload/download
¢ BitTorrent
¢ Bartered —Tit for Tat“ download bandwidth
¢ Download one (random) chunk from a storage peer, slowly
¢ Subsequent chunks barteredwith concurrent downloaders
¢ The more chunks you can upload, the more you can download
¢ Download speed starts slow, then goes fast
BitTorrent: A community foreachcontent
t rren To ET t G rren .to File
Anno unce Peer addre sses (Pee r set)
23 BitTorrent: A community foreachcontent
t rren To ET t G rren .to File
Anno unce Ind irizzi Peer
Seeders and leechers
¢ seeders have the entire file
¢ leechers
24 Peer and Piece Selection
¢ At the core of any P2P protocol
¢ Peer Selection
¢ Maximize capacity of service
¢ Foster reciprocation and prevent free riders
¢ Choice of the peers to uploadto
¢ Efficiency criteria
Peer and Piece Selection
¢ Piece selection
¢ Which pieces to download from peers
¢ Should guarantee a high piece diversity
¢ Always find an interesting piece in any other peer
¢ Do not bias peer selection
25 Choke and Rarest First Algorithms
¢ Choke algorithm
¢ Leechers: upload to the peers (unchoke) from which we are downloading the fastest
¢ Reevaluate periodically (10s)
¢ Optimistic unchoke
¢ Reevaluate periodically (30s)
¢ 3 unchoke + 1 optimistic unchoke
¢ Seeds: as leechers but upload to the fastest downloaders
¢ different algorithms in new versions
Choke and Rarest First Algorithms
¢ Rarest first algorithm
¢ Choose the pieces that are locally rarest
¢ For short: rarest first
¢ Random first policy
26 Some Real Numbers
¢ Torrent characteristics
¢ Torrent size: from a few peers to 100 000 peers
¢ popular torrents: between 10 000 and 50 000 peers
¢ Content size: from a few kBto 4GB
¢ TV series: 300MB, Movie: 600MB, DVD image: 4GB
¢ Piece size: {256,512,1024} kB
¢ Typical case: 1000 pieces for a content
¢ Peer set size: 80 peers
Evolutionof BitTorrent: support forUDP protocol
¢ toreduce the loadon trackers
27 Evolutionof BitTorrent: Multiple trackers
¢ In .torrent a list of trackers
¢ case 1): {{Tracker1},{Tracker2}}
ce oun nn eer A zi P iriz Ind 1
Anno unce Ind irizzi Peer
2
Evolutionof BitTorrent: Multiple trackers
¢ In .torrent a list of trackers
¢ case 2): {{Tracker1,Tracker2}}
unce Anno r zi Pee Indiriz 1 1
Anno unce Ind irizzi Peer
2 2
28 Evolutionof BitTorrent: Multiple trackers
¢ In .torrent a list of trackers
¢ case 3): {{Tracker1,Tracker2},{Tracker3}}
Structured P2P
¢ Second generation P2P overlay networks
¢ Self-organizing ¢ Load balanced ¢ Fault-tolerant
¢ Scalable guarantees on numbers of hops to answer a query
¢ Major difference with unstructured P2P systems
¢ Based on a distributed hash table (DHT) interface
29 Distributed Hash Tables (DHT)
¢ Distributed version of a hash table data structure ¢ Stores (key, value) pairs
¢ The key is like a filename
¢ The value can be file contents ¢ Goal: Efficiently insert/lookup/delete (key, value) pairs ¢ Each peer stores a subset of (key, value) pairs in the system ¢ Core operation: Find node responsible for a key
¢ Map key to node
¢ Efficiently route insert/lookup/delete request to this node
DHT Desirable Properties
¢ Keys mapped evenly to all nodes in the network
¢ Each node maintains information about only a few other nodes
¢ Messages can be routed to a node efficiently
¢ Node arrival/departures only affect a few nodes
30 Structured Overlays
¢ Properties
¢ Topology is tightly controlled
¢ Well-defined rules determine to which other nodes a node connects
¢ Files/Information placed at precisely specified locations
¢ Hash function maps file names to nodes
¢ Scalable routing based on file attributes
LookupProblem in P2P Systems
How do you find any given data item in a large P2P system in a scalable manner, without any centralized servers or hierarchy?
Distributed Structured P2P Overlay Application
Structured P2P systems implement the API Interface: Value DHTabstraction Put(Key,Value) Remove(Key) Value=Get(Key)
Distributed Hash Table
Peer Peer Peer Peer
Figure 1: Application Interface for Structured DHT-based P2P Overlay Systems (Luaet al. 2004)
31 Distributed Hash Tables (DHTs)
¢ In DHT-based Systems:
¢ Peers are assigned uniform random NodeIDs
¢ Data objects are assigned unique identifiers called keys
¢ Keys are mappedby the overlay network protocol to unique peers
¢ Overlay networks support the scalable storage and retrieval of {key,value} pairs:
¢ Put(key,value)
¢ Remove(key)
¢ Value=Get(key)
¢ Each peer maintains a small routing table consisting of its neighbouringpeers‘ NodeIDs and IP addresses
¢ Lookup queries are forwardedacross overlay paths to peers, with the NodeIDs that are closerto the key
DHT Routing Protocols
¢ DHT is a generic interface
¢ Different organization schemes for the data objects and its key space and routing strategies ¢ There are several implementations of this interface
¢ Chord [MIT]
¢ Pastry [Microsoft Research UK, Rice University]
¢ Tapestry [UC Berkeley]
¢ Content Addressable Network (CAN) [UC Berkeley]
¢ SkipNet [Microsoft Research US, Univ. of Washington]
¢ Kademlia [New York University]
¢ Viceroy [Israel, UC Berkeley]
¢ P-Grid [EPFL Switzerland]
¢ Freenet [Ian Clarke]
32 Content Addressable Network (CAN)
(0.5-0.75,0.5-1.0)
1.0 Keys are hashed into d-dimensional D • C (0.2,0.7) (0.6,0.7) coordinate space (0.75-1.0,0.5-1.0) (0.0-0.5,0.5-1.0) E (0.9,0.6)
• Every node owns its individual, distinct A (0.1,0.1) zone within the overall space B (0.6,0.3) 0.0 (0.5-1.0,0.0-0.5) • To store a pair (K,V): 0.0 1.0 node B’s virtual coordinate zone (0.0-0.5,0.0-0.5) ° K is mapped onto a point P in the coordinate space using a uniform hash function
° The (key,value) pair is stored at the node that owns the zone
Routing in a CAN
• A CAN node maintains a routing table that 6 2 holds information about its neighbour zones
3 1 5 A node routes a message towards its destination 1 • (d / 4)(n d ) by greedy forwarding to the neighbour closest to the destination 4 • For a d dimensional space partitioned into n equal zones: (x,y) 1 ° the average routing path length is (d / 4)(n d ) 1’s coordinate neighbour set = {2,3,4,5} ° individual nodes maintain 2d neighbours
Sample routing path from node 1 to point (x,y)
33 CAN: Node Join and Departure
P: (x,y)
• To join a CAN, a node: ° Retrieves a bootstrap node’s IP address ° Gets the IP addresses of several alive nodes and chooses one ° Randomly chooses a point P in the space ° A message is routed to the node in whose zone P lies ° This node splits its zone in half and assigns one half to the new node Bootstrap Node New Node ° Both new and old nodes’ neighbours are informed of the reallocation of space • A node leaves a CAN: ° explicitly hands over its zone and (key,value) pairs to one of its neighbours • A Node fails: ° handled through a takeover algorithm that ensures one of the failed node’s neighbours takes over the zone
Evolutionof BitTorrent: integrationof a KademliaDHT
Q uery for Content 1
T orrent 1 1 T orrent 2
34 DHT details
¢ Two different implementations (Azureus vs Mainline&others)
¢ About the DHT bootstrap nodes
¢ from the file .torrent
¢ embedded in the client
¢ from other peers
35