P2P Content Distribution BitTorrent and Spotify
Amir H. Payberah [email protected]
Amirkabir University of Technology (Tehran Polytechnic)
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 1 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 2 / 62 Possible Solutions for Content Distribution
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 3 / 62 Client-Server Model
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 4 / 62 Client-Server Model
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 5 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 6 / 62 I Single Point of failure?
The Client-Server Model Problems
I Scalability?
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 7 / 62 The Client-Server Model Problems
I Scalability?
I Single Point of failure?
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 7 / 62 The Client-Server Model Problems
I Scalability?
I Single Point of failure?
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 7 / 62 Client-Server Systems
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 8 / 62 The Client-Server Model Problem
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 9 / 62 Scalable and Fault-Tolerant Client-Server Model
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 10 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 11 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 12 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 13 / 62 Peer-to-Peer Model
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 14 / 62 Peer-to-Peer (P2P) Model
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 15 / 62 P2P Challenges
I Churn in the system
I Free-riding problem
I Bottleneck in the overlay network
I Connectivity problem, e.g., NAT
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 16 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 17 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 18 / 62 How To Discover Data?
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 19 / 62 Possible Solutions - First Generation
I Central directory
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 20 / 62 Possible Solutions - Second Generation
I Flooding
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 21 / 62 Possible Solutions - Third Generation
I Distributed Hash Table (DHT)
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 22 / 62 P2P Content Distribution Applications - File Sharing
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 23 / 62 P2P Content Distribution Applications - Media Streaming
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 24 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 25 / 62 BitTorrent
I BitTorrent is a system for efficient and scalable replication of large amounts of static data.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 26 / 62 BitTorrent Players
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 27 / 62 Files
I Files are broken into pieces of size between 64KB and 1MB.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 28 / 62 .torrent Files
I Metadata
I Contains: • URL of tracker • Information about the file, e.g., filename, length, ...
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 29 / 62 I It, then, connects to the tracker.
I The tracker tells the peers from which other peers to download the pieces of the file.
I Peers use this information to communicate with each other.
I The peers send information about the file and themselves to tracker.
The Core Idea
I A peer obtains .torrent file.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62 I The tracker tells the peers from which other peers to download the pieces of the file.
I Peers use this information to communicate with each other.
I The peers send information about the file and themselves to tracker.
The Core Idea
I A peer obtains .torrent file.
I It, then, connects to the tracker.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62 I Peers use this information to communicate with each other.
I The peers send information about the file and themselves to tracker.
The Core Idea
I A peer obtains .torrent file.
I It, then, connects to the tracker.
I The tracker tells the peers from which other peers to download the pieces of the file.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62 I The peers send information about the file and themselves to tracker.
The Core Idea
I A peer obtains .torrent file.
I It, then, connects to the tracker.
I The tracker tells the peers from which other peers to download the pieces of the file.
I Peers use this information to communicate with each other.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62 The Core Idea
I A peer obtains .torrent file.
I It, then, connects to the tracker.
I The tracker tells the peers from which other peers to download the pieces of the file.
I Peers use this information to communicate with each other.
I The peers send information about the file and themselves to tracker.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 30 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 31 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 32 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 33 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 34 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 35 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 36 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 37 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 38 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 39 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 40 / 62 What About Free Riders?
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 41 / 62 Tit-For-Tat
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 42 / 62 Question
I From which peers download the pieces?
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 43 / 62 Peer Selection
I Use choking algorithm to choose peer to download pieces.
I Decision to choke/unchoke based on tit-for-tat.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 44 / 62 Discover More Cooperating Peers
I Optimistic unchoking
I Allocate an upload slot to a randomly chosen uncooperative peer
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 45 / 62 Snubbed Peers
I If all its peers choke it.
I Increase the number of optimistic unchokes.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 46 / 62 Question
I Which piece?
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 47 / 62 I Random first piece: start-up need to get a complete piece
I Endgame mode: broadcast for all remaining blocks
Piece Selection
I Rarest first: common parts left for later
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 48 / 62 I Endgame mode: broadcast for all remaining blocks
Piece Selection
I Rarest first: common parts left for later
I Random first piece: start-up need to get a complete piece
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 48 / 62 Piece Selection
I Rarest first: common parts left for later
I Random first piece: start-up need to get a complete piece
I Endgame mode: broadcast for all remaining blocks
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 48 / 62 BitTorrent Extension
I Distributed tracker
I Peer-exchange
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 49 / 62 Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 50 / 62 Spotify
I Active users: over 50 million
I Number of songs: over 20 million
I Number of songs added per day: over 20000
I Number of playlists: over 1.5 billion created so far
I Available in 58 countries
I Legal
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 51 / 62 I Meanwhile, search P2P network for remainder.
I Switch back and forth between Spotify servers and peers as needed.
I Towards end of a track, start prefetching the next one.
The Core Idea
I Request first piece from Spotify servers.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 52 / 62 I Switch back and forth between Spotify servers and peers as needed.
I Towards end of a track, start prefetching the next one.
The Core Idea
I Request first piece from Spotify servers.
I Meanwhile, search P2P network for remainder.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 52 / 62 I Towards end of a track, start prefetching the next one.
The Core Idea
I Request first piece from Spotify servers.
I Meanwhile, search P2P network for remainder.
I Switch back and forth between Spotify servers and peers as needed.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 52 / 62 The Core Idea
I Request first piece from Spotify servers.
I Meanwhile, search P2P network for remainder.
I Switch back and forth between Spotify servers and peers as needed.
I Towards end of a track, start prefetching the next one.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 52 / 62 Main Problem in Using Spotify P2P Network
Peer Discovery
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 53 / 62 I Broadcast query in small (2 hops) neighborhood in overlay (Gnutella style)
I LAN peer discovery
Peer Discovery
I Sever-side tracker(BitTorrent style) • Only remembers 20 peers per track. • Returns 10 (online) peers to client on query.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 54 / 62 I LAN peer discovery
Peer Discovery
I Sever-side tracker(BitTorrent style) • Only remembers 20 peers per track. • Returns 10 (online) peers to client on query.
I Broadcast query in small (2 hops) neighborhood in overlay (Gnutella style)
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 54 / 62 Peer Discovery
I Sever-side tracker(BitTorrent style) • Only remembers 20 peers per track. • Returns 10 (online) peers to client on query.
I Broadcast query in small (2 hops) neighborhood in overlay (Gnutella style)
I LAN peer discovery
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 54 / 62 I If a peer is slow, re-request from new peers.
I When buffers are low, download from central server as well.
I If buffers are very low, stop uploading.
Downloading in P2P
I Ask for most urgent pieces first.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 55 / 62 I When buffers are low, download from central server as well.
I If buffers are very low, stop uploading.
Downloading in P2P
I Ask for most urgent pieces first.
I If a peer is slow, re-request from new peers.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 55 / 62 I If buffers are very low, stop uploading.
Downloading in P2P
I Ask for most urgent pieces first.
I If a peer is slow, re-request from new peers.
I When buffers are low, download from central server as well.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 55 / 62 Downloading in P2P
I Ask for most urgent pieces first.
I If a peer is slow, re-request from new peers.
I When buffers are low, download from central server as well.
I If buffers are very low, stop uploading.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 55 / 62 I Does not inform peers about downloaded blocks.
I Downloads blocks in order.
I Does not enforce fairness (such as tit-for-tat).
I Informs peers about urgency of request.
Spotify vs. BitTorrent
I One (well, three) P2P overlay for all tracks (not per-torrent).
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62 I Downloads blocks in order.
I Does not enforce fairness (such as tit-for-tat).
I Informs peers about urgency of request.
Spotify vs. BitTorrent
I One (well, three) P2P overlay for all tracks (not per-torrent).
I Does not inform peers about downloaded blocks.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62 I Does not enforce fairness (such as tit-for-tat).
I Informs peers about urgency of request.
Spotify vs. BitTorrent
I One (well, three) P2P overlay for all tracks (not per-torrent).
I Does not inform peers about downloaded blocks.
I Downloads blocks in order.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62 I Informs peers about urgency of request.
Spotify vs. BitTorrent
I One (well, three) P2P overlay for all tracks (not per-torrent).
I Does not inform peers about downloaded blocks.
I Downloads blocks in order.
I Does not enforce fairness (such as tit-for-tat).
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62 Spotify vs. BitTorrent
I One (well, three) P2P overlay for all tracks (not per-torrent).
I Does not inform peers about downloaded blocks.
I Downloads blocks in order.
I Does not enforce fairness (such as tit-for-tat).
I Informs peers about urgency of request.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 56 / 62 Caching
I Player caches tracks it has played.
I Use 10% of free space (capped at 10GB)
I Least Recently Used policy for cache eviction.
I Over 50% of data comes from local cache.
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 57 / 62 Spotify Data Usage
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 58 / 62 Spotify Says Goodbye to P2P
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 59 / 62 Summary
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 60 / 62 Summary
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 61 / 62 Questions?
Amir H. Payberah (Tehran Polytechnic) P2P Content Distribution 1393/8/27 62 / 62