Torrent Crawler: a tool for collecting information from BitTorrent networks Yeounoh Chung 1. Abstract structure and the unpredictability of the clients can cause problems in the network, making errors BitTorrent is a free peer-to-peer (P2P) difficult to detect and diagnosis. The reliance of a content-sharing application with a complex and node on node local views to maintain the overlay dynamic overlay structure due to loose coupling, high structure further compounds the problem. Defects or churn rate, and varying responsiveness of nodes. The anomalies such as partitioning in the overlay or load complexity and the dynamic nature of the overlay imbalance due to biased peer selection are structure can mask the problems in the network, undetectable without partial global information and a making errors difficult to detect and diagnosis in a good understanding of the various characteristics and timely manner. Furthermore, the heavy reliance of dynamic behaviors of the network. However, such an clients on the node local views compounds the understanding was often times missing or incorrectly problems such as partitioning in the network or load obtained from non-representative measurement imbalance due to biased peer selection. studies, in which a few instrumented clients were In an effort to provide the network with used to capture the desired properties. partial global information to resolve the network Collecting representative global information problems, this project looks into introducing a tool such as peer’s download rate, known neighboring that efficiently collects global information from peers, and the network’s churn rate, from the BitTorrent network. The tool, called Torrent Crawler complex and dynamic BitTorrent network is not (TC) uses a number of techniques to efficiently find simple. One way is to run a customized tracker to all participating peers of the swarm, collecting global track the distribution of particular content by information from the network. The crawler also publishing a torrent file. BitTorrent tracker will work collects the information unobtrusively to the network a centralized coordinator for the torrent file, making traffic. In this paper, we describe the design, the every downloading client to periodically contact the implementation, and the evaluation of TC. tracker and sending the list of peers to each client for its download. However, using a customized tracker to 2. Introduction collect global information has some drawbacks [1]: First, each peer updates tracker of its download and BitTorrent[8] is a free P2P content-sharing upload information once every 30 minutes. This application that enables a peer to distribute its content implies that variations on measurement in shorter to a large number of peers without having a large time scale are not available. Second, the tracker does upload bandwidth. A peer can utilize the aggregated not know accurate information about connectivity of upload bandwidth of the torrent swarm by duplicating individual peers. The tracker update message from a its content over several peers; a peer, who wishes to client does not contain the client’s connectivity download the content, downloads different parts of information; a client could use a gossip like protocol the content from different peers simultaneously. To to exchange its known peer list with other clients. download a file, a peer will first get a torrent file of Third, the tracker log does not provide any the content published by a seeder, a peer with the information about peers’ maximum achievable entire file. The torrent file contains the URL of the download and upload rates. centralized coordinator, called tracker, of the torrent Another approach to collecting information network, and the file’s metadata information hash. from the torrent network is to use several The downloading peer then contacts the tracker to instrumented clients to capture their observed discover the peers to download the file from. performances [4][5]. An instrumented client is a BitTorrent has gained a lot of popularity in normal BitTorrent client that can perform recent years. The loose coupling of nodes and the measurement on its observed network status and structural redundancy to handle high churn rates and received message statistics. But, this approach does varying node responsiveness of BitTorrent network not provide a representative view of all the results in a much complicated behavior of the participating peers or group information such as the network. The complexity of the overlay network number of seeders in the network, the availability of information is stored in S3 as public accessible pieces [1]. objects, clients can use web browser to view the Instead of the above two approaches, TC information. uses the network crawling based approach to collect In this survey, we present the design, the global information, interactively communicating with implementation, and the evaluation of Torrent both the tracker and the peers of any given torrent. In Crawler, a tool to efficiently collect global order to speed up the crawling, first, TC requests the information. By interactively communicating with tracker of more peers frequently than normal both the tracker and the participating peers of the BitTorrent clients. Second, TC uses BitTorrent Peer given torrent network, TC can provide a Exchange Protocol (PEX) to discover peers known to representative view of the swarm promptly. TC relies others from the other peers. And finally, TC on BitTorrent messages sent by other peers to collect advertises itself as a seeder, a peer with whole information, instead of exchanging any actual content, to the tracker and peers to have undiscovered contents to observe network performance, making the peers to connect to TC. Using TC to collect global measurement unobtrusive to the network traffic. information on all the participating peers and the Finally, the collected information is stored in torrent network, we plan on collecting a Amazon S3 for reliable accesses from any number of representative view of all the participating peers in clients, and to simplify data storing and organization. about 8 minutes. Although, BitTorrent relies on client 3. Related Work altruism for file sharing, because TC advertises itself as a seeder and does not upload any contents, other To gain a good understanding of various clients might punish TC for such behavior [9][10]. characteristics and dynamic behaviors of BitTorrent Fortunately, the punishment takes a form of reducing network, a lot of studies have been conducted from the bandwidth allowed to TC for downloading from statistical modeling to simulation, and measurement. the client, which is irrelevant to TC’s operation. And to gain the good understanding that is also BitTorrent community also maintains a list of IP representative of the entire network, people have ranges to block peers from unwanted organizations. tried using a customized BitTorrent tracker for a This IP blacklisting is to prevent certain given torrent file to monitor global information. organizations from accessing the network and is not However, BitTorrent tracker can only passively an issue for TC. collect information from each peer once every 30 Another interesting aspect of BitTorrent minutes. However, this approach has limitations in studies is BitTorrent’s contribution to network traffic. measurement time scale and efficiency. Many In this work, we make the measurement unobtrusive dynamics and variations happen in much shorter time to the network. To achieve this goal, TC never scale will be lost. More importantly, the approach actually downloads, requests, or uploads files to cannot detect the individual peer connectivity to observe network performance; it only uses the detect network topology, biased peer selection, and BitTorrent messages sent by other peers and the load imbalance [1]. Hence, it is desirable to have a tracker to measure performance and piece availability. more efficient and reliable tool for collecting global TC only sends out a single handshake message to information of the network. each peer and the tracker at the beginning of the A more common approach to study connection (or reconnection). Although, TC contacts BitTorrent network is to use instrumented clients to the tracker more often than it is advised to get the observe their performances in the torrent swarm global population, it would not burden the entire [4][5]. While instrumented clients provide accurate network much. information and measurement of the network In addition, Torrent Crawler uses Amazon performance from the clients’ standpoints, the Simple Storage Service (S3) to store the collected collected data often fails to represent the entire information. Amazon S3 is highly scalable, reliable, population of the torrent swarm [1]. and available distributed storage system with a In studying the effect of the torrent attacks simple and extensive API library. Amazon S3 initiated by media corporate and movie production provides a simple, reliable solution for any number of studio, a group of people have used a crawling based clients to view the stored results. Furthermore, Using technique to discover all the participating peers of a Amazon S3 to store and to publish the collected given torrent file [2]. They claim that their crawler is information separates this data storing and able to discover most of the participating peers (i.e. organization layer from the data collection layer of over 90% of the entire population) just under 8 the system,