A BitTorrent Performance Evaluation Framework Razvan˘ Deaconescu Razvan˘ Rughinis¸ Nicolae T¸ apus¸˘ University Politehnica of Bucharest Computer Science Department Splaiul Independent¸ei nr. 313, Bucharest, Romania frazvan.deaconescu, razvan.rughinis, [email protected] Abstract to be the killer application of the recent ears, by dominating the P2P traffic in the Internet [13]. In the last decade, file sharing systems have generally Most of the P2P and HTTP traffic in the Internet is mul- been dominated by P2P solutions. Whereas e-mail and timedia traffic. Current research and development are fo- HTTP have been the killer apps of the earlier Internet, a cused on expanding BitTorrent solution to video on demand large percentage of the current Internet backbone traffic is [8] and multimedia broadcasting and on creating social net- BitTorrent traffic [12]. BitTorrent has proven to be the per- works [2]. fect file sharing solution for a decentralized Internet, mov- As most of the current Internet traffic is video, BitTor- ing the burden from central servers to each individual sta- rent implementations focus on providing a media player in- tion and maximizing network performance by enabling un- terface, integrating video download and video rendering. used communication paths between clients. The BitTorrent protocol is also starting to integrate video Although there have been extensive studies regarding the streaming capabilities [14] [7]. performance of the BitTorrent protocol and the impact of As a decentralized system, a BitTorrent network is very network and human factors on the overall transfer quality, dynamic and download performance is influenced by many there has been little interest in evaluating, comparing and factors: swarm size, number of peers, network topology, analyzing current real world implementations. With hun- ratio enforcement. The innate design of the BitTorrent pro- dreds of BitTorrent clients, each applying different algo- tocol implies that each client may get a higher download rithms and performance optimization techniques, we con- speed by unchoking a certain client. At the same time, fire- sider evaluating and comparing various implementations walls and NAT have continuously been a problem for mod- an important issue. ern P2P systems and decrease the overall performance. In this paper, we present a BitTorrent performance eval- Despite implementing the BitTorrent specification [1] uation framework that we are using to test and compare and possible extensions each client uses different algo- current real world BitTorrent implementations. The frame- rithms and behaves differently on a given situation: it may work is fully automated and clients have been instrumented limit the number of peers, it may use heuristic information to output transfer status data and extensive logging infor- for an optimistic unchoke, it could choose a better client to mation. download from. Our paper presents a BitTorrent performance evaluation framework [9] that enables testing various BitTorrent imple- mentations and offering extensive status information about 1 Introduction each client. This information can be used for analysis, in- terpretation and correlation between different implementa- P2P sharing systems are continuously developing and in- tions and for the impact of a swarms state on the download creasing in size. There is a large diversity of solutions and performance. protocols for sharing data and knowledge which enable an Our BitTorrent performance evaluation framework is increasing interest from common users and commercial and fully automated. All clients have been instrumented to academic institutions [11]. use command line interfaces that enable automated actions. It is assumed [12] that BitTorrent is responsible for a Clients are started simultaneously and results are collected large portion of all Internet traffic. BitTorrent has proven after the file transfer is complete. 1 2 BitTorrent Clients For our experiments we have selected the BitTorrent clients that are most significant nowadays, based on the number of users, reported performance and features and his- tory. We have used Azureus, Tribler, Transmission, Aria, libtorrent rasterbar/hrktorrent, BitTornado and the mainline client (open source version). All clients are open source as we had to instrument them to use a command line interface and to output verbose logging information. Azureus, now called Vuze, is a popular BitTorrent client written in Java. We have been using version 2.3.0.6 of Azureus. The main issue with Azureus was the lack of a Figure 1. Infrastructure proper CLI that would enable automation. Though limited, a Console UI module enabled automating the tasks of run- ning Azureus and gathering download status and logging The reason for choosing to test it was because of a common information. background with Tribler. However, as testing revealed, it Tribler is a BitTorrent client written in Java and one of had its share of bugs and problems and it was eventually the most successful academic research project. Developed dropped. by a team in TU Delft, Tribler aims at adding various fea- BitTorrent Mainline is the original BitTorrent client tures to increase download speed and user experience. We written by Bram Cohen in Python. We have used version have used Tribler 4.2. Although a GUI oriented client, Tri- 5.2 during our experiments, the last open-source version. bler does offer a command line interface for automation. The mainline client provides a CLI and logging can be en- Extensive logging information is enabled by updating the abled through minor modifications of the source code. value of a few variables. Transmission is the default BitTorrent client in the pop- ular Ubuntu Linux distribution. Transmission is written in 3 Framework C and aims at delivering a good amount of features while still keeping a small memory footprint. The version we used for our tests was transmission 1.22. Transmission has As shown in Figure 1, the infrastructure on top of which a fully featured CLI and was one of the clients that were our framework runs consists of a command station, a repos- very easy to automate. Detailed debugging information re- itory and a set of client stations. garding connections and chunk transfers can be enabled by Communication between the CS and the BTS is handled setting the TR DEBUG FD evironment variable. over SSH for easy access and commanding. The repository Aria2 is a multiprotocol (HTTP, FTP, BitTorrent, Met- is used for storing the framework implementation and .tor- alink) download client. Throughout our tests we have been rent files for download sessions. Each BTS checks out the using version 0.14. aria2 natively provides a CLI and it was most recent framework version and the .torrent files to be easy to automate. Logging is also enabled through CLI ar- used. guments. Aria2 is written in C++ The framework is actually a set of small shell scripts en- libtorrent rasterbar/hrktorrent is a BitTorrent library abling the communication between the CS and the BTS and written in C++. It is used by a number of BitTorrent clients for running each BitTorrent client through its command line such as Deluge, qBittorent and SharkTorrent. As we were interface. looking for a client with a CLI we found hrktorrent to be For each download session the commander specifies the the best choice. hrktorrent is a lightweight implementa- .torrent file to be used and the mapping between each Bit- tion over libtorrent and provides the necessary interface for Torrent client and a BTS. Through SSH it runs a specific automating a BitTorrent transfer, albeit some modifications script on the target BTS enabling a BitTorrent client. The have been necessary. Libtorrent provides extensive logging BTS need not be in the same network. The only require- information by defining the TORRENT LOGGING and ment for the client stations is to run an SSH server and be TORRENT VERBOSE LOGGING MACROS. We have accessible through SSH. used version 0.13.1 of libtorrent and the most recent ver- Each BitTorrent client uses a specialized environment for sion of hrktorrent. storing logging and download status information. All these BitTornado is an old BitTorrent client written in Python. data can then be collected and analyzed. Figure 2. Test Swarm 1 Figure 3. Test Swarm 2 3.1 Repository access Table 1. Test Swarms Results Client Test1 Test2 Test3 Test 4 This project has been developed as part of the FP7 P2P- file size 908MB 4.1GB 1.09GB 1.09GB Next project. The project is open source. Anyone can acces seeders 2900 761 521 496 its repository [9] or use the web interface [10] to browse the leechers 2700 117 49 51 source. aria2c 1h17m 53m53s 8m 10m23s azureus 32m41s 38m33s N/A 7m bittorrent 4h53m 60m39s 26m 14m 4 Experimental setup libtorrent 9m41s 15m13s 2m30s 2m14s transmission 40m46s 53m 7m 5m Our experimental setup consisted of six identical com- tribler 34m 21m N/A N/A puters running the same operating system and sowftware. The hardware configuration includes CPU Pentium 4 2GHz, RAM 1 GB, HDD 160 GB. We have used Ubuntu 7.10 swarm. Although many data were collected, only the total Gutsy Gibbon. download time is featured in the table. The six computers are located in the same network and The conclusions drawn after result analysis were: are, thus using a common bandwidth to access the Internet. The computers have been firewalled from each other such • hrk/libttorrent is continuously surpassing the other that communication is enabled only with peers from the In- clients in different scenarios; ternet. Most of our experiments were simultaneous download • tribler, azureus and transmission are quite good clients sessions. Each system ran a specific client in the same time but lag behind hrktorrent; and conditions as the other clients. Results and logging data were collected after each client finished its download. • mainline and aria are very slow clients and should be dropped from further tests; 5 Results • swarms that are using sharing ratio enforcement of- fer better performance; a file is downloaded at least Our framework has been used to test different swarm 4 times faster within a swarm using sharing ratio en- (different .torrent files).