A BitTorrent Performance Evaluation Framework

Razvan˘ Deaconescu Razvan˘ Rughinis¸ Nicolae T¸ apus¸˘ University Politehnica of Bucharest Computer Science Department Splaiul Independent¸ei nr. 313, Bucharest, Romania {razvan.deaconescu, razvan.rughinis, nicolae.tapus}@cs.pub.ro

Abstract to be the killer application of the recent ears, by dominating the P2P traffic in the Internet [13]. In the last decade, file sharing systems have generally Most of the P2P and HTTP traffic in the Internet is mul- been dominated by P2P solutions. Whereas e-mail and timedia traffic. Current research and development are fo- HTTP have been the killer apps of the earlier Internet, a cused on expanding BitTorrent solution to video on demand large percentage of the current Internet backbone traffic is [8] and multimedia broadcasting and on creating social net- BitTorrent traffic [12]. BitTorrent has proven to be the per- works [2]. fect file sharing solution for a decentralized Internet, mov- As most of the current Internet traffic is video, BitTor- ing the burden from central servers to each individual sta- rent implementations focus on providing a media player in- tion and maximizing network performance by enabling un- terface, integrating video download and video rendering. used communication paths between clients. The BitTorrent protocol is also starting to integrate video Although there have been extensive studies regarding the streaming capabilities [14] [7]. performance of the BitTorrent protocol and the impact of As a decentralized system, a BitTorrent network is very network and human factors on the overall transfer quality, dynamic and download performance is influenced by many there has been little interest in evaluating, comparing and factors: swarm size, number of peers, network topology, analyzing current real world implementations. With hun- ratio enforcement. The innate design of the BitTorrent pro- dreds of BitTorrent clients, each applying different algo- tocol implies that each client may get a higher download rithms and performance optimization techniques, we con- speed by unchoking a certain client. At the same time, fire- sider evaluating and comparing various implementations walls and NAT have continuously been a problem for mod- an important issue. ern P2P systems and decrease the overall performance. In this paper, we present a BitTorrent performance eval- Despite implementing the BitTorrent specification [1] uation framework that we are using to test and compare and possible extensions each client uses different algo- current real world BitTorrent implementations. The frame- rithms and behaves differently on a given situation: it may work is fully automated and clients have been instrumented limit the number of peers, it may use heuristic information to output transfer status data and extensive logging infor- for an optimistic unchoke, it could choose a better client to mation. download from. Our paper presents a BitTorrent performance evaluation framework [9] that enables testing various BitTorrent imple- mentations and offering extensive status information about 1 Introduction each client. This information can be used for analysis, in- terpretation and correlation between different implementa- P2P sharing systems are continuously developing and in- tions and for the impact of a swarms state on the download creasing in size. There is a large diversity of solutions and performance. protocols for sharing data and knowledge which enable an Our BitTorrent performance evaluation framework is increasing interest from common users and commercial and fully automated. All clients have been instrumented to academic institutions [11]. use command line interfaces that enable automated actions. It is assumed [12] that BitTorrent is responsible for a Clients are started simultaneously and results are collected large portion of all Internet traffic. BitTorrent has proven after the file transfer is complete.

1 2 BitTorrent Clients

For our experiments we have selected the BitTorrent clients that are most significant nowadays, based on the number of users, reported performance and features and his- tory. We have used Azureus, Tribler, Transmission, Aria, libtorrent rasterbar/hrktorrent, BitTornado and the mainline client (open source version). All clients are open source as we had to instrument them to use a command line interface and to output verbose logging information. Azureus, now called Vuze, is a popular BitTorrent client written in Java. We have been using version 2.3.0.6 of Azureus. The main issue with Azureus was the lack of a Figure 1. Infrastructure proper CLI that would enable automation. Though limited, a Console UI module enabled automating the tasks of run- ning Azureus and gathering download status and logging The reason for choosing to test it was because of a common information. background with Tribler. However, as testing revealed, it Tribler is a BitTorrent client written in Java and one of had its share of bugs and problems and it was eventually the most successful academic research project. Developed dropped. by a team in TU Delft, Tribler aims at adding various fea- BitTorrent Mainline is the original BitTorrent client tures to increase download speed and user experience. We written by Bram Cohen in Python. We have used version have used Tribler 4.2. Although a GUI oriented client, Tri- 5.2 during our experiments, the last open-source version. bler does offer a command line interface for automation. The mainline client provides a CLI and logging can be en- Extensive logging information is enabled by updating the abled through minor modifications of the source code. value of a few variables. Transmission is the default BitTorrent client in the pop- ular Ubuntu Linux distribution. Transmission is written in 3 Framework C and aims at delivering a good amount of features while still keeping a small memory footprint. The version we used for our tests was transmission 1.22. Transmission has As shown in Figure 1, the infrastructure on top of which a fully featured CLI and was one of the clients that were our framework runs consists of a command station, a repos- very easy to automate. Detailed debugging information re- itory and a set of client stations. garding connections and chunk transfers can be enabled by Communication between the CS and the BTS is handled setting the TR DEBUG FD evironment variable. over SSH for easy access and commanding. The repository Aria2 is a multiprotocol (HTTP, FTP, BitTorrent, Met- is used for storing the framework implementation and .tor- alink) download client. Throughout our tests we have been rent files for download sessions. Each BTS checks out the using version 0.14. aria2 natively provides a CLI and it was most recent framework version and the .torrent files to be easy to automate. Logging is also enabled through CLI ar- used. guments. Aria2 is written in C++ The framework is actually a set of small shell scripts en- libtorrent rasterbar/hrktorrent is a BitTorrent library abling the communication between the CS and the BTS and written in C++. It is used by a number of BitTorrent clients for running each BitTorrent client through its command line such as Deluge, qBittorent and SharkTorrent. As we were interface. looking for a client with a CLI we found hrktorrent to be For each download session the commander specifies the the best choice. hrktorrent is a lightweight implementa- .torrent file to be used and the mapping between each Bit- tion over libtorrent and provides the necessary interface for Torrent client and a BTS. Through SSH it runs a specific automating a BitTorrent transfer, albeit some modifications script on the target BTS enabling a BitTorrent client. The have been necessary. Libtorrent provides extensive logging BTS need not be in the same network. The only require- information by defining the TORRENT LOGGING and ment for the client stations is to run an SSH server and be TORRENT VERBOSE LOGGING MACROS. We have accessible through SSH. used version 0.13.1 of libtorrent and the most recent ver- Each BitTorrent client uses a specialized environment for sion of hrktorrent. storing logging and download status information. All these BitTornado is an old BitTorrent client written in Python. data can then be collected and analyzed. Figure 2. Test Swarm 1 Figure 3. Test Swarm 2

3.1 Repository access Table 1. Test Swarms Results Client Test1 Test2 Test3 Test 4 This project has been developed as part of the FP7 P2P- file size 908MB 4.1GB 1.09GB 1.09GB Next project. The project is open source. Anyone can acces seeders 2900 761 521 496 its repository [9] or use the web interface [10] to browse the leechers 2700 117 49 51 source. aria2c 1h17m 53m53s 8m 10m23s azureus 32m41s 38m33s N/A 7m bittorrent 4h53m 60m39s 26m 14m 4 Experimental setup libtorrent 9m41s 15m13s 2m30s 2m14s transmission 40m46s 53m 7m 5m Our experimental setup consisted of six identical com- tribler 34m 21m N/A N/A puters running the same operating system and sowftware. The hardware configuration includes CPU Pentium 4 2GHz, RAM 1 GB, HDD 160 GB. We have used Ubuntu 7.10 swarm. Although many data were collected, only the total Gutsy Gibbon. download time is featured in the table. The six computers are located in the same network and The conclusions drawn after result analysis were: are, thus using a common bandwidth to access the Internet. The computers have been firewalled from each other such • hrk/libttorrent is continuously surpassing the other that communication is enabled only with peers from the In- clients in different scenarios; ternet. Most of our experiments were simultaneous download • tribler, azureus and transmission are quite good clients sessions. Each system ran a specific client in the same time but lag behind hrktorrent; and conditions as the other clients. Results and logging data were collected after each client finished its download. • mainline and aria are very slow clients and should be dropped from further tests;

5 Results • swarms that are using sharing ratio enforcement of- fer better performance; a file is downloaded at least Our framework has been used to test different swarm 4 times faster within a swarm using sharing ratio en- (different .torrent files). Most scenarios involved simulta- forcement. neous downloads for all clients. At the end of each session download status information and extensive logging and de- 5.1 Integration with MonALISA bugging information were gathered from every client. Fig- ure 2 and figure 3 are comparisons between different BitTor- We extended our framework to use MonALISA for real rent clients running on the same environment in the same time monitoring of download speed and other factors. Each download scenario. client can be configured to send data to a MonALISA agent. Table 1 presents a comparison of the BitTorrent clients The MonALISA agent parses that data and creates a close to in four different scenarios. Each scenario means a different real time graphical evolution of the download speed. There The current SSH communication means that BTS can be commanded only if their SSH server can be contacted. This makes it very difficult for clients that are behind NAT or firewalls to participate in a testing scenario. We aim to de- velop and deploy a server that accepts incoming connection regardless of NAT/firewall constraints and uses these con- nections to command BitTorrent clients. At the same time we intend to expand the reporting in- terface with information related to processor and memory usage, number of connections for easy interpretation and analysis. The MonALISA interface will also be expanded for live reporting of various transfer parameters. Figure 4. Close-to real-time monitoring References is a small delay between the client collecting sufficient data [1] http://wiki.theory.org/BitTorrentSpecification and sending it and the MonALISA agent processing it. Figure 4 is a screenshot of a download session rendered [2] J. A. Pouwelse, P. Garbacki, J. Wang, A. Bakker, J. using MonALISA. By using a MonALISA client, one can Yang, A. Iosup, D. H. J. Epema, M. Reinders, M. R. have a close to real time evolution of a BitTorrent download van Steen, H. J. Sips, ”TRIBLER: a social-based peer- session. to-peer system”, Concurency and Computation: Prac- tice and Experience, Volume 20 Issue 2

6 Related work [3] Lei Guo, Songqing Chen, Zhen Xiao, Enhua Tan, Xi- aoning Ding, and Xiaodong Zhang, ”A Performance While there are extensive studies and proposals regard- Study of BitTorrent-like Peer-to-Peer Systems”, IEEE ing the internals of the BitTorrent protocol, there has been Journal on Selected Areas in Commmunications, Vol. little concern about comparing and evaluating real world 25, No. 1, 2007 implementations. Guo et. al [3] developed and extensive study of P2P systems but focusing on how the overall be- [4] J.A. Pouwelse, P. Garbacki, D.H.J. Epema, and H.J. havior of the swarm affects the performance. Pouwelse Sips, A Measurement Study of the BitTorrent Peer- et. al [4] [6] have also gathered large amount of informa- to-Peer File-Sharing System”, Technical Report PDS- tion and used it for detailing BitTorrent swarm behavior. 2004-003, 2004 Bharambe et. al [5] studied the BitTorrent mainline client [5] A. Bharambe, C. Herley, and V. N. Padmanabhan, ”An- and measured the impact of number of peers, neighbor set alyzing and Improving BitTorrent Performance”, Tech. on network performance. Rep. MSR-TR-2005-03, 2005

7 Conclusions and future work [6] J.A. Pouwelse, P.Garbacki, D.H.J. Epema, and H.J. Sips, ”The BitTorrent P2P file-sharing system: Mea- surements and Analysis”, Fourth International Work- We have presented a BitTorrent performance evaluation shop on Peer-to-Peer Systems (IPTPS), 2005. framework. While there have been extensive studies regard- ing the performance of the BitTorrent protocol and how it [7] A. Vlavianos, M. Iliofotou, and M. Faloutsos, ”BiToS: can be improved by using carefully crafted algorithms, there Enhancing BitTorrent for Supporting Streaming Appli- have been little studies on analyzing and comparing real cations”, 2006 world implementations of the BitTorrent specifications. We have created an easily deployable solution that enables au- [8] J.J.D. Mol, J.A. Pouwelse, M. Meulpolder, D.H.J. tomated testing of different BitTorrent clients in real world Epema, and H.J. Sips, Give-to-Get: An Algorithm for situations. P2P Video-on-Demand, 2007 Our results identified the libtorrent rasterbar implemen- tation as the fastest client, clearly ahead of other implemen- [9] http://svn.tribler.org/abc/branches/razvan/perf tations. We intend to analyze the logging output and inves- [10] http://www.tribler.org/browser/abc/branches/razvan/perf tigate its source code to identify the clever spots that enable such an improvement over the other clients. [11] http://www.p2p-next.org/ [12] http://arstechnica.com/news.ars/post/20070903-p2p- responsible-for-as-much-as-90-percent-of-all-net- traffic.html [13] http://torrentfreak.com/p2p-traffic-still-booming- 071128/

[14] http://torrentfreak.com/bittorrent-launches-ad- supported-streaming-071218/