Which Network Measurement Tool is Right for You? A Multidimensional Comparison Study Esma Yildirim, Ibrahim H. Suslu and Tevfik Kosar Department of Computer Science & CCT Louisiana State University 70803 Baton Rouge, Louisiana, USA fesma, ihsuslu, [email protected] Abstract tools differ in the application of the usage of the probe data. Because of that they may also differ greatly in the accu- Network performance measurement and prediction is racy of results they gained and have different effects on the one of the most prominent and indispensable components underlying resources with a characteristic of intrusiveness in distributed computing environments. The selection of the and overhead generated. While some of them may be very most advantageous network measurement tool or system for accurate in predicting the performance and have little sys- specific needs can be very time consuming and may require tem requirements, they may as well lay additional burdens detailed experimental analysis. The multi-dimensional as- over the resource in terms of intrusiveness, may not scale pects and properties of such systems or tools should be well for applications serving a wide number of resources considered in parallel. In this paper, we take two of the or respond very poorly to failures. Also tools may be even most widely used and accepted network measurement tools chosen based on the quality of just being easy to use. The as a case study: Iperf and Network Weather service. We dimensions of differentiality are so many in number and not compare these two prediction tools by listing the pros and considered altogether in the previous studies. cons based on accuracy, overhead, intrusiveness, system Among the performance measurement tools, Iperf and requirements, capabilities, reliability, scalability and re- Network Weather Service (NWS) are the most widely ac- sponse time. We present different methodologies used to cepted and used. Especially Iperf has become a standard measure their performance in previous experiments and run that it is used as a base case for the actual transfer rate to experiments for comparing them to actual FTP, GridFTP compare to other measurement tools [10]. There are vari- and SCP transfers based on different parameters. ous network measurement studies that use Iperf and NWS for measuring performance of network and compare them to other tools [10, 14, 11, 12, 9, 15]. The methodology 1. Introduction that is used in those studies varies and it is important to present different experimental paths the authors followed to The heterogeneity of the resources that make up the com- get a clear idea of which parameters to use in our own ex- putational and data grid environment plays a major factor in periments. While Iperf shows similar characteristics to the the efficiency and effectiveness of the services they provide other tools in terms of design, NWS differentiates itself as a to the user community of such systems. The resources have large-scale performance prediction service which is capable different capabilities and are quite variable over time. Net- of many qualities rather than just being a measurement tool work is among the most variable resources, and measure- [15]. ment tools for predicting the performance of its availability There are studies that compare Iperf and NWS sepa- have become an important part of Grid architectures long rately to actual GridFTP transfers [14, 13, 7] and show ago. Various tools have been designed not only to measure the measurement differences among Iperf-GridFTP and the performance of network but also cpu, memory, disk and NWS-GridFTP. However because those studies represent file systems [5, 4, 3, 2, 6]. the experiments done in different environments using A commonality between the network performance mea- different parameters, it is hard to get a clear idea of which surement tools is the idea of imitating TCP transfers and gives more accurate results to specific protocol transfer sending dummy probe data. However the measurement rates. of window size and stream number has been deliberately In this study, we compare Iperf and NWS in terms of selected so that they do not monotonically increase or de- multi-dimensional aspects such as accuracy, overhead, in- crease preventing any trend for biasing the result [9, 7]. trusiveness, system requirements, capabilities, reliability, Also to be able to see the affect of transfer on the RTT, pings scalability and response time. We give categorization of are issued within regular intervals both during and after the experimental techniques used and define our own experi- transfer [9]. mental analysis path. We also present experimental results The comparison to be made between Iperf and NWS regarding Iperf and NWS’ success of predicting different thus may have the following difficulties based on the tech- types of protocol transfer rates such as FTP, GridFTP and niques described above. The NWS probe size and Iperf s SCP by running extensive tests in aspects of dimensions transferred data has a great difference in amount. While such as accuracy and overhead and compare our results. NWS sends 64KB size probes Iperf makes 10sec duration transfers to be able to capture the accurate bandwidth. Un- 2 Measurement Techniques der those circumstances, we could say that NWS measures available bandwidth while Iperf measures achievable band- width. Of course NWS probe size could be readjusted how- We believe that identifying a correct measurement tech- ever that will have an incredible impact on the non-intrusive nique is essential in the way that it affects the results gained design goal of NWS. and the inference we make about them. In this section we Since we want to deal with large file transfers that may present the techniques and parameters used to test either take long time, the following issues may arise. The units of performance of a network or measurement tools. transfers are different as in the case of Iperf and GridFTP. The different parameters used in the studies [10, 11, 12, While the former does the transfer in seconds, the latter does 9] are TCP buffer size, probe size, file size, measurement it in bytes. How can we make sure that Iperf transfers the duration, window size and probe interval. Another impor- sufficient amount of data as to predict the transfer proto- tant aspect is the current load of the traffic during measure- col(e.g. GridFTP)? Though Iperf has an option that will ments. As that property is quite variable over time, it may take data to be transferred from an input file, if the file is too be seen that inference made on the results gained during small Iperf may not reach to the steady-state and may report that traffic can be biased and may not represent rules to be inaccurate bandwidth measurement. If it is too large we also applied all the time. For that reason, some of the studies know that Iperf has a limit on memory for transfers and after used controlled cross traffic so that they could easily see the that threshold it reports also inaccurate results.The response effect of their transfers. These are called synthetic traffic time also will be affected depending on the probe/transfer load [11]. However in that case the consequences of a real size we select. In the end we may find ourselves making variable traffic is ignored and assumptions made can have a sacrifice from accuracy to gain more from response time. a quite inverse effect. The real network load may not rep- When NWS enters at that point, it will have different pa- resent the patterns that have been looked for. Hence there rameter options and use a different underlying method for are other studies that also use generally wide area network measurement other than Iperf . So we have to design a mea- loads. In that case, the measurements are repeated over time surement strategy that will take into account all the possible in regular or random periods. Generally the results are rep- problems that may occur and must decrease the effects on resented as median values and the quartile ranges. each other to be able to do healthy experiments. While the performance of different measurement tools are tested and also with actual transfers, it is important to run the tools back-to-back for the same period of time so 3 Tool Evaluation that the tools can experience the similar network variation for a specified set of parameters. However for the same The evaluation criteria focuses on three purposes. First, tool, each measurement has a regular or random time in- we need to find the parameters that affect the accuracy and terval between the previous one. While a tool’s accuracy overhead of Iperf and NWS. After defining which param- is compared to an actual transfer, the measurements how- eters to tune, we should optimize those parameters to ef- ever may not be done with the same time interval between ficiently use both tools and observe behaviors of the tools successive iterations. In Vazhkudai et al [14], 400 GridFTP in both local and wide area networks. Finally, the ability transfers occurred comparing the 1500 NWS probes during of Iperf and NWS to predict different data sizes by differ- the total 12 hour period with an NWS probe size of 64KB ent types of protocols(e.g. GridFTP, FTP, SCP) must be to a randomly selected file transfers of size ranging between measured since the insights gained from these results can 10MB and 1GB. be helpful for Grid applications trying to make intelligent To be able to see the effect of window size and paral- data transfer choices and tuning (e.g. data-aware sched- lel streams another strategy has been used. The sequence ulers) based on network conditions.