Survey of End-To-End Mobile Network Measurement Testbeds
Total Page:16
File Type:pdf, Size:1020Kb
1 Survey of End-to-End Mobile Network Measurement Testbeds, Tools, and Services Utkarsh Goel, Mike P. Wittie, Kimberly C. Claffy, and Andrew Le Abstract—Mobile (cellular) networks enable innovation, but can also stifle it and lead to user frustration when network performance falls below expectations. As mobile networks become the predominant method of Internet access, developer, research, network operator, and regulatory communities have taken an increased interest in measuring end-to-end mobile network performance to, among other goals, minimize negative impact on application responsiveness. In this survey we examine current approaches to end-to-end mobile network performance measurement, diagnosis, and application prototyping. We compare available tools and their shortcomings with respect to the needs of researchers, developers, regulators, and the public. We intend for this survey to provide a comprehensive view of currently active efforts and some auspicious directions for future work in mobile network measurement and mobile application performance evaluation. Index Terms—Mobile network, measurement, testbeds. ✦ 1 INTRODUCTION users without adequate testing [12], [13]. Researchers lack network performance data, or tools to acquire such OBILE (cellular) network applications deliver in- data, in order to rapidly test hypotheses and focus teractive services, generally supported by back- M on realistic network performance problems. Network end logic deployed on cloud infrastructure. These ap- operators need to monitor and troubleshoot end-to-end plications support a wide breadth of functionality, such network performance without degrading base station as live video, social gaming, communication services, throughput. Finally, regulators have a limited view of and augmented reality [1]–[4]. Future services will in- network performance, especially with respect to traffic creasingly leverage cloud-based datasets and processing shaping by network providers, impeding their ability power for innovative applications of live speech transla- to tackle performance challenges and roadblocks for tion, real-time video analysis, or other computationally sustained innovation in the mobile space [14], [15]. intensive tasks [5], [6]. As the frequency of interactions This paper provides a comparative analysis of cur- between mobile devices and back-end servers increases, rently available network measurement platforms for application responsiveness will be increasingly tightly end-to-end mobile network measurement, monitoring, coupled with end-to-end network performance. and experimentation. We further categorize measure- To innovate in the interactive mobile application space, ment platforms as research testbeds for network ex- developers deploy communication protocols with so- perimentation, extensible distributed measurement tools, phisticated data delivery techniques that support re- and services for widespread monitoring of networks sponsive communications under a range of network performance. In the following sections describe the most arXiv:1411.5003v3 [cs.NI] 19 May 2015 conditions [7]–[11]. These techniques are not always suf- salient features of each platform, and how some features ficient and developers are sometimes forced to redesign differ across them. Table 1 compares the testbeds and application functionality to mask poor network per- tools in terms of their experimentation flexibility, device formance. However, these latter optimizations require selection criteria, resource protection, and other features. detailed network performance data that is often not readily available, which results in challenges across the Based on our review of current measurement efforts, cellular ecosystem. For example, developers face the we observe that although existing approaches comprise undesirable choice of evaluating performance of their only a patchwork of needed functionality, they already mobile applications in limited private deployments that generate powerful insights to guide development, re- lack geographic diversity, or distributing their code to search, and regulatory actions. However, in spite of the relative maturity of several measurement platforms, • U. Goel and M.P. Wittie are with the Computer Science Department, Montana State University, Bozeman, MT 59717. daunting problems remain including support for wide- E-mail: utkarsh.goel, [email protected] scale application prototyping and deployment, detection of traffic shaping, and long-term network performance • KC Claffy is with UCSD/CAIDA, La Jolla, CA 92093 E-mail: [email protected] monitoring. Most existing mobile measurement tools have been developed in isolation, and one motivation • A.Le is with Mintybit, Santa Barbara, CA 93111 for this survey is to foster more concerted and coopera- [email protected] E-mail: tive efforts at standardization of measurement libraries, 2 Network Testbeds Network Tools Network Services Network Network Uncurated Curated Standalones Libraries Discovery & Monitoring Diagnosis Seattle PhantomNet PhoneLab SciWiNet LiveLabs WindRider Mobitest ALICE NetPerform MITATE NDT PortoLan FCC SpeedTest MySpeedTest RILAnalyzer Mobiperf Ookla SpeedTest RadioOpt OpenSignal Netalyzr Measurement Capabilities Traffic shaping/DPI X X X X X Active measurements X X X X X X X X X X X X X X X X X X X Passive measurements X X X X X X X X X Measurement data publicly 2 X 2 X X X X available Custom packet content X X X X 1 Peer-to-peer traffic 2 X X ICMP traceroutes 2 X X X 3 X Programmable execution 4 X X X X X X environment Access to mobile device X 2 X X X 5 6 7 5,6 5,7 5 5,6, 5,6, 5,6, 5,7 X sensor data 7 7 7 Experiments can be sched- X X X X X X X X uled on specific clients IPv6 support 2 2 X X X Allow traffic on ports < 3 3 X 1024 Reports network problems X X X Supported mobile OS plat- 8 8,9, 8 8 8 8,9 8,9 10 8 8,9, 8 8 8 8,9, 8,9, 8,9 8,9 8 8 8 form 12 11 10,13 10,11 Network coverage map X X Device Selection Criteria Geographic location X X X X X Q Device model X X X Device type (GSM/CDMA) Q Battery charge level X X X Q Carrier signal strength X X Q Network carrier X X Q Network type (Wi- X X X Q Fi/Cellular) Time of day X X X X X X X Resource Usage Limit Transmission rate X Bandwidth cap X X X X X X X X X Minimum battery charge X 2 X X Port restrictions X X Misc. Measurement scheduling X X X X X X API Supports devices behind X X X X X X X X X X X X X X X X NAT/Wi-Fi Requires rooted phones X X X X Open to public X X X X X X X X X X X X X X X X X X X User incentive model R R,A U S,U C C C A C,S S C C,S C,S C,S S S A Experiments require IRB X X approval Open-source X X X X X X X X X Currently active D X D X D D X X X X X X D X X X X X X Records hardware specs O X X X X X X X X Records hardware perfor- O O O X 6 X X mance 3 Legends: 1 – measurements can be directed to specific Web servers. 2 – planned functionality. 3 – on rooted phones only. 4 – through multiple experiment rounds on the same device. 5 – GPS readings. 6 – battery readings. 7 – radio state. 8 – Android. 9 – iOS. 10 – Windows. 11 – Blackberry. 12 – Nokia. 13 – Amazon FireOS A – user Altruism to support measurement capacity. D – under Deployment. O – Optionally. C – user Curiosity to understand their own network performance. Q – only at query time. R – Reciprocal (tit-for-tat). S – provides Service to clients other than measurement data. TABLE 1 Experimentation flexibility matrix of end-to-end measurement testbeds, tools, and services. privacy policies, and technology exchange [16]–[19]. remain bandwidth-constrained, which motivates ISPs to The rest of this paper is organized as follows. Section 2 deploy traffic shaping mechanisms on video streaming reviews goals of end-to-end mobile network measure- and P2P traffic to increase the usable bandwidth for all ment. Sections 3, 4, and 5 respectively discuss measure- mobile users [22]. Traffic shaping can induce high latency ment testbeds, tools, and services for end-to-end mobile that impedes the performance of dynamic content appli- network measurement. Section 6 presents directions for cations such as interactive Web, live video, and group future work and concluding thoughts. communication and collaboration tools [23]. To cope with the complexity of mobile network per- 2 GOALS OF END-TO-END MOBILE formance dynamics, developers need to measure and NETWORK MEASUREMENT incorporate mitigation strategies in their applications. Although not all mobile applications are equally affected The 2014 CAIDA workshop on Active Internet Mea- by poor network performance, the responsiveness of net- surements (AIMS 2014) brought together developers, work applications such as games, interactive video, and, researchers, network operators, and regulators interested to a lesser extent, in-car navigation and augmented real- in mobile (and wireless) network performance [18]. ity requires low latency, stable bandwidth, or both [24]. Although these communities share the goals of under- To improve application responsiveness when latency is standing and improving performance of current mobile high, developers might redesign application communica- networks, they focus on different metrics, and thus the tion protocols and message structures to pack more data tools they produce (Section 3) take different approaches. in fewer round trips between mobile clients and back- end servers [25]. Developers might also strategically co- 2.1 Developers' View of Network Performance locate back-end servers in areas, or networks, where Developers want to provide a responsive application users tend to experience higher latencies [26]. To coun- experience to their users. Although much of the delay ex- teract the effects of low bandwidth, developers, might perienced by user requests is due to back-end processing reduce the size and/or resolution of images and video, and front-end rendering [10], as hardware and software or reduce the frequency of application state updates by processing speed improves, network latency becomes using techniques such as CloudFlare’s Mirage [27] or a dominant concern.