Measuring Home Network Performance Using Dedicated Hardware Devices

CheesePi: Measuring Home Network Performance Using Dedicated Hardware Devices

BINIAM GEBREGERGS GUULAY

Master’s Thesis at KTH Laboratory of Communication Networks Supervisors at SICS: Dr. Liam McNamara and Dr. Ian Marsh Examiner: Prof. Gunnar Karlsson

TRITA-EE 2015:114

Abstract

Internet users may not get the service quality promised by their providers, and also may not know what service they can receive. When users experience poor Internet connection performance, it is not easy to identify the source of the problem. We develop CheesePi, a distributed measurement system that measures the Internet connection experience of home users based on some network performance attributes (e.g. latency, packet loss rate, and WiFi signal quality). The CheesePi runs on a Raspberry Pi (a credit card sized computer) connected to the user’s home network as a measurement agent. It is important to measure the network performance from the user’s side since it is diﬃcult to measure each individual’s link from the operator (provider) side. Each measurement agent conducts measurement periodically without disturbing the user’s Internet quality. Measurements are conducted during popular media events from SICS (Swedish Insti- tute of Computer Science) and student accommodations. The measurement results show customers with an Ethernet connection experienced signiﬁcantly better latency and packet loss compared to WiFi users. In most of the measurements users at SICS perceived better latency and packet loss compared to the users at the student accommodation. We also quantify how customers experienced lower performance when streaming from websites which do not use CDN technology compared to the websites which do use CDN, particularly during popular media events. Sammanfattning

Internetanvändare får kanske inte den kvalitet på deras inter- netuppkoppling som leverantörerna låvar, och vet kanske inte hel- ler hur bra uppkoppling de potentiellt kan få. När en användare upplever dålig prestanda på internetuppkopplingen så är det svårt att identifiera vad som är källan till problemet. Vi har utvecklat CheesePi, ett distribuerat mätningssystem som mäter hemmaan- vändarens upplevelse av internetuppkopplingen baserat på nät- verksegenskaper (t.ex. latens, andel förlorade paket, styrkan på WiFi signalen). CheesePi kör på en Raspberry Pi (en dator stor som ett kreditkort) som är kopplad till användarens hemnätverk som en mätningsagent. Det är viktigt att mäta nätverksprestanda från användarens sida eftersom det är svårt att mäta varje indi- vids länk från leverantörens sida. Varje mätningsagent genomför mätningar periodiskt utan att störa användarens internetkvalitet. Mätningar genomförs under stora media-event från SICS (Swe- dish Institute of Computer Science) och studentlägenheter. Mät- ningsresultaten visar att användare som använder Ethernet för sin uppkoppling upplever en märkbart bättre latens och färre för- lorade paket jämfört med WiFi-användare. I de flesta mätningar upplevdes bättre latens och färre förlorade paket hos använda- re i SICS jämfört med i studentlägenheter. Vi kvantifierar också hur användare upplevde sämre prestanda när de strömmade in- nehåll från webbplatser som inte använder sig av CDN teknologi jämfört med webbplatser som använder CDN, speciellt vid stora media-event. Acknowledgements

First and foremost thanks to the almighty God for everything. I am so grateful to my supervisors Dr. Liam McNamara and Dr. Ian Marsh for giving me the opportunity to work on this master thesis, and the continuous support, guidance, and constructive suggestions throughout the whole thesis. I would like to express my gratitude to Prof. Gunnar Karlsson for being supportive and patient as my examiner for this master thesis. I want to extend my sincere thanks and appreciation to my family and friends for their unconditional support and love. This master program would have never been possible without your support and encouragement. I also would like to thank SICS for providing me an oﬃce and all necessary materials to accomplish this project. And many thanks goes to other master thesis students Urban Peterson and Gustaf Lindstedt for all the fruitful dis- cussions, especially Gustaf for translating the abstract part of this report to Swedish. Contents

List of Tables

List of Figures

1 Introduction 1 1.1 Motivation ...... 1 1.2 Goals ...... 2 1.3 Organization of the report ...... 2

2 Background 3 2.1 Internet Protocol (IP) Networks ...... 3 2.2 Access technologies ...... 5 2.2.1 Ethernet ...... 5 2.2.2 WiFi ...... 6 2.2.3 Mobile/cellular connection ...... 6 2.3 Network performance measurements ...... 6 2.4 Network performance measurement tools ...... 7 2.4.1 Ping ...... 7 2.4.2 Httping ...... 8 2.4.3 Traceroute ...... 8 2.4.4 Measurement device ...... 8 2.4.5 Storage ...... 9 2.5 Content delivery approaches ...... 9 2.5.1 Single server ...... 9 2.5.2 Multiple servers ...... 9 2.5.3 Content delivery network (CDN) ...... 10

3 State of the art within network monitoring 11 3.1 RIPE Atlas ...... 11 3.2 ARK ...... 11 3.3 Samknows ...... 12 3.4 NetBeez ...... 12 3.5 Software based network monitoring systems ...... 13 3.5.1 SpeedTest ...... 13 3.5.2 ICSI Netalyzr ...... 14 3.5.3 Fathom ...... 14 3.5.4 PingER ...... 14

4 Methodology: CheesePi development and experiment conducting approaches 15 4.1 CheesePi architecture ...... 15 4.2 ICMP-based measurements ...... 16 4.3 HTTP-based measurements ...... 17 4.4 Traceroute measurement ...... 18 4.5 Database design ...... 18 4.5.1 Measurement agent design I ...... 18 4.5.2 Measurement agent design II ...... 20 4.5.3 The Central Server Design ...... 21 4.6 Sampling of network packets ...... 22 4.7 Measurement frequency scheduling ...... 23 4.8 Data analysis tools ...... 24 4.9 Measurement agent’s location ...... 25

5 Network performance evaluation 27 5.1 Measurement motivation ...... 27 5.2 Popular media events ...... 28 5.2.1 Boxing - fight of the century ...... 29 5.2.2 Eurovision 2015 ...... 35 5.2.2.1 Eurovision 2015 semi final one ...... 36 5.2.2.2 Eurovision semi final two ...... 42 5.2.2.3 Eurovision 2015 final ...... 44 5.2.2.4 One week after the Eurovision 2015 final . . . 48 5.2.3 2015 UEFA Champions League final ...... 51

6 Discussion 63

7 Conclusions 67

8 Future work 69

9 Lessons learned 71

Bibliography 73 A SICS open house 2015 77 List of Tables

4.1 Network measurement from an MA in SICS to www.bbc.com in different time intervals...... 24 4.2 First few probes of the first five minutes network measurement from an MA in SICS to www.bbc.com...... 24 4.3 Measurement agent’s location and Internet connection type. . . . 25

5.1 HTTP latency to HBO from different MAs during boxing match between Mayweather and Pacquiao...... 32 5.2 HTTP latency to HBO from SICS Ethernet one week later after the boxing match between Mayweather and Pacquiao...... 33 5.3 Network and HTTP latency to Philstar from SICS Ethernet during and one week after the boxing match between Mayweather and Pacquiao...... 34 5.4 HTTP latency comparison among different MAs during boxing match between Mayweather and Pacquiao...... 35 5.5 Comparison of HTTP latency from SICS Ethernet to different targets before, during, and after the program of Eurovision semi-final one...... 38 List of Figures

2.1 Internet building blocks...... 4 2.2 The TCP/IP Model...... 5

4.1 CheesePi Architecture...... 16 4.2 Ping and httping database design...... 19 4.3 Traceroute database design...... 20 4.4 The CheesePi dashboard...... 21 4.5 Central server database design...... 22

5.1 HTTP latency to different targets from SICS Ethernet during the boxing match between Mayweather and Pacquiao...... 31 5.2 HTTP latency variability to HBO from SICS Ethernet during and one week later after the boxing match between Mayweather and Pacquiao...... 33 5.3 Network and HTTP latency to Philstar from SICS Ethernet during the boxing match between Mayweather and Pacquiao...... 34 5.4 HTTP latency to different targets from an MA at SICS (Ethernet) during Eurovision 2015 semi-final one...... 37 5.5 HTTP latency from a MA at SICS (Ethernet) to EurovisionTV during Eurovision 2015 semi-final one...... 39 5.6 Comparison of network and HTTP delays from a MA at SICS (Ethernet) to EurovisionTV during Eurovision 2015 semi-final one. 39 5.7 HTTP latency from a MA at SICS (Ethernet) to ORF during Eu- rovision 2015 semi-final one...... 40 5.8 Comparison of HTTP latency to EurovisionTV from different MAs during Eurovision 2015 semi-final one...... 41 5.9 Comparison of HTTP latency to SVT among different MAs during Eurovision Semi-final two...... 43 5.10 Comparison of HTTP latency to EurovisionTV among different MAs during Eurovision Semi-final two...... 44 5.11 Network and HTTP latency of different targets form SICS Ethernet during Eurovision 2015 final...... 45 5.12 Network and HTTP latency of different targets form SICS Ethernet during Eurovision 2015 final...... 46 5.13 Comparison of HTTP delay to EurovisionTV from different MAs during Eurovision 2015 final...... 47 5.14 Comparison of HTTP delay to BBC iPlayer from different MAs during Eurovision 2015 final...... 48 5.15 Comparison of HTTP delay from SICS Ethernet to different targets during Eurovision 2015 final and one week after the final...... 49 5.16 Comparison of HTTP latency to EuroVisionTV from student apartment WiFi and SICS Ethernet between the measurements during Eurovision final and one week after the final...... 50 5.17 HTTP latency from student apartment to EurovisionTV during different measurements...... 51 5.18 Network, HTTP, and network packet loss relation of measurement from SICS Ethernet to ATDHE during UEFA 2015 Champions League final...... 53 5.19 HTTP latency of different MAs to ATDHE during UEFA 2015 Champions League final...... 55 5.20 Network latency of different MAs to ATDHE during UEFA 2015 Champions League final...... 56 5.21 HTTP latency of different MAs to CricHD during UEFA 2015 Champions League final...... 57 5.22 Network latency of different MAs to CricHD during UEFA 2015 Champions League final...... 58 5.23 HTTP delay of different MAs to CRICFREE during UEFA 2015 Champions League final...... 59 5.24 HTTP latency of different MAs to ESPN during UEFA 2015 Cham- pions League final...... 60 5.25 HTTP and network latency to ESPN from student apartment WiFi during UEFA 2015 Champions League final...... 61

A.1 Average network latency of different MAs to BBC during SICS open house 2015...... 77 A.2 Minimum network latency of different MAs to BBC during SICS open house 2015...... 78 A.3 Maximum network latency of different MAs to BBC during SICS open house 2015...... 78

Chapter 1

Introduction

This chapter gives an overview of this master thesis focus area to help readers understand what the project is essentially about. Next the existing problems that motivate this work will be discussed, followed by the project’s goal. Thereafter the speciﬁc goals and contribution of this master thesis in the project is going to be presented. At last the organization of the report will be explained.

1.1 Motivation

The Internet is one of the very essential needs in the modernised world. We use the Internet for communication, sharing information, watching videos and TV, shopping, and many other activities. According to the Internet World Statistics [22], there are about 9.2 million Internet users in Sweden, 604 million in Europe, and 3.3 Billion around the world. Approximately 85% of Internet users watch online videos [1]. Users become frustrated when videos pause while the system downloads additional data before resuming video playback. According to a new study from the University of Massachusetts Amherst and Akamai Technologies [23], many broadband Internet users give up on an online video if it doesn’t display within two seconds (probably not true in developing countries). Most Internet users, when they perceive a connection problem, they have little idea about its root cause. The problem can be due to a variety of reasons, sometimes it can be a local problem due to poor WiFi connection in the user’s environment. It could be also due to the Internet Service Provider’s (ISP) lack of bandwidth, or several other reasons. Measuring the Internet connection is important for the home user to ascertain if they are receiving the promised service. It is also important from the service providers and regulators point of view, to characterize the whole network and assessment

1 CHAPTER 1. INTRODUCTION of potential degradation in the services they provide. In other words it is monitoring the network and services they provide. Researchers have been trying to measure end-to-end quality-of-service (QoS) for many years, and we want to add our contribution for users’ better experience.

1.2 Goals

The main goal of this project is to build a measurement platfrom (CheesePi) that helps home Internet users to characterize their connections and with collated data to help ISPs and operators to ﬁx network problems. The main goals of this master thesis are:

1. Design the database of the diﬀerent measurement tools in the Raspberry Pi and the central server.

2. Build test modules for ping, httping, and traceroute.

3. Run several scheduled measurements periodically.

4. Perform speciﬁc measurements during popular media events to popular streaming websites and integrate the results into the database.

5. Conduct analysis on the collected results and draw conclusion on the observations.

1.3 Organization of the report

Chapter one introduces the motivation and objective of the project, and the contribution of the master thesis. Chapter two introduces the area of the thesis, and introduce the network tools used in this project. Chapter three discusses the related projects. Chapter four is going to explain the methodology followed for building this project and conducting the experiments. Chapter ﬁve presents the results of the experiment and small discussion about the observations. Chapter six discusses the observations and the followed approach during the master thesis. Chapter seven draws conclusion about this master thesis. Chapter eight presents the possible future works of this project. Chap- ter nine presents the lessons learned during this master thesis. The last part of this report presents the references of articles and technical reports used for this master thesis and an appendix.

2 Chapter 2

Background

First in this chapter, the general purpose of the Internet and how it works will be explained first. Then, different ways of accessing the Internet are going to be introduced. After that, different network performance measurement approaches followed by the tools used for measuring network performance will be discussed. At the end, different content delivery techniques used by different media streaming services will be introduced.

2.1 Internet Protocol (IP) Networks

The Internet is a network of many networks. The connected devices and networks differ in hardware, software and design, so they need a common protocol in order to communicate with each other. The Transmission Control Proto- col/Internet Protocol (TCP/IP) protocol enable inter-connected networks to perform end-to-end communication, so that any node on the Internet should have the ability to communicate with any other node. The TCP/IP protocol suite is divided into four layers for division of labor and easiness of implementation and is called a protocol stack [3]. The four layers are: Application, Transport, Network, and Link layers. The application layer provides a service that enables a user to communicate over the TCP/IP model, for example services such as: HTTP (web), FTP (file transfer), and SMTP (mail). The transport layer provides an end-to-end data transfer between applications or peers, and the most used protocols are the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP). Data is multiplexed in the network stack at this level from information in the port numbers. This is because network flows arriving at this machine, are being used by different applications. The network layer transfers data from source to destination by performing routing functions, and the IP is the most promi-

3 CHAPTER 2. BACKGROUND

Figure 2.1: Internet building blocks.

nent protocol at this layer. In order to demultiplex flows at the router level, the IP addresses are used. And the Link layer is the interface to the network hardware, for example, 802.11, ATM, and X.25. For example, a WiFi user connected to the home router wirelessly wants to send data to a target server. The frame is sent to the first hop (WiFi router), then the router sends it to the next router, and it goes like that until it reach the target server. The frame contains source and the next hop MAC (media access control) address (link layer), source and destination IP address (network or Internet layer), the source and destination port (transport layer), and the data. The destination port tells about the used application layer protocol, for example, if the destination port is 80, it is an HTTP protocol for accessing the web. Unlike the IP addresses, the source and destination MAC addresses change every time during the path till it reaches the destination host. IP defines an addressing scheme to uniquely identify a host and devices on a network. When a packet is sent from the source to a targeted destination,

4 2.2. ACCESS TECHNOLOGIES

Figure 2.2: The TCP/IP Model. it should include the IP address of the sender and the receiver. The routers in the network select where to forward the packet based on the destination IP address. The IP version 4 (IPv4), a widely deployed protocol. IPv4 uses a 32 bit addressing schema to represent a device in the network, which allows more than 4 billion (232) addresses, whilst IP version 6 (IPv6), created to resolve shortage of addresses in IPv4, uses a 128 bit addressing schema. IP does not provide delivery guarantees, and hence is called a best eﬀort delivery, this means packets can be lost, corrupted, and received out of order. It is up to the upper layers like Transmission Control Protocol (TCP) to guarantee the (eventual) delivery.

2.2 Access technologies

Users link to the network via diﬀerent technologies. There are several choices if a user wants to access the Internet namely: WiFi, Ethernet (typically via a cable), and Mobile Data (Cellular Network) are common available choices in 2015.

2.2.1 Ethernet We are not going to discuss the details of the Ethernet protocol in the link layer of the TCP/IP stack. Ethernet is used to connect computers in a home or company to a modem (or router) for Internet access. It typically uses a cable to connect a computer to a modem (router) and does not suﬀer from

5 CHAPTER 2. BACKGROUND medium contention [5]. Some users prefer Ethernet for speed, security or even (simpler) conﬁguration reasons.

2.2.2 WiFi The popularity of laptops and smartphones make WiFi a widely used access method to the Internet. Many people at homes, oﬃces, and public areas use WiFi technology to access the Internet. In WiFi, there is no physical connection between the mobile device and a ﬁxed device that has permanent, often non-mobile and stationary connection to the Internet. Most home users are familiar with a wireless router or an access point. When a user connects to the Internet all the data transferred to the access point is sent over unlicensed radio spectrum. A user should be within the coverage range in order to have access. A periodic beacon sent by the router, needs to be seen to join the wireless network, via a secure protocol exchange. For example, for a wireless router that runs the 802.11n standard, a user should be in the range of 70 meters indoors or 250 meters outdoors [6].

2.2.3 Mobile/cellular connection In a cellular network the data from a mobile device is transmitted on licensed spectrum via a radio-to-air interface [7]. The signal is received at a ﬁxed base station. Each base station is responsible for a physical area, often called a cell. Frequency allocation is an important issue, so not to interfere with adjacent cells and competing providers in the same physical area. 3G/4G networks use CDMA division to achieve separation (as well as frequency bands), however 5G technologies will use CDMA and BDMA. The coverage area of the base station depends on diﬀerent factors: frequency, number of users, antenna, location (existence of mountains and buildings), and other parameters, but they cover larger areas compared to access points using WiFi technology.

2.3 Network performance measurements

It is not easy for customers to monitor the quality of the service they are provided. Currently, from the customer perspective, the performance of the Internet typically means measuring the speed: how fast is my network? or how quickly can I download the data?. But does speed mean everything? The answer is simply no, because speed is not the performance requirement of every online application. The most popular performance measurement metrics for diﬀerent applications are:

6 2.4. NETWORK PERFORMANCE MEASUREMENT TOOLS

1. The upload and download speeds.

2. The delay or latency between two end points.

3. The variation of the delay between selected packets, which is called jitter.

4. The ratio of discarded packets to the total number of packets sent (packet loss ratio).

5. To what extent a sequence of packets is reordered within the network, or even duplicated by the network (packet error ratio).

The purpose of network monitoring is to observe and quantify the network measurement results based on the performance metrics. The two common network monitoring approaches are based on passive and active measurements. The active measurement approach injects test packets into the network by sending packets to servers (testing network reachability and performance) and applications (network reachability and server software, e.g. Apache). The passive approach does not inject test traffic into the network and does not rely on test servers. Instead, passive monitoring uses a device in promiscuous mode to sense existing data traffic. Both approaches have advantages and disadvantages. In active measurements, the traffic is artificial and creates extra load on the network, whilst passive monitoring measures the real traffic originated from the user and does not add extra traffic to the network. Since the passive approach may require sensing all packets on the network, there can be privacy or security issues about how to access/protect the data gathered [12]. The active monitoring provides control on generating the traffic which is not possible in the passive approach, and it helps to have a full control on what and when to conduct measurement.

2.4 Network performance measurement tools

This section introduces some of the tools and device used in a network performance measurement.

2.4.1 Ping Ping is used for measuring the network delay or the round trip time (RTT) taken for a packet from a client to a chosen destination. It is supported by most operating systems. Ping uses the ICMP protocol to send a so called ’echo’ request and reply packets. It puts a timestamp in the packets to measure the RTT, and a sequence number to calculate the packet loss for the unreceived

7 CHAPTER 2. BACKGROUND packets. The main problem of ping is: it can be ignored by some servers due to the suspicious nature of malicious attacks, albeit of a high frequency nature, signiﬁcantly higher than the default one of second inter packet time.

2.4.2 Httping Httping is used for measuring the latency of the network path and behavior of a web server. Since the httping requests a full HTTP response from the server, the returned size of the data might be diﬀerent for each Uniform Re- source Locator (URL). Httping works on most operating systems, plus there is an Android version. Httping can include the DNS (Domain Name System) lookups delays as well.

2.4.3 Traceroute Traceroute is a network diagnostic tool that tracks the route (path) of a packet from a source to a destination and measures its transit delays across an IP network. It is used to trace the path a packet follows in a network, and can be used to identify connectivity problems. By default it sends a sequence of UDP packets addressed to a destination host; ICMP Echo Request or TCP SYN packets can also be used [13]. Traceroute starts by sending a packet with a Time-To-Live (TTL), also known as hop limit, at a value of one. The ﬁrst router decrements the TTL value, and sends an ICMP "time exceeded" reply to the source due to the TTL value in the packet being zero. Then the source sends the next packet with a TTL value incremented by one. The previous router forwards the packet (as the value is now valid) but the next router drops the packet as the previous one decremented the counter to zero. In other words, traceroute acts like an expanding ring search, increasing the scope of the packets. This continues until the destination is reached and returns an ICMP "Port Unreachable" reply or a max (30 or 64) is reached. This is because traceroute packets should not circulate the Internet indeﬁnitely. Three probes (by default) are sent at each TTL setting, and the terminal (console) displays the address of the gateway or the router in the path and the round trip time of each probe. It is supported (included) by most of the modern operating systems. Some routers block ICMP/UDP packets.

2.4.4 Measurement device There are a number of ways to host a measurement suite. Some of them will be discussed in the state of the art section. We have chosen a standard hardware

8 2.5. CONTENT DELIVERY APPROACHES device, with our dedicated software platform, CheesePi. The Raspberry Pi is a credit card sized single board small computer developed in UK by the Raspberry Pi foundation [10]. The price of a Raspberry Pi is very aﬀordable and about 5 million units have been sold [11]. A Raspberry Pi2 models cost around $35. Due to its robustness, low price, low power consumption, size, and other factors Raspberry Pis are becoming popular and start to be used in many projects including network measurement projects (NetBeez, ARK) see below.

2.4.5 Storage One can store measurement data in a number of different ways. Common log or CSV files are still the predominant format. However within network measurements there has been a trend towards more structured data (M-lab for example). A database is one way to store the data. The data is organised in a way it can easily be accessed, managed and updated by other computer programs. Relational databases, a table-based database represents entities as a schema and allows users to relate data fields and tables with others stored in the database [15]. Recently, non-relational databases, document based and key-value pairs with wide column fields are becoming a popular alternative technology for analytics of large-scale data.

2.5 Content delivery approaches

When users visit a website to watch a movie or read news they don’t know how the content is delivered to their device. In this section we are going to discuss diﬀerent content delivery approaches.

2.5.1 Single server The website has only one server, the request for every service from the connected clients is processed by a single server.

2.5.2 Multiple servers One of the most commonly used applications of load balancing is to provide a single Internet service from multiple servers. For Internet services, the load balancer is usually a software program that is listening on the port where external clients connect to access services. The load balancer forwards requests to one of the "backend" servers, which usually replies to the load balancer. This

9 CHAPTER 2. BACKGROUND allows the load balancer to reply to the client without the client ever knowing about the internal separation of functions.

2.5.3 Content delivery network (CDN) A content delivery network or content distribution network is a system that replicates digital content over mirrored servers, that are located in suitable places in the Internet [8]. The goal of the CDN is to deliver fast and reliable data by distributing the content to caching servers located closer to the end consumers than the original servers. Most of the popular web streaming like BBC, SVT, YouTube, and Netﬂix use CDN technology to deliver their content. If a user requests an object or a ﬁle from a website that uses CDN technology, instead of the hosting server the CDN takes care of it. This is typically done through the DNS request returning an address to the CDN server, despite the user typing in the commonly known address. In the UNIX system the command "host" can reveal where a CDN is being used. The CDN will consider the geo-location of the visitor and the object will be served from the closest caching server. If the assigned cache server does not have the requested object, it asks other cache servers, then the original (host) server [9]. Note, sometimes the CDN might have to remove some content, if the total catalog size it holds is large and the popularity of items is highly variable.

10 Chapter 3

State of the art within network monitoring

This chapter introduces network monitoring projects and how they are related to CheesePi. It also explains the network tools used during the project. We begin with monitoring systems that use a hardware-based measurement agent. Then it explains the software based network monitoring systems.

3.1 RIPE Atlas

RIPE Atlas is a project measuring Internet quality around the world using measurement probes. The goal is to measure Internet traﬃc while providing users with information about their connection [24]. Probes perform measurements using ping and traceroute with support from DNS and SSLcert. Users with active hardware probes produce rendezvous points that can be used to set up user deﬁned measurements involving other probes. The source code and RIPE data is available to the public. Around eight thousand probes have been deployed as of 2015. An open source approach is common with CheesePi, but RIPE uses dedicated hardware whilst CheesePI uses commodity hardware.

3.2 ARK

The Archipelago Measurement Infrastructure (ARK) uses 120 Raspberry Pis deployed to perform network monitoring to quantify topology (connectivity) and dynamics (congestion) using active measurements for both IPv4 and IPv6 [25]. As of August 2015, the developers UCSD, have deployed 120 nodes. Quantifying the congestion is done by a Time-Sequence Ping (TSP), a crafted sequence of pings along a selected network path. The project conducts mea-

11 CHAPTER 3. STATE OF THE ART WITHIN NETWORK MONITORING surements on the global ARK overlay network predominantly using traceroute. Variations in the diurnal variation in delay have been quantiﬁed. ARK project is a pure networking research project, but CheesePi is a measurement system targeting home internet users. As said both ARK and CheesePi use Raspberry Pi as the measurement agents.

3.3 Samknows

It is a project that measures broadband performance for consumers, ISPs, and government (regulators) around the world [26]. They use Whiteboxes (a modified router) to measure the performance of users’ broadband. The hardware is connected to the user’s router and runs tests based on consumers interest without disturbing the Internet experience. It also sends measurement data to a central server for permanent storage. They also use a web-based measurement similar to Whitebox, which runs tests from the user’s computer. The Whitebox is configured to run several performance measurements. It measures download and upload speed (throughput) between the Whitebox and closely located test node (server). It also measures Voice over IP quality, UDP packet loss and latency under normal and loaded conditions, FTP throughput, DNS resolution latency, peer to peer throughput, internet connection lost, web page loading latency, email relaying latency, bitrate reliability and startup latency during video streaming, video quality of experience for popular media streaming web servers (e.g. YouTube, Netflix, and BBC iPlayer), performance of multicast IPTV over RTP. They use a dedicated device for continuous measurement and a dashboard for visualizing the results based on consumers interest, which makes it similar to CheesePi. But customers have no freedom of controlling the Whitebox and can not use it for other purposes, whereas the Raspberry Pi which runs the CheesePi platform can be used for many different purposes. They have one or more test nodes (servers) in the popular cities around the world from network operators [27], and the stored data in the central server is used by customers, ISP, internet regulators and academia. Samknows has a good collaboration with different organizations, such as Ofcom in the UK and EU research projects.

3.4 NetBeez

NetBeez is a distributed network monitoring system, which is mainly targeted at companies with oﬃces in diﬀerent locations and a centralized IT depart-

12 3.5. SOFTWARE BASED NETWORK MONITORING SYSTEMS ment [28]. The NetBeez agent (Beez) is deployed behind a switch in every remote location as if it was a normal user. The agents then simulate users in the organisation and monitors the network, if a problem occurs it makes troubleshooting easier by using the diagnostic information from the Raspberry Pi. The agent, which is a Raspberry Pi, is configured to run several tests. It runs ping to verify the connectivity and round trip time between two agents (or other chosen host), the DNS resolution latency, traceroute to check if there is a change in the network path and observe the bottleneck hops, Iperf to test the stress of the bandwidth, and measure the performance of an http server by evaluating the http test packets. All the starting and stopping of the measurement tests are controlled by the NetBeez server, it also collects the test results from the NetBeez agents. The users are able to control the function- alities through a NetBeez web based dashboard, and they can visualize the performance on different aspects of the network (locations, targets, historical data). Similar to CheesePi, NetBeez also uses Raspberry Pis as measurement agent. However, CheesePi targets are home users and NetBeez customers are companies who have offices in different locations. CheesePi is free measurement platform while NetBeez offers subscription based services.

3.5 Software based network monitoring systems

In this section monitoring systems, which do not use dedicated hardware for conducting the measurements, or use a software based approach. Of course deploying software is somewhat easier than a hardware-based approach and can achieve larger scale measurements and coverage. We explain those that are either well known, or are similar to CheesePi.

3.5.1 SpeedTest Speedtest is a web service that provides free analysis of Internet access parameters [29]. Probably from the user perspective it is the most well known one, globally. It was founded by Ookla in 2006, The service measures the bandwidth and latency of a visitor’s Internet connection against closest host servers. As of October 2015 [30],They have 4085 distributed host servers around world. This makes it somewhat similar to Bredbandskollen (a Swedish web-based speed test) and SamKnows. Each test measures the data rate for the download di- rection, i.e. from the server to the user computer, and the upload data rate, i.e. from the user’s computer to the server. The tests are performed within the user’s web browser using the http protocol. As of 2015, over 7 billion

13 CHAPTER 3. STATE OF THE ART WITHIN NETWORK MONITORING speed tests have been completed. The site also oﬀers detailed statistics based on test results. This data has been used by numerous ISPs and researchers in the analysis of Internet access around the world.

3.5.2 ICSI Netalyzr Netalyzr is a network measurement and debugging service/tool that evaluates users’ Internet connectivity [31]. The tool aims to be both comprehensive in terms of the properties measures, and easy to use as it is web-based for users with little technical background. Netalyzr is a Java applet that communicates with a suite of measurement-specific servers. Traffic between the two then probes for a diverse set of network properties, including outbound port filtering, hidden in-network HTTP caches, DNS manipulations, NAT behavior, path MTU issues, IPv6 support, and access-modem buffer capacity. In addi- tion to reporting results to the user, Netalyzr also forms the foundation for an extensive measurement of edge-network properties. Along with describing Netalyzer’s architecture and system implementation, they presented a detailed study of 130,000 measurement sessions from 99,000 public IP addresses that the service has recorded since being made publicly available in June 2009.

3.5.3 Fathom Fathom has been used in home network environment [32] which has similar goals to ours, but diﬀers in the software/hardware and the external/embedded measurement approaches. Fathom is a Firefox extension that implements the measurement primitives of Netalyzer to measure network characteristics using JavaScript. It adds 3% to the times to load a web page and can time-stamp network events to within one millisecond.

3.5.4 PingER The PingER (Ping End-to-end Reporting) project monitors the end-to-end performance of Internet links worldwide [33]. Developed by the IEPM group at the SLAC National Accelerator Lab in California, PingER has a decade old data repository of network performance measurements from and to sites around the world. Currently, there are over 60 active monitoring hosts in 23 countries monitoring over 300 sites in over 165 countries that cover 99% of the world’s Internet connected 3.2 billion population [22].

14 Chapter 4

Methodology: CheesePi development and experiment conducting approaches

In this chapter we will discuss the research methodology used in the CheesePi project. We applied a quantitative methodology to conduct the research [14]. We used several networking tools to measure the quality of the Internet service and design a database to store the results. Then we conducted analysis on the collected results. This chapter will discuss how the measurement tools and the database have been designed and the important techniques used for the sampling and scheduling the performance measurements.

4.1 CheesePi architecture

The measurement agent (MA) runs on a Raspberry Pi connected to a home network. This measurement agent runs tests periodically without disturbing the perceived Internet experience of the user. In practice this means using as little capacity as possible. A controller distributes schedules on how tests should run on the MAs. There is a database for storing the measurement result of the four tools namely; ping, httping, traceroute, and VoIP in both the MA and the central server. A collector organizes the results collected from the MAs and stores them in the central server database. A tool recorded the WiFi contention is also part of the system. The user can visualize the real-time and history of the experienced Internet performance on a web-based dashboard, which is hosted locally on the Raspberry Pi. The detail of each test modules will be explained later.

15 CHAPTER 4. METHODOLOGY: CHEESEPI DEVELOPMENT AND EXPERIMENT CONDUCTING APPROACHES

Figure 4.1: CheesePi Architecture.

4.2 ICMP-based measurements

We use the default installed ping on Raspbian, an optimized Debian based operating system for Raspberry Pi. The controller in the central server schedules the MA to do the delay measurements. The schedule consists of the number of measurements to perform, the landmarks (the target destinations), and when to start the measurements. In CheesePi, the controller decides how and when the measurements should be conducted. For this master’s thesis, we use a crontab, a wall-clock job scheduler in Unix operating systems to schedule the measurements. For example, the cron job schedule can be, every five minutes conduct ten ping measurements to BBC web server with an interval of one second. The default ping packet size we used was 64 bytes, which is eight bytes of the ICMP header and 56 bytes of (null) payload. Before we choose the ping packet size we conducted an experiment to see how the packet size affects the network performance. We sent 100 probes each to www.bbc.com with packet sizes of 64, 128, 256, and 512 bytes in parallel. The experiment was taken at SICS, which the download and upload speed is around 100 Mbps. The average round trip times (RTTs) were 30.75, 30.79, 30.88, and 31.44 ms for the ping measurements with 64, 128, 256, and 512 byte packet sizes. The results indicate that the latency is similar for the different packet sizes and

16 4.3. HTTP-BASED MEASUREMENTS we decided to use 64 bytes. We conduct the experiment by considering users with limited data plans, or some form of constrained access. The cron job runs the Python module that conducts delay and packet loss measurements, according to a schedule, for example every ﬁve minutes. The Python module works as follows:

1. It runs a number of probes, for example, "ping -c 10 www.facebook.com"

2. It reﬁnes and organizes the ping results in a suitable format to be stored in the database.

3. It stores the ping results and other necessary information, which helps to uniquely identify the MA measurements in the central database, in a local database. For example, average RTT, IP address of the MA, destination domain name, MAC (Media Access Control) of the MA, and other information.

4. It also stores the measurement results in a ﬁle (optional).

According to the schedule, a program sends its collected results to the CheesePi central server. For example, if we decide to send the data every 23:00 UTC (Coordinated Universal Time), at 23:00 the program checks if there are accumulated results since the last submission and it transmits the data to the central server. The controller is responsible for the distribution of the data submission time. Because if all MAs (we assume the number of MAs is large) send their data at the same time, the central server can be busy and this can lead to improper data handling.

4.3 HTTP-based measurements

We installed httping on all Raspberry Pis for measuring the performance of a web server. Similar to ping, in CheesePi the controller in the central server schedules when the http-delay measurements should be conducted. For this master thesis, the cron job runs the python script that performs httping measurements at the same time as the delay measurements. By running both the httping and ping scripts consecutively, we can ascertain whether it is the network or the webserver that is the source of the latency to that particularly host. Unlike ping, we do not choose the packet size for the httping probes. Since the httping program returns the HTTP header of the target web server, the size can be diﬀerent for each destination. The httping python script runs httping measurements in a very similar way to ping measurements, which organizes the results in a suitable format

17 CHAPTER 4. METHODOLOGY: CHEESEPI DEVELOPMENT AND EXPERIMENT CONDUCTING APPROACHES for the database, stores them in a local database and a ﬁle (optional). Then the module uploads the results to the CheesePi central server in a similar manner as ping results uploaded.

4.4 Traceroute measurement

We use the default installed traceroute in Raspbian in all MAs. Unlike ping and httping, traceroute sends three packets at a time and the packets may follow different paths. Therefore the Python module, that does traceroute measurements, has to take into account this potential hurdle. The traceroute module works as follows. 1. It runs traceroute command "traceroute www.facebook.com". 2. It refines and organizes the traceroute results in a suitable format to be stored in the database as in Figure 4.3. 3. It stores the traceroute results and other necessary information, which helps to uniquely identify the MA measurements in the central database, in a local database. 4. It also stores the results in a file (optional). Then according to the schedule chosen by the controller the upload module sends the accumulated results to the CheesePi central server.

4.5 Database design

This section describes how the database was designed in the MA and the central server. Choosing an appropriate database for CheesePi was evaluated and relational (SQL, MySQL) and non-relational (Time series, Influx) databases were evaluated during the development of this project. The different databases considered in this project and the scientific approach followed for choosing them will be discussed next. Since CheesePi is a system that helps users to characterize their real-time network performance, how these databases in the MA were designed and what are the advantages and disadvantages of each will be discussed next.

4.5.1 Measurement agent design I Relational databases are well suited in carrying out complex queries. And MySQL, an open source relational database was chosen for CheesePi because

18 4.5. DATABASE DESIGN of its open source freedom and online support, scalability and ﬂexibility, high performance, strong security options for data protection, popularity, and management ease. Below are some cases where a single query is used to retrieve data from multiple tables. These are used during a visualization task and analysing the network performance.

Figure 4.2: Ping and httping database design.

In the MA the ping and httping measurements are each represented by a single table. The table designs of httping and the ping are the same since the metrics used by both tools are similar. Within each tool there can be differences in the output, however to keep the tables manageable we use similar metrics. Some of the attributes like delay (RTT) and packet loss (%) are performance metrics. Other attributes like timestamp and starting and ending time are used in management of the data and measurements. The source and destination addresses plus the target hostname identifies the two end points. And the Ethernet MAC address is used to identify an MA in the central server. As one can see in Figure 4.3 the traceroute measurement result is represented by three tables due to its different behavior from the previously discussed tools. The tables: Traceroute_Seq, Traceroute, and Hop are related by an attribute ID. The Hops table which stores the details of the results, the routers in the path and their delay, should be related with the Traceroute table which stores the measurement information about the time when the measurement took place, and the details about source and destination nodes. So a unique ID (Ethernet MAC address of the MA + auto-increment integer) is a primary key in the Traceroute table to uniquely identify every traceroute measurement and a foreign key in the Hop table to relate the details for each measurement. We make the ID a combination of the Ethernet MAC address

19 CHAPTER 4. METHODOLOGY: CHEESEPI DEVELOPMENT AND EXPERIMENT CONDUCTING APPROACHES

Figure 4.3: Traceroute database design. and auto-increment integer to uniquely identify the MA’s result in the central server. Since it is impossible to auto-increment a text (varchar), before a new row is inserted into the Traceroute table, a trigger auto-increment the integer ID in the Traceroute_Seq table and concatenate it with the Ethernet MAC address of the MA. For example, if the last inserted ID in the Traceroute_Seq and Traceroute was 9 and b8:27:eb:16:7d:a6-9, in the next traceroute probe before the results inserted the the ID in the Traceroute_Seq table incremented by one and the ID in the Traceroute table becomes b8:27:eb:16:7d:a6-10.

4.5.2 Measurement agent design II We don’t want to limit users to store the measurement results only on MySQL. So it is good approach to let users store their results in any database or other ﬁle formats. CheesePi uses Grafana, an open source application for visualizing large-scale metrics data, as a dashboard for displaying the measurement results. Currently, Grafana supports a few time series databases (Graphite, InFluxDB, and Open TSDB). Even though InFluxDB is a schemaless time series database, which is incapable of retrieving results from multiple tables with a single query, there is no need of complicated queries for displaying the

20 4.5. DATABASE DESIGN

Figure 4.4: The CheesePi dashboard. results in the dashboard. So, the CheesePi project decided to use InFluxDB as a default database for storing the measurement results, and a user has a decent looking (and editable) dashboard. The attributes (ﬁelds) in all the performance measurement tools are similar to the MA design I (MySQL).

4.5.3 The Central Server Design Most of the measurement analytics are done in the central server, so it is important to design the database carefully in order to create relations among tables so that data can be retrieved with relatively simple efficient SQL queries. For example, a query might want to retrieve the number of users in Stockholm who experience a delay (RTT) value greater than 250 ms during measurements to www.bbc.com on 20th of May 2015 between 13:00-14:00 UTC. So we need a number of tables which are related by key fields to get this information. The tables are PI, User, Connection, Operation, Ping, Traceroute, Tracer- oute_Hop, and Httping. The PI table stores the Pi ID and its Ethernet MAC address to identify the Pi (MA) uniquely. The User table stores the Pi ID, email address of the user, and alias (temporary name) to identify the user of each MA, as a user can have multiple MAs. Another table is a Connec- tion table, which stores the connection ID, Pi ID, and MAC address. During measurements the MA can use different access technologies, either WiFi or Ethernet, and the connection table identifies the connection type during the measurements. We need to maintain which operation was run, which tool was used during the measurement, the connection type, the start and end time and the success/failure, and the Operation table is responsible to store such information. The ping, httping, and traceroute tables in the central server

21 CHAPTER 4. METHODOLOGY: CHEESEPI DEVELOPMENT AND EXPERIMENT CONDUCTING APPROACHES

Figure 4.5: Central server database design. have similar design to the tables in the MA’s, except the additional Pi ID and Op ID attributes. We want CheesePi to be a robust and scalable the MA was built to store the measurement data in many formats. As one can see in Figure 4.1, the MA can store the measurement results in any format based on user’s choice, but in the central server the collector will organize the data in a format designed for complicated queries.

4.6 Sampling of network packets

In passive monitoring, a real data is used to measure the performance of the network, while active monitoring generates its own data for measurement. Ideally, the sampled measurement probes, the packet size and the frequency of the measurements, should represent the real user experience. But CheesePi should not disturb the browsing experience of the user, so active sampling should be as lightweight as possible, while still gathering data. As was mentioned in section 4.2 an experiment is conducted to choose the minimum packet size for network performance measurement with a minimum extra load on the network. The result shows that no significant difference in the delay and packet loss for the different chosen packet sizes, which are the key metrics of

22 4.7. MEASUREMENT FREQUENCY SCHEDULING

CheesePi for network performance measurement. So the default packet size for a network (ping) measurement in CheesePi is 64 Bytes, but the user can change the default size. In the HTTP measurement it is not possible to select the packet size. The only alternatives are the HEAD and GET HTTP requests. If the httping requests an HTTP GET, the complete page/ﬁle is returned. However, the main purpose of using httping in this project is to measure the latency and packet loss during the connection, not the throughput of the web server. So HEAD is used as a default HTTP request option since its packet size is less than the GET one.

4.7 Measurement frequency scheduling

CheesePi runs the user’s Internet performance measurement periodically. The frequency of the measurements and the sampling duration is different for different applications, so it is important to consider what can happen within the measurement duration and in the interval between the two successive measurements. In CheesePi’s case, the system is trying to measure the general trend of network performance at the transport, network and link layers of the TCP/IP stack. If the interval between two successive measurements is large, it is possible to miss important performance information, and it is also possible to disturb the user’s quality of experience if the interval is small. We conducted an experiment in order to choose a measurement schedule that balances the capturing of the connection behavior and not disturbing the user’s Internet connection. As one can see in Table 4.1 and Table 4.2, first we ran a continuous one hour network measurement of 3600 probes, and the interval between each probe was one second (1 X 3600). This is good to capture the connection behavior but such large probes can disturb the user’s connection experience. We decided to minimize the number of probes, so the next step is comparing the 60 probes of every minute (first minute, second minute, third minute and so on) to sample the network behavior by smaller probes. And as one can see in Table 4.1, the results show the average network delay and packet loss of every minute (60 probes) in a five minute measurement is almost similar to the average five minute (300 probes), which is around 32 ms delay and 0% packet loss. Then we decided to represent the five minute network performance by one minute measurement. But if the measurement is to several different targets, still the number of probes can influence the user experience. So, is it possible to sample the network behavior with less number of probes? We took the first ten probes of every minute measurement and we compare

23 CHAPTER 4. METHODOLOGY: CHEESEPI DEVELOPMENT AND EXPERIMENT CONDUCTING APPROACHES them with the average ﬁve minute measurement, and can be seen in Table 4.2. The average delay and packet loss of the ten probes of every minute is almost the same (around 32 ms and 0%) to the average delay and packet loss of the total ﬁve minute measurement (300 probes). At the end we decided upon ten probes at an interval of one per second to sample the network performance over a 5 minute interval. The path of the network doesn’t change often and we decided to only run the path (traceroute) measurement once every 15 minutes.

Average ping RTT (ms) Ping packet loss (%) 1 hour (3600 probes) 32.42 0 ﬁrst 5 minutes (300 probes) 32.42 0 1st minute (60 probes) 32.38 0 2nd minute (60 probes) 32.36 0 3rd minute (60 probes) 32.34 0 4th minute (60 probes) 32.398 0 5th minute (60 probes) 32.368 0

Table 4.1: Network measurement from an MA in SICS to www.bbc.com in diﬀerent time intervals.

ﬁrst 10 probes of every Average ping RTT (ms) Ping packet loss (%) minute 1st minute (10 probes) 32.32 0 2nd minute (10 probes) 32.3 0 3rd minute (10 probes) 32.33 0 4th minute (10 probes) 32.48 0 5th minute (10 probes) 32.32 0

Table 4.2: First few probes of the ﬁrst ﬁve minutes network measurement from an MA in SICS to www.bbc.com.

4.8 Data analysis tools

In this master thesis we used diﬀerent Python packages for plotting and analysis. For example, we used numpy package for analysis the results, and mat-

24 4.9. MEASUREMENT AGENT’S LOCATION plotlib for plotting the graphs. All the latency vs. time graphs of this report are smoothed by a window size of 50.

4.9 Measurement agent’s location

All the measurement for this master thesis were conducted in Stockholm. Most of the experiments were done at SICS and student apartments in Kista Galleria and Lappis. The internet provider in the student apartments and SICS are Bredbandsbolaget and SUNET respectively.

Measurement Agent Connection Type Location rpi1 Ethernet SICS rpi2 Ethernet SICS rpi3 WiFi SICS rpi4 WiFi Student accommodation (Kista) rpi5 Ethernet Student accommodation (Lappis) rpi6 Ethernet Student accommodation (Kista) rpi7 Mobile Data (3G) Home (Stockholm)

Table 4.3: Measurement agent’s location and Internet connection type.

Chapter 5

Network performance evaluation

This chapter will discuss the motivations behind the conducted measurements. The measurement results gathered from several locations of Stockholm are also going to be discussed. We are also going to look how the Internet performance of home users behave during popular media events.

5.1 Measurement motivation

In order to determine the source of a user’s Internet performance degradation, the quality measurement should consider different access technologies (cable, WiFi, mobile data) allowing us to determine their differing performance characteristics. We can then identify if the performance problems are due to the access technology. As an example, we can measure the Internet performance of two neighbors who have the same service provider with similar service level agreements, but one of them uses a WiFi router to access the Internet and the other one is directly connected through Ethernet. Suppose the network and HTTP measurements are taken from the neighbours at the same time, if the delay and packet loss rate is high in the user who uses WiFi technology, then the problem can be narrowed down to the WiFi network. Further diagnosis such as channel interference, speed limit of the router, contention and so on can be a separate measurement effort. Another approach is to determine the Internet performance of users while targeting different server destinations. In this case, different popular media streaming websites (servers) are chosen as a destination landmarks for measurement from our MAs. The target server can influence the user’s Internet performance due to several reasons:

• The distance (network topological) between the MA and the target server. The closer the MA to the target server (less number of hops),

27 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

the better the performance.

• The technology used by the server (single server or CDN). If the server uses CDN technology, its performance will probably be more stable. But a single server’s performance will be highly inﬂuenced by a large number of visitors.

• The memory and processing capacity of the target server also has an inﬂuence on the performance. The better processing capacity the server has the better performance can be obtained.

It is also good to compare the measurements collected from different service providers. For example, the measurement results of two users from the same geographic location but with different service providers. We assume the target server for measurements use a CDN technology, both users are connected to the Internet through Ethernet cable, and both of them are located in the same area. If one of the users has higher delay and packet loss rate compared to the other one, it is highly probable the service provider to be the cause. During this master thesis we didn’t conduct experiment from home Internet users with different operators (service providers), but the service providers of SICS and the student accommodation are different operators. Finally, we want to observe, how the Internet performance or streaming servers behave during popular media events: sports, music contests, etc. To see whether Internet users experience more during big sport games than any other normal days on video buffering (i.e. pauses). The streaming servers become very busy serving a huge number of customers. The performance experienced by the Internet user during such events depends on the network and how well the server handles the requests and what technology it supports. Later in this chapter, we are going to discuss the measurement results obtained from some popular media events.

5.2 Popular media events

We were lucky during this master’s thesis, quite a few popular media events happened such as: a boxing match between Mayweather and Pacquiao ("Fight of the century"), Eurovision 2015, the Champions League ﬁnal between Barcelona and Juventus, and an Open House event at SICS with more than 400 guests (results in appendix). Apart from SICS Open House, all the popular media event measurements were not conducted with the standard CheesePi tools. As it was explained in section 4.7 we designed CheesePi to sample the network

28 5.2. POPULAR MEDIA EVENTS performance by conducting small number of measurements, but in the popular media events the measurements were continuous with high frequency (1200 probes per hour), and we were improving the way of measurements based on the lesson learned from the previous media event. So we decided to conduct separate measurements for the popular media events and at the end to import the results to CheesePi database (this might not be the ideal approach but I didn’t want to miss the measurements). Some of the more interesting results of the events will be presented in this section.

5.2.1 Boxing - fight of the century On the 3rd of May 2015 a boxing match between Mayweather and Pacquiao happened at Las Vegas. The match started at 04:00 UTC and it took 12 rounds, which lasted around 1 hour. The media call it Fight of the Century, reports indicate that Mayweather vs. Pacquiao was the most-watched Pay Per View (PPV) event of all time [16]. For those who live in USA it was costing $89.95 to order "Mayweather vs. Pacquiao" and an additional $10 for the high- definition (HD) experience [17]. In such matches major providers struggle to handle the load of last minute demand, and this was a good opportunity for us to measure the network behavior. After we heard about this event the next task was to choose which streaming servers should be measured. We choose five landmarks among the listed streaming websites by a promotional article in The Guardian, and the reasons for selecting the landmarks can be explained as follows:

• www.hbo.com

– Home Box Oﬃce (HBO) was the organizer of the event. – One of the top American premium cable and satellite television network.

• www.sho.com

– Showtime (SHO) is one of the top American premium cable and satellite television network with large number of subscribers.

• www.philstar.com

– Pacquiao (one of the boxers) is from Philippines, and Philstar is a Philippine news and entertainment portal for the Filipino global community

• my.sky.com

29 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

– UK’s largest pay-TV broadcaster with 11 million customers as of 2015.

• www.viaplay.se

– This thesis is conducted in Stockholm, Sweden and Viaplay is a major broadcaster in Sweden.

The measurement was conducted at Stockholm in a student apartment and SICS. The measurement in most of the MAs started around 2015-05-02 23:10 UTC and ended around 2015-05-03 21:43 UTC. The ping packet size was the default 64 bytes, but the size of the httping probes were diﬀerent for all the landmarks based on server replies for the HTTP head request. During the measurement the httping packet size for each target was as follows:

• www.hbo.com (477 bytes).

• www.sho.com (297 bytes).

• www.philstar.com (569 bytes).

• my.sky.com (165 bytes).

• www.viaplay.se (286 bytes).

During this measurement, we were not checking whether the servers are CDN technology enabled or not. But after some event measurements we started to consider CDN as one of the selecting criteria for the target. We were using CDN Finder http://www.cdnplanet.com/tools/cdnﬁnder/ as a tool to check if the target uses CDN or not (i.e. single server). According to CDN ﬁnder and MaxMind https://www.maxmind.com/en/geoip-demo the location of the target servers and their CDN supportability is listed as follows:

• www.hbo.com is located in Boardman, Oregon, United States, North America, and no CDN technology was found behind the hostname.

• www.sho.com located in Amsterdam, North Holland, Netherlands, Eu- rope and Akamai, a leading CDN service provider, was running behind the hostname.

• www.philstar.com located in Tempe, Arizona, United States, North Amer- ica and no CDN technology was found behind the hostname.

30 5.2. POPULAR MEDIA EVENTS

• my.sky.com located in Amsterdam, North Holland, Netherlands, Europe and Akamai is running behind the hostname.

• www.viaplay.se located in Dublin, Dublin City, Leinster, Ireland, Europe and no CDN technology was found behind the hostname.

Among the ﬁve targets, the HBO and Viaplay servers were not responding to ping requests. So for those two servers, we only conduct httping measurement. It would have been desirable if we could have run ping tests speciﬁcally for HBO, since it was the organizer of the match , and probably has the most customers (at least in the US).

Figure 5.1: HTTP latency to diﬀerent targets from SICS Ethernet during the boxing match between Mayweather and Pacquiao.

Except for HBO, the HTTP latency from SICS Ethernet to all the targets did not behave diﬀerently during the match (4-5 AM). The average HTTP latency from SICS Ethernet to HBO before 04:27 AM was around 461.6 ms, but after 04:27 till the end of the measurement there were only 21 probes with RTT of less than 500 ms and the average RTT was 580.9 ms, and this is clearly shown in Figure 5.1. This latency increase is observed in all MAs. As we can see in table 5.1 the average HTTP latency to HBO was 479, 582, 515 ms in SICS Ethernet, student apartment WiFi, and student apartment Ethernet respectively. After the match began, in all MAs the latency started to

31 CHAPTER 5. NETWORK PERFORMANCE EVALUATION increase at the same time. After the match began, the average HTTP latency increased by 21.6%, 22.5%, and 38.6% for SICS Ethernet, student apartment WiFi, and student apartment Ethernet respectively and it remained almost constant until the end of the measurement. This latency increase can be related to diﬀerent causes. It could be due to a path change or a congested router on the path creating the delay. It can be also due to the delay created by load on the target server. It was my mistake not conducting traceroute measurement during this event, and this creates complexity in ﬁnding out the source of the cause

RTT (ms) Be- RTT (ms) Dur- RTT (ms) After fore the match ing the match the match (05:00 (21:00 - 04:00) (04:00 -05:00) -22:00) Academic envi- 478 ± 86 582 ± 26 581 ± 65 ronment (SICS, Ethernet) Student apart- 582 ± 364 713.371 ± 86 717 ± 276 ment (WiFi) Student apart- 515 ± 141 714 ± 22 714 ± 104 ment (Ethernet)

Table 5.1: HTTP latency to HBO from diﬀerent MAs during boxing match between Mayweather and Pacquiao.

In order to observe if the boxing had an effect on the obtained performance, I did the same measurement one week after the match even though I repeated the same mistake by not including traceroute measurements. But as it is shown in Table 5.2 and Figure 5.2, unlike the average latency to HBO during the match day, the average latency to HBO one week later was almost similar in the time before, during, and after the match, and the latency was also less variable. This tells us the source of the latency increase that happened during the match day can be the packet filtering measurement taken by HBO, in order not to disturb the quality experience of the subscribers (it is a personal guess). The HTTP and network latency to Sky and SHO were much smaller compared to the other targets. A traceroute taken after this measurement showed the SHO and Sky servers were located closer to the MAs; the number of hops in order to reach the destination was less compared to the other targets. The HTTP latency from SICS Ethernet to Philstar started behave differ- ently around 14:00. The RTT value started to increase in the time between

32 5.2. POPULAR MEDIA EVENTS

Latency (ms) Latency (ms) Latency (ms) 21:00-04:00 04:00-05:00 05:00 -22:00 SICS Ethernet 498 ± 68 499 ± 28 493 ± 47

Table 5.2: HTTP latency to HBO from SICS Ethernet one week later after the boxing match between Mayweather and Pacquiao.

Figure 5.2: HTTP latency variability to HBO from SICS Ethernet during and one week later after the boxing match between Mayweather and Pacquiao.

14:38 and 18:00. Then later it stopped increasing and relatively stayed stable in the time between 18:00 and 20:28. Finally, the RTT value started to decrease util it reaches the average value before the change, and it happened between 20:28 and 21:28. Such behavior is observed in all MAs, and as it is indicated in Figure 5.3, a similar behavior is also reflected in the ping measurement. Since the behavior is also reflected in the ping result, the increased latency was likely due to the network. The Philippines time zone is UTC+8, so the local time at which the latency started to increase was approximately 22:00, which may be the time people usually start to watch TV. We conduct the same experiment one week after the fight to check if the observed behavior was due to the boxing. As it is shown in Table 5.3, unlike the Philstar latency results during the match the latency obtained one week later was relatively smaller and consistent, especially in the time between. So the latency increased to Philstar during the match day could be the congestion created on the links before the

33 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

Figure 5.3: Network and HTTP latency to Philstar from SICS Ethernet during the boxing match between Mayweather and Pacquiao. target server due to the replay of the match or there was some other live TV program.

HTTP la- HTTP la- Network la- Network latency (ms) tency (ms) tency (ms) tency (ms) (23:00-14:00) (14:00- 22:00) (23:00-14:00) (14:00- 22:00) During the 374 ± 87 484 ± 155 175 ± 8 227 ± 37 match One week 352 ± 37 352 ± 48 170 ± 0.5 170 ± 0.7 later

Table 5.3: Network and HTTP latency to Philstar from SICS Ethernet during and one week after the boxing match between Mayweather and Pacquiao.

The measurement results for HBO in Table 5.1 showed that the latency before the match from the Student apartment connected by a WiFi is higher and greater variance than the latency from the MA in student apartment connected via a cable. And it is similar for all the other targets. The laptops and smartphones of the two students who live in the apartment were connected

34 5.2. POPULAR MEDIA EVENTS to the Internet through WiFi. The contention and channel interference in the WiFi network likely created a higher latency compared to the Ethernet connection.

HBO SHO Sky Philstar ViaPlay SICS Eth- 544 ± 87 79 ± 130 24 ± 62 418 ± 128 113 ± 70 ernet Student 649 ± 148 75 ± 105 19 ± 86 351 ± 227 105 ± 71 appr. Ethernet Student 668 ± 313 108 ± 233 48 ± 221 No data 144 ± 232 appr. collected WiFi

Table 5.4: HTTP latency comparison among diﬀerent MAs during boxing match between Mayweather and Pacquiao.

In the boxing event we observed, the CDN enabled streaming websites are rarely aﬀected by the match. While those which are not CDN enabled, showed diﬀerent characteristics during the match day. The MA from the student Ethernet experienced better latency than the MA connected by WiFi. The network latency from the wirelessly connected MA was 22.8% larger than the latency of the MA connected by cable, and also less variable.

5.2.2 Eurovision 2015 The next big event we found was Eurovision song contest. Eurovision is the longest running annual TV song competition held, primarily, among the member countries of the European Broadcasting Union (EBU) since 1956. It is also one of the most watched non-sporting events in the world. For example Eu- rovision 2015 was seen by around 197 million viewers [18]. Vienna was the host of Eurovision 2015 and the contest was held in 3 stages (semi-final one, semi-final two, and grand final) on 19, 21 and 23 of May 2015, respectively. In this event, we measured from different places in order to get a variety of locations and access technologies. One good aspect of this event was the three stages of the contest, we got a chance to adapt and rethink the measurement approach for the grand final contest. For example, the HTTP was crashing during the continuous high frequency measurements in the first and second semi-final contests, and so were were able to follow a better approach for the final contest.

35 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

5.2.2.1 Eurovision 2015 semi final one The Eurovision 2015 semi-final one was on 19th of May 2015. It started at 19:00 UTC and ended at 21:00 UTC. In most of the MAs the measurements started and ended at 17:00 and 23:59 UTC respectively. As it was mentioned before the measurement was conducted from different places (team member’s apartment and SICS) and the event was not during the working hours. So there was a difference in the starting time of the measurement, but it does not affect the analysis since all measurement nodes cover the whole events. Choosing targets for the measurement is tricky because it was not easy to identify which website users would use to watch the event. We tried to look for popular streaming websites of many participant countries, and also asked friends to get suggestions and check the popularity of the chosen targets. The semi-final one measurement targets were the following:

• www.eurovision.tv - a European broadcasting union’s streaming website for Eurovision. We choose this target due to its popularity and it was also an organizer of the event.

• www.svtplay.se - SVT (Sveriges Television AB), a Swedish public TV broadcaster. The measurement is conducted in Stockholm (Sweden), and Eurovision is very popular in Sweden. So we decide to include a popular streaming website which broadcast Eurovision in our targets.

• tvthek.orf.at - Osterreichischer Rundfunk (ORF) is an Austrian national public broadcaster. The reason we chose the target was: its dominance in Austrian broadcasting media and Austria was also the host of the event.

In order to compare and analyse the results, it is good to know the location of the target servers and the technology they support. The location of the target servers and its CDN technology supportability was as follows:

• www.eurovision.tv - CloudFlare, one of the top CDN providers, distributes the contents of the target. The target server during the measurement was located in Stockholm, Sweden.

• www.svtplay.se - Akamai distributes the content of SVT and the target server during the event was located in Stockholm, Sweden.

• tvthek.orf.at - According to CDN Finder and MaxMind, ORF does not use CDN technology to distribute its content and its server is located in Austria.

36 5.2. POPULAR MEDIA EVENTS

The ping and httping were running in parallel and the interval between each probe was 3 seconds. The packet size of a ping probe for each target was 64 bytes, and 479, 206, and 477 bytes for each httping probe of EurovisionTV, SVT, and ORF respectively.

Figure 5.4: HTTP latency to diﬀerent targets from an MA at SICS (Ethernet) during Eurovision 2015 semi-ﬁnal one.

From the previous boxing event, we observed that performance from the Ethernet connection was better than the performance from the WiFi connection. So we decided to first start looking at the results for each target from the MA at SICS connected to the Internet via a cable. As it is shown in Figure 5.4 and Table 5.5 the HTTP delay of SVT was consistently small between 6 to 8 ms with a small variance but there were very few large ( > 205 ms ) delays during the contest in the time between 20:00 and 21:00, and after 23:00. The latency of ORF and EurovisionTV vary a lot especially ORF. From the SVT delay results we can understand that the CDN server performance was good, there was no congestion in the path between the MA and the target. Next we are going to look at EurovisionTV and ORF results. The HTTP delay of EurovisionTV from SICS Ethernet started to vary many times from 17:20 till the end of the contest (21:00) as it is clearly shown in Figure 5.5 and Table 5.5. This indicates that the event affected the HTTP delay to EurovisionTV to some extent. But we can not be certain the increase in the RTT value was due to the delay created by the server or traffic created on

37 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

Eurovision SVT ORF Mean Med. Stdv. Mean Med. Stdv. Mean Med. Stdv. Before 23.87 8.44 203 7.83 6.43 5.68 181 94 865 Program (17:00 - 19:00) During 23.75 8.31 241 17 8.09 205 147 94 591 Program (19:00 - 21:00) After 8.35 7.97 1.7 11 7.84 93 370 93 1772 Program (21:00 - 00:00) The whole 17.21 8.17 169 12 7.67 125 251 94 1287 Mea- surement (17:00 - 00:00) Table 5.5: Comparison of HTTP latency from SICS Ethernet to diﬀerent targets before, during, and after the program of Eurovision semi-ﬁnal one. the network. So to get an idea about the causes we should compare the network delay with the HTTP delay (network and server’s computation latency).

38 5.2. POPULAR MEDIA EVENTS

Figure 5.5: HTTP latency from a MA at SICS (Ethernet) to EurovisionTV during Eurovision 2015 semi-ﬁnal one.

Figure 5.6: Comparison of network and HTTP delays from a MA at SICS (Ethernet) to EurovisionTV during Eurovision 2015 semi-ﬁnal one.

39 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

The network delay was 1.84 ms with a tiny variation of 0.53 ms, as it is clearly shown in Figure 5.6 the delay was small and consistent. The larger network delays mostly happened a few minutes after the contest started with peaks 44.44% larger than the average network delay during the whole measurement. While the HTTP delay was varying before and during the contest. As it is shown in Figure 5.6 and Table 5.5 the deviation before and during the contest was around 10 times the average delay. Many of the peaks were more than ﬁve times the average latency of the httping measurement to the target. So the results indicate the large HTTP delays were due to the delay created in the server. But what about the large HTTP delay variance to ORF?

Figure 5.7: HTTP latency from a MA at SICS (Ethernet) to ORF during Eurovision 2015 semi-ﬁnal one.

There are many servers which do not respond to a ping (ICMP) request, and ORF’s server is one of these servers. It is diﬃcult to draw a conclusion based only on HTTP measurements. But since httping results are the performance of the whole TCP/IP stack, it is possible to have a general understanding of the connection’s performance. As one can see in Figure 5.7 the HTTP delay to ORF are peculiar. It is common for the latency to increase and ﬂuctuate a lot while watching a popular media event from a website which uses a single server to handle all the user requests. But the average HTTP latency (147 ms) and its variance (591) during the contest was relatively better compared to the latency and variance before (181 ms, 865 ms) and after

40 5.2. POPULAR MEDIA EVENTS

(370 ms, 1772 ms) the contest. When we looked at the path based measurement results to find the cause, we found that, the IP address of the target was changing between 194.232.116.172-176. During this measurement we did not include "-r" parameter in the httping command, which makes it only resolve the hostname once at the start. This tells us we were measuring five different ORF servers and this might be the cause of the frequent and large latency variation. So ORF was using several servers for load balancing to handle user request, but still it does not save users from experiencing larger delays for some moments.

Figure 5.8: Comparison of HTTP latency to EurovisionTV from diﬀerent MAs during Eurovision 2015 semi-ﬁnal one.

We compared the latency experienced by different MAs. The HTTP delays to SVT and ORF experienced by the MAs in SICS and the student accommodations are more or less similar except some higher delays for the wirelessly connected MAs. But as one can see in Figure 5.8 the HTTP latency experienced by SICS and the student accommodations is different. The middle subplot of Figure 5.8 shows, the HTTP delay from both the students apartment have a similar pattern and an average latency. This is due to both MAs served by the same operator, and after the second hop the packets follow a similar path to reach the target. However, there is a big latency difference between the results obtained from SICS and the student apartment. Even though the traceroute shows both of them have almost similar number of hops, SICS

41 CHAPTER 5. NETWORK PERFORMANCE EVALUATION uses Nordnet network which is directly connected to the designated edge CDN server (Stockholm), whereas due to the students accommodation ISP routing policy, the packets from the student apartment were leaving Sweden network to Norway and came back to Sweden network. So the main reason for the large latency difference from the student apartment and SICS was the delay created by the path followed to reach the target. The next step was to look how the event affected the latency in the student apartment and comparing the latency between the WiFi and cable connection. The average HTTP delay to Eurovision from both the Ethernet and WiFi connected MAs in the time from the beginning of the measurement till the end for the contest (17:00 - 21:00) was 311±446 and 324±396 ms respectively. This latency is much higher and deviates more compared to the average HTTP delay experienced after the contest (21:00 - 00:00) by both the Ethernet and WiFi connected MAs, which is 90 ± 89 and 109 ± 133 ms respectively. This behavior is clearly shown in Figure 5.8. Such pattern is also reflected in the network measurement, so this indicates there was higher traffic in the network during and few hours before the contest. We will see in the next contests if this latency variation was due to the Eurovision effect on the Telenor network that goes to the target. From the presented results the Ethernet connection in the student apartment was as bad as the WiFi in terms of delay and deviation.

5.2.2.2 Eurovision semi final two The Eurovision 2015 semi-final two was held two days after the semi-final one on 21st of May 2015. It started at 19:00 UTC and ended at 21:00 UTC. In most of the MAs, the measurements started and ended at 16:05 and 23:05 UTC respectively. During this contest, we add one more MA which used 3G to connect to the Internet. All the targets were the same targets as the semi-final one.

42 5.2. POPULAR MEDIA EVENTS

Figure 5.9: Comparison of HTTP latency to SVT among diﬀerent MAs during Eurovision Semi-ﬁnal two.

Before the contest, the average HTTP delay to SVT from the student accommodation in Kista with an Ethernet connection was 16 ± 320 ms, the large deviation is due to a single large delay that happened in 18:19 which was around 19 seconds. The average HTTP delay of the SICS Ethernet was 15.4± 171 ms which is almost similar to the student accommodation. But in the student accommodation during the contest at around 19:54 the HTTP delay increased by large amount and continued like that till 20:38, the average delay during that time was 215±115 ms. As one can see in Figure 5.9, such behavior was also observed in the WiFi connection of the student apartment. Similar case but at diﬀerent time also happened in another student accommodation in Lappis, which started at 19:12 and ended at 19:36. In all the MAs, the cause for the delay increase was a continuous 6 packet loss, and after that the target IP address changed and the latency increased by more than 15 times. The average HTTP delay from the 3G connection was three times of the Ethernet connection delay in the student apartment.

43 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

Figure 5.10: Comparison of HTTP latency to EurovisionTV among diﬀerent MAs during Eurovision Semi-ﬁnal two.

In the MAs at the student accommodation the HTTP delay to Eurovi- sionTV started to increase when the contest was about to begin and it started decreasing after the end of the contest and becomes relatively stable after 22:00, which is similar to Eurovision semi-ﬁnal one. During the HTTP delay measurement to EurovisionTV, the MA with a 3G connection experienced a better average delay and deviation (85 ± 115ms) compared to the MA at the student accommodation (148 ± 194 ms). This is clearly shown in Figure 5.10. The latency increase pattern observed in both Eurovision semi-ﬁnal one and two in the measurements from the student accommodation to EurovisionTV will be analysed later in the next sections.

5.2.2.3 Eurovision 2015 ﬁnal

The Eurovision 2015 grand ﬁnal was held on 23rd of May 2015. The program started at 19:00 UTC and ended at 23:00 UTC. The voting started at 21:14 and ended at 21:40 UTC. The result announcement started at 21:52 and ended at 22:49 UTC. In most of the MAs, the measurements started and ended at 16:30 and 04:00 respectively. During this contest we wanted to make some changes in the targets due to ORF website was not streaming the grand ﬁnal, and both EurovisionTV and

44 5.2. POPULAR MEDIA EVENTS

SVT were using CDN technology. Thus we decided to remove ORF and add more targets, for example, SBS (Special Broadcasting Service), an Australian public broadcasting, and the BBC’s iPlayer. We added SBS because it was the ﬁrst time for Australia to participate in Eurovision and SBS is one of the top broadcasting services in Australia. We dedicated a MA at SICS with an Ethernet connection, to measure the performance by probing to SBS, BBC iPlayer, and BBC UK. The other MAs were probing to SVT, BBC UK and EurovisionTV.

Figure 5.11: Network and HTTP latency of diﬀerent targets form SICS Eth- ernet during Eurovision 2015 ﬁnal.

As one can see in Figure 5.11, except the large HTTP delay (5042 ms) at 21:12, the network and the HTTP delay to BBC iPlayer from the SICS Ethernet was consistent, which is 9 ± 46 ms and 1.6 ± 0.3 ms average HTTP and network delays respectively. It was also similar for the SVT measurement but when the contest started (21:00 - 21:10) ﬁve large delays with an RTT value more than 5 seconds are recorded. This might be related to large number of users request, related with the beginning of the voting. The HTTP delay to EuroVision (8 ± 21 ms) was deviating less after 21:00 compared to the measurement before 21:00 (16 ± 158 ms).

45 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

Figure 5.12: Network and HTTP latency of diﬀerent targets form SICS Eth- ernet during Eurovision 2015 ﬁnal.

The SBS server was located in Australia and the network and HTTP delay was large 356±3 ms and 713±182 ms respectively. Out of 12600 httping probes to SBS 90 of them recorded with more than 2000 ms delay. No special pattern was observed during the contest, except 3 probes with more than 5 second HTTP delay. But this large delay can affect the quality of different online applications. In the measurements to SBS we observed, the farther the target is from the client in the network topology the more the performance degraded. We also compared the latency between BBC UK and the BBC iPlayer. The HTTP delay of BBC iPlayer (12 ± 111 ms) was smaller compared to BBC UK (78 ± 65 ms). The small delay of BBC iPlayer is due to it uses Amazon CloudFront for distributing its content. This is clearly shown in Figure 5.12. As one can see in Figures 5.13 and 5.14 the HTTP delay to BBC iPlayer and EurovisionTV from the student apartment is higher than the HTTP delay from SICS. As we previously discussed in Eurovision semi-final one, it is due to the routing policy of the service providers. In the SICS network the DNS server returns the Amazon CloudFront server nearest to the MA and the packets were reaching the target after few hops. However the DNS server for the student accommodation was returning different Amazon CloudFront server, which is located in Amsterdam (according to the DB-IP Database) which can not be reached close to the MA. From this delay comparison we

46 5.2. POPULAR MEDIA EVENTS can understand that the routing policy of the service provider or its agreement with CDN providers aﬀects the user’s Internet performance.

Figure 5.13: Comparison of HTTP delay to EurovisionTV from diﬀerent MAs during Eurovision 2015 ﬁnal.

However the HTTP delay to SVT from the student apartment (16 ± 102 ms) is similar to the HTTP delay from SICS (14 ± 171 ms). In this case, even though both the DNS servers in the student accommodation and SICS network return diﬀerent CDN servers and did not share common path to reach the target, the probes from both networks are able to reach the target quickly. This might be due to 108,507 Akamai servers deployed worldwide in over 110 countries and within more than 1,400 networks around the world [35]. The distribution of the CDN servers responsible for distributing the streaming websites content and the agreement between the ISP (operator) and the CDN provider inﬂuence the Internet performance of a user.

47 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

Figure 5.14: Comparison of HTTP delay to BBC iPlayer from diﬀerent MAs during Eurovision 2015 ﬁnal.

5.2.2.4 One week after the Eurovision 2015 ﬁnal

In order to understand the effects of the popular media event on the performance of the Internet quality, we repeated the experiment exactly one week after the Eurovision 2015 final to observe if some of the performance changes during the measurement were due to the media event. The time the measurement were conducted, the places, the targets, and the frequency of the measurement were the same as during the Eurovision 2015 final. The red graphs in the top part of Figure 5.15 are the HTTP delay of different targets from the SICS Ethernet connection during Eurovision 2015 final, while the bottom measurement graphs in blue are the HTTP delay to the targets after one week. During the Eurovision final the average HTTP delay to SVT was (14 ± 171 ms), the HTTP delay of 87.12% of the probes were less than 10 ms, and for the measurement conducted one week after 87.85% of the probes recorded less than 10 ms HTTP delay. There were few peak points observed around the beginning of the program (19:00) and voting time (21:15) which did not happen in the measurement conducted after one week. So the graph tells us the large latency of the few probes may indeed be due to the load on the web server or the link due to large number of users around the mentioned time. But the average latency in both measurements are almost

48 5.2. POPULAR MEDIA EVENTS similar, which is 14 ± 171 ms and 11 ± 60 ms for the Eurovision ﬁnal and one week after measurements respectively.

Figure 5.15: Comparison of HTTP delay from SICS Ethernet to different targets during Eurovision 2015 final and one week after the final.

The average HTTP delay to BBC iPlayer from SICS Ethernet during the Eurovision Final was 10±46 ms. However, when the voting was about to start there was a peak with a latency around 250 ms and we initially thought it was due to an increase in number of visitors at that time. In the measurement conducted one week later, until 18:07 the average HTTP delay was 11.9 ms but suddenly it increased to an average of 220 ms and stayed for about 2 hours till 20:03. We try to analyse the reason behind the sudden latency increase. First we checked if it was due to continuous packet drop or a target IP address change, but there was no packet loss and the IP was changing frequently before the latency started to increase. When we checked the traceroute, the path was changed at the moment the latency started to increase. During the smaller latency the HTTP packets were reaching the target after few hops with a very small RTT. However during the higher latency period the number of hops increased because it was following a diﬀerent route. When the path to the target returned to the previous path the latency decreased closer to its previous average delay (26 ms). This artifact is clearly shown in the bottom right plot of Figure 5.15. The cause of the sudden latency increase may be due

49 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

(a) From SICS Ethernet. (b) From Student apartment WiFi.

Figure 5.16: Comparison of HTTP latency to EuroVisionTV from student apartment WiFi and SICS Ethernet between the measurements during Euro- vision ﬁnal and one week after the ﬁnal.

to target IP address changing its load balancing. The new path for reaching the target requires more time and affects the end to end delay. The HTTP latency from SICS Ethernet to EurovisionTV during the Eu- rovision final has fewer peaks than the experiment conducted one week after the final. For example, in the measurement during the Eurovision final there were 29 HTTP probes recorded with delay greater than 100 ms, whereas one week later there were 60 probes with HTTP delay of more than 100 ms. Dur- ing the Eurovision final measurement we thought the high latencies were due to users visiting EurovisionTV to check the programs and watch the contest at the time before the contest started and as the program started. But the measurement that conducted one week after Eurovision final shows the high latency were probably not due to the event. The high latency in the time between 16:30 and 22:00 can be due to network congestion or the CDN server is relatively busy everyday during the given time interval. In order to determine the cause of the behavior we should look at the network latency from SICS to EurovisionTV and other MAs’ HTTP latency. This helps to check if the behavior (pattern) is due to the characteristics of the network, or the target server. As one can see in Figure 5.16 (a), the network latency to Eurovision TV from SICS Ethernet during Eurovision final and one week later showed low variation, as the coefficient of variation (the standard deviation / mean) is less than 1, a thumb rule for the variation in dimensionless values, given the measured values of 1.77 ± 0.4 ms and 1.72 ± 0.28 ms respectively. So it is unlikely the network was the cause of the previously observed large HTTP

50 5.2. POPULAR MEDIA EVENTS delays. But if we look at HTTP latency from student apartment (WiFi) to EurovisionTV, in both measurements the latency is higher in the time between 16:30 to 22:00. As it is clearly shown in Figure 5.17 that behavior is observed in all EuroVision measurements. So this higher latency in the time between 16:30 and 22:00 could be due to the load on the CDN server everyday during the speciﬁed time, or it may give a priority to other tasks instead of HTTP head request packets.

Figure 5.17: HTTP latency from student apartment to EurovisionTV during diﬀerent measurements.

5.2.3 2015 UEFA Champions League final The UEFA Champions League, known simply as the Champions League, is an annual continental club football competition organised by the Union of Euro- pean Football Associations (UEFA) and contested by top-division European clubs. The 2015 UEFA Champions League final was the final match of the 2014-15 UEFA Champions League, and it was played at the Olympiastadion in Berlin, Germany, on 6 June 2015, between the Italian side Juventus and the Spanish side Barcelona. A massive global audience on TV and social media confirmed the UEFA Champions League final’s status as the world’s most watched annual sporting event [19]. We thought this popular media event

51 CHAPTER 5. NETWORK PERFORMANCE EVALUATION would create a behavioral change in the Internet quality of users, and we conducted measurements during the event to observe how it aﬀects the Internet performance of users. It started at 18:45 UTC and ended at 20:45 UTC. And the trophy cer- emony ended at 21:00 UTC. In most of the MAs the measurements started and ended at 16:00 and 04:00 (the next day) respectively. From the previous measurements we learned that the targets which use a CDN technology are less aﬀected by popular media events. So during this measurement we tried to focus more on popular free streaming targets which do not use CDN technology. And we selected four targets: one subscription based websites, and three free streaming websites which do not use CDN technology. Unfortunately we realized later two of the free streaming targets were using CloudFlare to distribute their content. The chosen targets were the following:

• www.espnplayer.com - ESPN (entertainment and sports programming network) is a U.S.-based global cable and satellite television channel owned by ESPN Inc. We choose ESPN as our target because as of February 2015, ESPN has approximately 94,396,000 subscribers, and it was one of the Champions League ﬁnal broadcasters. The streaming service crash during most of the world cup 2014 watched live streaming [21] was another reason for choosing ESPN. It uses Akamai to distribute the content on large live events but during the Champions League ﬁnal the target server was not an Akamai server. The server was located in USA.

• www.cricfree.sx - CRiCFREE is a popular free sport streaming portal website. According to [20] it has millions of visitors every month. We chose the target to observe how free streaming websites handle user requests during popular media events. It uses CloudFlare to distribute its content and the closest server was in UK.

• www.crichd.in - CricHD is also a poplular free sport streaming portal. After we conducted the measurement, we realized it is another CRiCFREE with diﬀerent domain name with a small changes in the front page of the website.

• www.atdhe.ws - ATDHE is a another popular free sport streaming portal. It doesn’t use a CDN technology to distribute the content and the target server is located around Netherlands.

In some of the previous httping measurements we faced diﬃculties with some conclusions due to frequent IP address change of the target. During

52 5.2. POPULAR MEDIA EVENTS this measurement we include the "resolve hostname once" parameter in the httping command, so the measurement was only to one web server. This helps the measurement to be more stable when compared to a measurement to a hostname with several IP addresses for load balancing. Based on the knowledge of networking and communication systems, when large number of users visit a specific server at the same time, the server becomes busy handling all the requests and creates high traffic over the link to the target. So during a popular media events we expect there will be higher latency and packet loss during the match. Now let’s look at the result obtained from different MAs during the match.

Figure 5.18: Network, HTTP, and network packet loss relation of measurement from SICS Ethernet to ATDHE during UEFA 2015 Champions League ﬁnal.

First we should see how the Champions League final affected the performance measurement to ATDHE. Before we look on all MAs, let’s look at the results collected from SICS Ethernet. Figure 5.18 shows the latency of the HTTP and the network, and its relations to the ping packet loss. The red line graph in the figure represents the HTTP latency, the blue graph in the middle subplot represents the network latency, and the green graph in the bottom subplot of the figure represents the network packet loss rate in every five minute across the whole measurement. The HTTP delay was 94 ± 19 ms. However when the match was about to start at 18:22, the HTTP delay variance increased (104 ± 84 ms) and continued until 20:30. The coefficient

53 CHAPTER 5. NETWORK PERFORMANCE EVALUATION of variance for these two events are 0.2 and 0.8 respectively, which again captures, somewhat simply, the change in the experienced network service. This was just before and during the match, which is similar to our expectation. Then it went back to its previous "state" (94 ± 36 ms), and there were few spikes between 21:30 and 22:30 as one can see in Figure 5.18. Now, let’s look at the network latency and packet loss to understand the cause of the latency (change). A network latency of 44.7 ± 0.9 ms throughout the whole measurement phase except for four probes with delays between 100 and 104 ms, could be taken as "stable". There were also few ping (network) packet losses throughout the whole measurement. From the results we understand the latency increase during the match are due to the latency created by the server, because the behavior observed in HTTP delay measurement was not reflected in the network measurements. So this can be due to large number of visitors to the ATDHE website during the Champions League final, and this load on the single server created the HTTP delay increase. This artifact is clearly shown in the top subplot of Figure 5.18. From the previous measurements we observed that some behavioral patterns were observed only on specific Networks (MAs). So let’s look if the HTTP delay to ATDHE was larger during the match in all other MAs. As it is shown in the blue graph of the top subplot in Figure 5.19, the SICS WiFi also experienced higher latencies (110 ± 142 ms) around the time the match took place from 18:35 - 20:45. Whereas, the average HTTP delay before the match (16:00 - 18:35) was (96 ± 28 ms). The WiFi and Ethernet connection of the student apartment also experienced higher HTTP delay during the match but there were also latency spikes every few minutes. During the measurement to ATDHE, the student apartment experienced lower HTTP delay (59 ± 93 ms) compared to the delay from SICS (96 ± 48 ms). According to the traceroute results, the student apartment packets were passing through fewer hops to reach ATDHE server compared to the packets from SICS. But as it is shown in the middle and bottom subplots of Figure 5.19 the HTTP delay from the student apartment was deviating more both in the WiFi and the Ethernet environments, especially during the match. For example, during time 18:35 - 21:00 the HTTP delay was 79 ± 87 ms and 65 ± 118 ms for the WiFi and Ethernet respectively. We doubt this latency variation was due to the network congestion on the path, and thus we decided to look the latency of the network from the student apartment and SICS.

54 5.2. POPULAR MEDIA EVENTS

Figure 5.19: HTTP latency of diﬀerent MAs to ATDHE during UEFA 2015 Champions League ﬁnal.

The ﬁrst subplot of Figure 5.20 shows the network latency from SICS WiFi and Ethernet to ATDHE. The network latency of the Ethernet was very smooth (44 ± 0.9 ms), while the latency from the WiFi had few larger delays (47 ± 10 ms). This tells us the latency on the network was very stable while there were some delay increase during the match and due to the WiFi. The network results tell us, the larger HTTP delay during the match was not due the network congestion, it is due to the load on the single ATDHE server. As shown in the middle and bottom subplots of Figure 5.20, the student apartment experienced less network latency than SICS but there were a lot of spikes throughout the whole measurement, especially from the WiFi. These spikes from the WiFi might be due to high wireless-medium contention. Two students were watching the match and a movie using wirelessly connected laptops, and there were also two other smartphones connected to the Internet through the WiFi. The contention among wireless devices and the channel interference from neighbor routers, who use the same channel as the WiFi router of the student apartment, could be the cause of the spikes.

55 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

Figure 5.20: Network latency of diﬀerent MAs to ATDHE during UEFA 2015 Champions League ﬁnal.

We have examined how the Champions League ﬁnal aﬀected the measurement performance to a popular server which does not use CDN technology to distribute its content. What about the targets which use CDN technology to distribute their content and ESPN (supposed to use a CDN)?

56 5.2. POPULAR MEDIA EVENTS

Figure 5.21: HTTP latency of diﬀerent MAs to CricHD during UEFA 2015 Champions League ﬁnal.

Even though CricHD uses CDN technology the average HTTP delay from every MA was more than 200 ms. This is shown in all the subplots of Figure 5.21. The CDN server, which was handling the HTTP request to CricHD, was located in UK and the average network delay from all the MAs in Stockholm was around 40 ms. We did not expect a big difference among the network and the HTTP latencies since CricHD uses CDN (CloudFlare), however, the average latency was 160 ms more. These HTTP latencies were consistent and never went below 180 ms for any of the MAs. It can’t be due to the packet size (326 Bytes), because we tried the ping measurement with different packet size up to 526 Bytes but the difference was not significant. It seems there may have been some load balancing or reverse proxy behind the CDN server that makes the HTTP delay this much larger. Beside the consistent large HTTP latency from SICS to CricHD there were also some spikes a few minutes before and during the match. While the latency from student apartment to CricHD was higher (301 ± 461 ms) during the match (18:35 - 21:00) compared to the average HTTP delay (220 ± 363 ms). We try to figure out if the latency increase is due to the network or the target server. As it is shown in the middle and bottom blue subplots of Figure 5.22 the network latency to CricHD from the student apartment started to increase at around 17:00 and became higher (66 ± 13 ms) between

57 CHAPTER 5. NETWORK PERFORMANCE EVALUATION

18:35 and 21:20 (during the match) compared to the average network delay (42 ± 17 ms) and the delay after 22:55 (28 ± 1.5 ms). This tells us the larger delay observed in the HTTP performance measurements are influenced by the network performance. Then we tried to find which network link is creating the latency by looking at the traceroute results. The router in Norway "tiscali- 1.ti.telenor.net" was creating these larger delays. It might be because the router was busy at that time or there were some priority to other packets. When the latency to the "tiscali-1.ti.telenor.net" started to decrease and later became stable, the network latency to CricHD also became stable. So it seems a specific router on the path can strongly influence the end-to-end delay.

Figure 5.22: Network latency of diﬀerent MAs to CricHD during UEFA 2015 Champions League ﬁnal.

Some unofficial video streaming portals use different domain names with a small front page, designed to keep streaming if the main domain of the website is seized by intellectual property protection organizations. In our case, CRICFREE and CricHD looked the same, and providing similar services with a slight difference on their front page. We also trace the route to both targets, and they follow the same path till the last hop to the target. CRICFREE also uses CloudFlare to distribute the content, and the closest server to our MAs was located in the UK similar to CricHD. As it is shown in Figure 5.23 that the general behaviors observed during httping and ping measurements to CricHD also happened while measuring to CRICFREE. However, the average HTTP

58 5.2. POPULAR MEDIA EVENTS delay to CRICFREE was smaller compared to the CricHD HTTP delay. For example the average HTTP delay to CRICFREE from the student apartment with Ethernet connection ( 147±167 ms) was smaller than the average HTTP delay to CricHD (220±363 ms). In most of the MAs, the average HTTP delay to CRICFREE was less than 150 ms. This indicates, the delay created in the CRICFREE server was smaller compared to the delay created by CricHD server.

Figure 5.23: HTTP delay of diﬀerent MAs to CRICFREE during UEFA 2015 Champions League ﬁnal.

Even though ESPN is one of the top on-demand streaming services, the HTTP and the network delays were large. The HTTP and network delays from all MAs were very similar, and it was around 235 and 112 ms for the HTTP and the network respectively. Figure 5.24 shows, the HTTP delay from the SICS Ethernet and WiFi connections were 231 ± 70 ms and 237 ± 180 ms respectively. And in the student accommodation the HTTP delay was 229±87 ms and 246 ± 101 ms from the Ethernet and WiFi connections respectively. As one can observe, the deviation was larger in the wirelessly connected MAs. In the student apartment WiFi there were continuous periodic spikes. We tried to ﬁnd out the causes of the periodic spikes, and we tried to compare the HTTP and network latencies. As clearly shown in Figure 5.25 the periodic latency increase is in both the network and HTTP. This means, something

59 CHAPTER 5. NETWORK PERFORMANCE EVALUATION was periodically happening in the network. In order to find the cause we have to look the ping results carefully. When we looked the ping results the average network delay was around 113 ms before the periodic increase, but every multiple of 5 minutes the delay increased. For example, the RTT was going constantly around 113 ms then at 5th minute it became 500 ms and return to its previous state, then again after multiples of 5 minutes it increases. We tried to examine the ping measurement results to other targets from the student apartment with a WiFi connection. And the behavior is observed in all ping measurement of different targets. We did not observe such a behavior in the results from the Ethernet connection. So this periodical delay increase may be due to the periodic PSM (Power Saving Mode) of the WiFi dongle in the MA. When using PSM the wireless interface is put in a sleep mode when the device is not transmitting a data. If the Access Point (WiFi router) receives data of the client while the client was in a sleep mode, it buffers the data until it receives the next beacon from the client. So such periodic WiFi interface sleep mode of the MA can be the cause of the periodic large RTTs. And in a Raspberry Pi the PSM is enabled by a default.

Figure 5.24: HTTP latency of diﬀerent MAs to ESPN during UEFA 2015 Champions League ﬁnal.

One can only watch the match on ESPN if one is a subscribed customer. We think the measurements to ESPN may not represent the traﬃc during the

60 5.2. POPULAR MEDIA EVENTS match at that time. Because, our target was "www.espnplayer.com", which was located in US, but for a subscribed customer it would be diﬀerent server located closely. So we can not say the performance observed on the measurements to ESPN during the Champions League ﬁnal are the performance experienced by the customers.

Figure 5.25: HTTP and network latency to ESPN from student apartment WiFi during UEFA 2015 Champions League ﬁnal.

Chapter 6

Discussion

WiFi experiences medium contention and so can be slower and more variable than Ethernet. Our measurement results confirmed and quantified customers with an Ethernet connection experienced lower latency and packet loss compared to WiFi users. In most of the measurements the SICS achieved better performance in terms of latency compared to the student apartment. SICS uses SUNET, which enables Swedish universities and institutions to connect with a well-developed and effective national and international data communication and related services. And the student accommodation uses Bredbands- bolaget ISP owned by Telenor. Before the large media event measurement results were analyzed, we ex- pected that users who connect to a website on a CDN would receive data with a lower delay than a non-CDN solution. However our experiments showed the effectiveness of CDN technology on minimizing the end to end latency depends on how close the server is located to the customer, and the functioning of the Internet service provider’s routing policy (or possibly the CDNs). If the designated edge server is very close (in number of hops and geographi- cally), and the path chosen to reach the desired domain is short, the latency is low. But if the path chosen to reach the target is long, the latency will be highly dependent on the network status. Such behavior is observed during measurement to EurovisionTV from the SICS and student apartment during Eurovision. During our httping experiments to a CDN host, we observed a situation where significant packet loss occurred when the IP address of the target domain changed to a different edge server. This happened during the second Eurovision semi-final, as part of the measurements from a student apartment to Swedish Television (SVT). After six contiguous packet losses the edge target changed, which required a more hops to be reached by the measurement agents (MA). The HTTP delay increased by more than 15 times, and all the ping

63 CHAPTER 6. DISCUSSION packets were lost after the IP address change. The ping packets lost were due to the hostname only being resolved once at the beginning of the experiment. In the student accommodation (Kista), this large delay continued for more than 40 minutes until the target address returned to its previous (edge) CDN server. This show how changes made in the network have a great influence on the latency. The target IP address changed and caused communication to be routed to a different destination IP. Following the new path to communicate with the target may require more time and this affects the end to end delay. Such a similar case also happened during measurement to the BBC iPlayer, one week after the Eurovision final. And the average HTTP delay increased from 11.9 ms to 220 ms and this continued for approximately two hours. We can only speculate at this point on the reasons, possibly technical failure, possibly load balancing, but it would require further studies. It is common to observe some patterns in streaming websites with a single server. For example, if we do an HTTP measurement to a website which streams a popular TV show (e.g. "Breaking Bad") on Sunday between 21:00 - 21:50 using a single server, the HTTP delay on Sunday between 20:45 - 21:55 can be larger compared to the HTTP delays on a different time. The HTTP delay increased because of the greater load due to a large number of visitors during that time. But most of the popular streaming websites, which use CDNs, are capable of handling a large number of requests, and it is very rare to observe such delays. However during our measurement to EurovisionTV which uses CDN, during the time between 16:30 and 22:00 in all Eurovision measurements a higher HTTP delay (for example, 143 ± 136 ms during Eurovision final) was observed compared to the HTTP delay after 22:00 (for example, 64 ± 33 ms during Eurovision final). The pattern is not reflected in the ping measurement and it is most likely cause by the server load. Even though it is difficult to precisely know the source of such a pattern since a measurement is not conducted in the server-side, but it can be due to the high load on the CDN server everyday in the specified time or it may give priority to other tasks instead of HTTP head request packets. One of the main purposes of CheesePi is to collect data for operators to troubleshoot the source of problems in a network. During measurement to CRICHD in the 2015 UEFA Champions League final, there was a latency increase before and during the match. By tracing back the results from all the tools the source of the problem was identified as a specific router or the link that connects it to neighboring routers. The delay increase can be related to the congestion created on the link or other some reasons, but the job of CheesePi is to collect and quantify the observations of the problem. It is not easy to measure the exact performance of the Internet due to its complexity, but by following smart approaches and using the right tools,

64 measurements can characterise the performance experienced by a user. All the measurements conducted during this master thesis use ping, httping, and traceroute network tools to measure the performance of users’ Internet quality. However, it is difficult to make concrete conclusions on some situations based only on the results obtained from ping, httping, and traceroute measurements. The tools selection and their output data defines the possible conclusions that can be drawn. More advanced tools will allow more advanced conclusions. For example, to observe the behavior of TCP we could use a tool such as tcpdump. The longer HTTP delays, likely caused by packet loss being fixed by TCP re- transmissions would be exposed by this tool. In some of the popular media event measurements the httping was crashing several times when an error occurred. During such measurements it became a challenge not to have a complete results from some MAs.

Chapter 7

Conclusions

We have developed CheesePi, a distributed measurement platform which helps users to characterise their Internet experience. The CheesePi measurement agent runs on a Raspberry Pi connected to a home router. The measurement agent conducts ICMP, HTTP, traceroute and VoIP measurements periodically. Additionally, by increasing the number and distribution of the measurement agents the network can be characterized better. A dashboard which is locally hosted on the Raspberry Pi, is used to visualise the experienced connection behavior. The measurement results show customers with an Ethernet connection experienced better latency and packet loss compared to WiFi users. For example, the average network delay for all targets during Eurovision final from SICS WiFi was 4 ms and was more than double the average network delay from an Ethernet connection (1.5 ms). And the average network delay for all targets during Eurovision final from the student accommodation WiFi was 1.4 times than the average network delay from the Ethernet connection. In all the Eurovision measurements, the student apartment experienced signifi- cantly (26-57 times) worse performance in terms of latency compared to the SICS site. The relatively larger network delays from the student accommodation during Eurovision were for targets which use CloudFlare CDN service, and this might be the traffic carrying agreement between the student’s accommodation ISP and other ISPs. ISPs prefer to send their traffic through ISPs which do not charge them for carrying their traffic (peering), and sometimes this might cause relatively larger delay. It can be also the content owners business re- lationship with ISPs for their CDN. In boxing and Champions League final both experienced almost comparable performance, with 1.x times better or worse performance, the .x is 0 - 99 depending on the particular site we chose. For example, 1.5 times better performance.

67 CHAPTER 7. CONCLUSIONS

Our measurements showed us that the performance of streaming from most of the CDN enabled websites is not inﬂuenced by large number of visitors during popular media events, which shows that CDNs are well provisioned. Another factor was the distance of the CDN server to the user, despite being on a highly available and well dimensioned network the eﬀect of the network delay, can dominate. As an example, the average HTTP delay from SICS Ethernet to ATDHE, a streaming website which is not CDN enabled, (but close in terms of hops) was 96 ms. The average HTTP delay from SICS Ethernet to CRICHD, which was CDN enabled streaming website, showed an average delay 212 ms, but far in terms of hops. This shows that the network delay, can dominate. Of course the counter case has been seen to be true too, for example, the average HTTP delay from SICS Ethernet to SBS, a streaming website which is not CDN enabled, was 713 ms. Whereas the average HTTP delay from SICS Ethernet to BBC iPlayer, a CDN enabled streaming website, was 12 ms. The CDN server was close to the user, as we found from traceroute. We showed, it is possible to characterise the network performance using network tools such as ping, httping, and traceroute. By comparing ping and httping measurement results, we are able to distinguish whether performance degradation is created by the network or by the server. If the performance issue is known due to the network, by looking the traceroute results, we are able to identify where exactly the source of the problem is. We also able to identify local performance degradations because of the WiFi problem by comparing WiFi with Ethernet measurement results. This project shows that it is possible to characterize most of the user’s Internet experience using a few network tools. By using more additional network tools, and increasing the number of measurement agents, this project can help home Internet users and ISPs on identifying and troubleshooting existing network problems.

68 Chapter 8

Future work

CheesePi is an on-going project, of which my work was only a part. There are many ways it can be improved or extended. The most most important one is to conduct measurements from additional locations in Sweden. As of October, Raspberry PIs, running CheesePi have been installed at Karlstad, Västerås and Luleå universities. We could also add other parts of the Nordic Internet. Deployment of more CheesePi nodes would enable a more detailed network characterisation. Extending the number of Pis to further home users is the final goal, of course. In the experiments we have done mostly landmark measurements. With more deployed nodes we could expand to intra-Pi measurements. This would help understand the traffic on more links than to the landmarks. And specif- ically between home users, rather than from home users to large servers. We have seen path changes during our measurements and how it affects the user experience, the reasons are not clear at the end of this thesis. Further studies could and should look at this phenomenon, particularly considering longer measurement periods (at least a week) and again during popular events with video traffic. We have done some analysis on the measurement results, however this could be improved, for example, looking at the correlation between variables. Also some more detailed time series work could be done. Is there a trend in the data or are the delay measurements cyclic. We have used standard networking tools. More advanced networking tools and database storage engines could be included. For example tcpdump, paris- traceroute, one-way ping or a short TCP connection. These are done in the other works, such as Fathom or ARK, described briefly earlier.

Chapter 9

Lessons learned

This master thesis has helped me to have a much better understanding of TCP/IPs behavior. I read some technical reports and articles which helps me to understand some popular network monitoring systems. I learned how to conduct experiments better. This master thesis was a good opportunity to gain experience with lower level network tools. Writing this scientific report was the best experience during this master thesis. The thesis report might not be perfect, but I learned a lot about how to summarize my ideas, how to convey my thoughts in a scientific and accurate way. The continuous feedback from my supervisors helps me a lot to improve my report. I improved my programming skills mainly in Python and to some extent in bash scripting. Before this master thesis my Python knowledge was rather basic, but since all the measurement modules are implemented using Python, it helps me to improve my programming skills. It also introduced me to different libraries for data analysis. I also acquired basic bash scripting skills during this project. The project was implemented with another master student, and it was a good experience to develop my team working skills. We had to share our knowledge and convey our thoughts properly to each other. Guidance from our supervisors was very helpful, they helped us to see things in different perspectives. Holding weekly meeting helped me to plan and present my work. I was presenting what I did every Tuesday and this help me to plan my weekly tasks ahead and to present my weekly work. It was a reasonable time frame to report my work and to consider the feedback I got from my supervisors. SICS is a great place, and I had a chance to meet different researchers and students. It was good experience for me to meet people from different backgrounds and cultures, and everyone around was very supportive. There

71 CHAPTER 9. LESSONS LEARNED was also a weekly paper review session in the Network Embedded Systems department, the comments from the researchers who supported and opposed the paper were a good lesson in understanding research. This master thesis helps me on how to take initiative and be alert in order to accomplish your goals. I took some initiatives to conduct network measurements even not included in this master thesis.

72 Bibliography

[1] G. Bobolo, “5 digital video trends to watch in 2015.” http://www. inma.org/blogs/innovative-advertising-solutions/ post.cfm/5-digital-video-trends-to-watch-in-2015, 2015. Last visited 10th of 2015.

[2] J. F. Kurose and K. W. Ross, Computer Networking a Top Down Ap- proach 6th Edition. 2013.

[3] L. Parziale, D. T. Britt, C. Davis, J. Forrester, W. Liu, C. Matthews, and N. Rosselot, TCP/IP Tutorial and Technical Overview. 2006.

[4] Novell, “NetWare TCP/IP Administration Guide.” http://www. novell.com/documentation/nw6p/?page=/documentation/ nw6p/tcpipenu/data/hbnuubtt.html, 2015. Last visited 12th of 2015.

[5] A. S. TanenBaum and D. J. Wetherall, Computer Networks 5th Edition. 2011.

[6] O. Bonaventure, Computer Networking : Principles, Protocols and Prac- tice Release 0.25. 2012.

[7] “Cellular Network.” https://en.wikipedia.org/wiki/ Cellular_network, 2015. Last visited 17th of August 2015.

[8] “Content Delivery Network.” https://en.wikipedia.org/wiki/ Content_delivery_network, 2015. Last visited 17th of August 2015.

[9] “How does a CDN work.” http://www.cloudvps.com/ community/knowledge-base/how-does-a-cdn-work/. Last visited 18th of August 2015.

73 BIBLIOGRAPHY

[10] “What is a Raspberry Pi?.” https://www.raspberrypi.org/ help/what-is-a-raspberry-pi/. Last visited 18th of August 2015.

[11] “About 6 million Raspberry Pis have been sold.” https://blog.adafruit.com/2015/06/08/ about-6-million-raspberry-pis-have-been-sold-raspberry_ pi-raspberrypi-mattrichardson-twit-newscreensavers/. Last visited 18th of August 2015.

[12] “Passive Vs. Active monitoring.” https://www.slac.stanford. edu/comp/net/wan-mon/passive-vs-active.html. Last visited 19th of August 2015.

[13] “Traceroute(8) Linux Man page.” http://linux.die.net/man/8/ traceroute. Last visited 20th of August 2015.

[14] A. HÃ¥kansson, “Portal of Research Methods and Methodologies for Research Projects and Degree Projects.,” July 2015.

[15] C. Ordonez, Y. Song, and C. Garcia-Alvarado, “ARelational versus Non- Relational Database Systems for Data Warehousing,” October 2010.

[16] “Sports TV Ratings: How Many People Watched Mayweather vs. Pacquiao, The Kentucky Derby And NFL Draft?.” http://www.ibtimes.com/ sports-tv-ratings-how-many-people-watched-mayweather-vs-pacquiao-kentucky-derby-nfl-1907833, 2015.

[17] “Here’s how you can watch Floyd Mayweather ﬁght Manny Pacquiao tonight.” http://www.eurovision.tv/page/news?id=nearly_ 200_million_people_watch_eurovision_2015, 2015.

[18] “Nearly 200 million people watch Eurovision 2015.” http://www.mmamania.com/2015/5/2/8468715/ how-to-watch-floyd-mayweather-fight-manny-pacquiao-tonight-ppv-boxing-free-online, 2015.

[19] “Kevin Ashby, Berlin ﬁnal captures the world’s imagination.” http://www.uefa.com/uefachampionsleague/news/ newsid=2255318.html, 2015.

[20] “CRICFREE bounces back after UK police domain seizure.” https://torrentfreak.com/

74 cricfree-bounces-back-after-uk-police-domain-seizure-140524/, 2015. [21] “Streaming 101: ESPN Vs Univision In World Cup.” https://www.bizety.com/2014/06/28/ streaming-101-espn-vs-univision-world-cup/, 2015. [22] “Internet World Stats.” http://www.internetworldstats.com/ stats.htm, 2015. [23] S. S. Krishnan and R. K. Sitaraman, “Video Stream Quality Impacts Viewer Behavior: Inferring Causality Using Quasi-Experimental De- signs,” 2012. [24] “RIPE Atlas.” https://atlas.ripe.net/about/. Last visited 8th of October 2015. [25] “The Archipelago Measurement Infrastructure (ARK).” http://www. caida.org/projects/ark/. Last visited 8th of October 2015. [26] “Samknows.” https://www.samknows.com/. Last visited 8th of Oc- tober 2015. [27] “Test Node Brieﬁng Technical information relating to the SamKnows test nodes.” https://www.samknows.com/broadband/uploads/ methodology/SQ302-001-EN-Test-Node-Briefing-D01. pdf. Last visited 8th of October 2015. [28] “Netbeez.” https://netbeez.net/. Last visited 8th of October 2015. [29] “SpeedTest.” http://www.speedtest.net/about. Last visited 8th of October 2015. [30] “OOKLA.” https://www.ookla.com/. Last visited 8th of October 2015. [31] C. Kreibich, N. Weaver, B. Nechaev, and V. Paxson, “Netalyzr: Illumi- nating The Edge Network,” 2010. [32] M. Dhawan, J. Samuel, R. Teixeira, C. Kreibich, M. Allman, N. Weaver, and V. Paxson, “Fathom: A Browser-based Network Measurement Plat- form,” 2012. [33] “PingER.” https://www.slac.stanford.edu/grp/scs/net/ talk09/pern-pinger-initiative.pdf. Last visited 8th of October 2015.

75 BIBLIOGRAPHY

[34] “MLAB.” http://www.measurementlab.net/. Last visited 8th of October 2015.

[35] “Facts and ﬁgures about Akamai.” https://www.akamai.com/us/ en/about/facts-figures.jsp. Last visited 15th of October 2015.

76 Appendix A

SICS open house 2015

The SICS open house took place at SICS oﬃce in March 19, 2015. Over 400 guests came to listen to presentations of around 100 projects. People started to come around 7:00 UTC and the event continued till in the afternoon 13:00 UTC. We conduct network latency measurement to several targets from diﬀerent MAS, which are connected through SICS Ethernet, SICS Employees’ WiFi, and SICS guest WiFi.

Figure A.1: Average network latency of diﬀerent MAs to BBC during SICS open house 2015.

77 APPENDIX A. SICS OPEN HOUSE 2015

Figure A.2: Minimum network latency of diﬀerent MAs to BBC during SICS open house 2015.

Figure A.3: Maximum network latency of diﬀerent MAs to BBC during SICS open house 2015.