The Network Predictors Monitoring Network Measurements to Predict Corporate Performance Before It Is Public Knowledge

The Network Predictors Monitoring Network Measurements To Predict Corporate Performance Before It Is Public Knowledge

Roberto Canale Technische Universiteit Delft The Network Predictors Monitoring Network Measurements To Predict Corporate Performance Before It Is Public Knowledge

Roberto Canale in partial fulfillment of the requirements for the degree of

Master of Science in Management of Technology

at the Delft University of Technology

To be defended in public on November 21th 2019

Student number: 4731387 Project duration: February, 2019 – September, 2019 Thesis committee: Chairperson Prof.dr. ir. M. J. G. van Eeten, TU Delft, Organisation & Governance First Supervisor Dr.Ing. T. Fiebig, TU Delft, Information & Communication Technology Second Supervisor Dr. E. Schröder, TU Delft, Economics of Technology & Innovation

An electronic version of this thesis is available at http://repository.tudelft.nl/. Page intentionally left blank. Foreword

This Master Thesis project is the result of several monthslong project, started in February 2019 and completed in October 2019. It is the final examination for a Master of Science in “Management of Technology” at the Delft University of Technology. This research project tries to discover and prove the existence of a relationship between network measurements and companies. In particular, we tried to see if it is possible to predict company performance through network measurements and provide insider information on the company. It is an explorative study connecting networks and companies, intended for computer science researchers as well as investors and business analysts. Lastly, it also gives network measurements relevance in another field, possibly paving the way for further research. This project challenged me and my skills greatly, proving to be a unique learning experience. I developed a solid knowledge of computer networks, their history and structure, along with discovering the new trends on companies operating through the Internet. Developing and planning a research method has also been a unique challenge from which I believe to have learned a lot. I was also able to discover firsthand how companies communicate with consumers and investors, while understanding, at least partly, their internal operations. The project also taught me many aspects of LateX and how to put together a large and heterogeneous document. Lastly, I was also able to improve my skills in Python and data analysis, especially learning how to display data correctly. First of all, I would like to thank my supervisor Dr.Ing. Tobias Fiebig, for providing me with this interesting research topic. I want to thank you for providing my thesis with interesting measurements to study and helping choose other measurements to use in this thesis. I thank you for your patience, communicating with me through Skype while I was not in Delft and helping me straightening my project when it was not complying to the standards of this university. I thank kindly thank Prof. dr. ir. M. J. G. van Eeten and Dr. E. Schröder, for providing me with essential feedback and helping me bringing back on track my project when needed. I also want to tank Full Professor Marco Mellia, coordinator of the SmartData@PoliTo centre and Martino Trevisan, a PhD candidate at PoliTo. I thank you for sharing with me the traffic measurements you collected on Italian networks, as they have proved essential for this thesis. On a personal note, the thesis proved to be a great personal challenge. While working on the project, in fact, I tore my Achille’s tendon, forcing me to go fly home to Italy for surgery and spending two months in bed with a cast. For an active and energetic person like me, this has proven extremely stressful. This accident delayed my work on the thesis and the communication with my supervisor, as the project slumped for quite a while. I want to deeply thank my family, Francesco, Carla and Chiara, for supporting assisting me at home during my convalesce and providing me with the kindest care and attention during that hard time. I would not have gotten through it without you, as you are the most important people of my life. I also want to thank my Delft roommates and rugby mates for assisting me after the injury. In particular, I want to thank Tom van de Kemp, as you spent the evening with me at the hospital and then drove me to the airport to return to Italy for the immediate surgery I needed. I hope you will enjoy reading this thesis, as it is a different approach to network measurements and how they can be used in the world of business.

Roberto Canale Delft, October 2019

iii Page intentionally left blank. Executive Summary

The thesis investigates the relationship between corporations operating online, providing digital ser vices, and network measurements. In particular, the goal of the thesis is verify whether network measurements can predict company performance or provide information on a company before it is publicly available. Considering digital companies strictly related to the Internet, the thesis builds three case studies showing how network measurement predict company performance. Network measurements are the monitoring and collection of computer networks describing its char acteristics; company performance is a composition of heterogeneous indicators used to describe and evaluate the state of the business. Companies and markets have changed dramatically in the last few decades. The Internet has brought disruptive changes to our society, and the way companies and corporations, communicate and conduct business. Simultaneously, markets have seen the unprecedented rise and growth of ebusinesses, i.e. companies and activities partly or fully operating online through the network infras tructure. These companies operate through the Internet, digitalizing traditional markets or completely creating new ones for products and services. It might, therefore, seem trivial that their performance and success is strictly related to the Internet. But the generally lacking literature on the topic gives this project that the opportunity to study companies through network measurements arises. Specifically, it questions whether network measurement can provide the researcher with usable information or data to predict company performance. Knowing this kind information before it is made public, could prove extremely valuable for researchers, managers and analysts. Network measurements are a varied set of metrics that describe the characteristics of the Internet for multiple purposes. [1] and [2] measure the speed of bytes flow through TCP connections and therefore dealing with Internet speeds. Others, like [3], use Internet Protocol addresses to study the relationship between elements on the network and addressing Internet topology issues. These are just a few examples of what researchers measure on the Internet and their purposes; however, little attention is given on how companies and their performance is related to its own network measurements. The thesis assumes that there is a relationship between companies and their measurements, and to show the existence of such a connection, it focuses on predictability. There is no scientific literature that predicts company performance by means of network measurements. The purpose of the thesis is to show that company performance can be predicted by using and monitoring network measurements. This means acquiring valuable knowledge on a company from the measurements, hence showing a connection and discussing it. After such a relationship is found, the thesis discusses whether it is or not is an interesting subject that needs further research. The research has, therefore, an explorative purpose, as it develops a topic that has not been thoroughly analyzed yet.To prove the existence of such relationship, the thesis answers the following Research Question:

Do network measurements predict company performance?

To answer this research question, the thesis develops three subquestions and uses literature research and case study analysis as core methodology. The case study is based on [4] and its work, defining the objectives of the cases study and the proper structure and steps that need to be done in a casestudy methodology. The project presents and analyses cases where network measurements predict changes in company performance or relevant announcements before they are known to the public By answering this research question, it is possible to explore the relationship between Internet metrics, i.e. network measurements, and company performance. The thesis defines what company performance is and how it can be used in the cases, related to the measurements. The concept of company performance depends on the company and its market, and cannot be modelled into a single set of metrics valuable for each company. For the purpose of this thesis, various factors are considered as representative of company performance, most of which are referred

v vi Executive Summary to as Key Performance Indicators, or KPIs. Next, the necessary background on network measurements is provided through literature research. This means understanding what network measurements, what they can represent and how they are used; then the measurement sets are selected to build the case studies. The problem is approached with a case study methodology and three cases studies are built; historical measurements and data are used to be able to verify and validate the findings from a future perspective. The three cases consider network measurements for Salesforce.com, Inc., Netflix, Inc. and Snap. Inc.. Salesforce’s measurements were collected via active measurements techniques [5] and show the company’s number of IPv6 addresses and therefore its number of hosts. Netflix’s and Snapchat’s measurements were collected via passive techniques [6] and show the company’s users traffic and number in Italy. The cases consider the company, its measurements, its performance indicators and relevant infor mation of various kinds, such as companyrelated news. KPIs and meaningful news that can be related to the measurements at hand are collected. The measurements are monitored to establish whether, at a certain point in time, it would have been possible to predict a certain performance indicator or relevant news. The cases show that for or each company and service, the measurements, at a certain point in time, it is possible predict changes in company performance before it is made public. The re sults are then analyzed together through crosscase analysis as [4] instructs and a discussion on them is provided. A crosscase analysis studies why it was possible to predict certain performance indicator and what does it show on the relationship between companies and networks. Lastly, the research answers the main research questions and provides suggestions for future research on the subject. The thesis tackles the subject of predictability of company performance through network measure ments. It shows that in some cases and for some companies, it is possible to predict and anticipate com pany performance with such measurements. For Salesforce, a meaningful increase in IPv6 addresses predicts the acquisition of major clients and a couple of beyondownforecast quarterly revenues. For Netflix, user traffic shows which of its series are more popular and that it is steadily increasing its userbase abroad. Lastly, a year after Snapchat’s user traffic starts to decrease sharply, the company reports the firsttimeever a loss of users. These are the main findings from each case, along with minor findings discussed in the specific cases. By combining together the results from each case, the project provides a definite answer to the main research question. Network measurements therefore predict increase in subscribers, beyondforecast revenues, acquisition of clients and loss of users. It is possible to predict meaningful changes and announcements in company performance while monitoring network measurements. There are strong limitations to our claim but it is a valid starting point for future research. There is a gap in the scientific literature on the relationship between companies and network traffic measurements. The answer to the research question proves that this is a still unexplored subject that should be studied and analyzed further. The research is a first step in correlating businesses operating online to network measurements and contribute to the literature with novel findings. The research found the existence of a predictive relationship between the measurements and company performance, showing the need for further and broader research on the subject. First of all, a larger set of measurements is needed, to have a broader perspective on performance predictability from the measurements: the next step is to prove the correlations found in a statistical way. If proven effective, the research also suggests establishing a decisionmaking framework for companies, managers and investors that includes the measurements as a reliable source. Next, a detailed framework to establish which types of measurements can predict which performance indicators and why some KPIs or factors were not predicted: it would explore in detail the relationship between company and measurements. The research also suggests to companies, managers and investor, to explore the possibilities of network measurements to predict performance. Companies should monitor or protect their own or others’ network measurements, as it can be a powerful tool on the market. At the same time, it also warns them of awareness of their own traffic. Ultimately, the thesis encourages the scientific community to explore this relationship further. A much larger dataset and set of companies are required to statistically evaluate the usefulness and validity of measurements as a performance predictor. Keywords: Networks, Network Measurements, Company Performance, KPI, Performance Predic tion, Salesforce, Netflix, Snapchat Page intentionally left blank. Contents

Executive Summary v List of Figures xi List of Tables xiii 1 Introduction 1 1.1 Problem Introduction: Tumblr Loss of Users ...... 1 1.1.1 Construct Introduction ...... 1 1.1.2 Literature Review ...... 2 1.2 Research Gap and Problem Statement ...... 3 1.2.1 Research Gap ...... 3 1.2.2 Research Objectives ...... 4 1.3 Research Question, SubQuestions & Methodology Approach ...... 4 1.3.1 SubQuestions ...... 5 1.3.2 Methodology Approach ...... 6 1.4 Research Contribution...... 6 1.4.1 Scientific Relevance ...... 6 1.4.2 Management of Technology Affinity ...... 7 1.5 Document Structure ...... 8 2 Methodology 9 2.1 Background ...... 10 2.2 Case Study Review ...... 11 2.2.1 Fitness of the Case Study Methodology...... 11 2.2.2 Designing a Case Study ...... 11 2.2.3 Case Study Selection ...... 13 2.2.4 CrossCase Conclusions...... 14 2.3 Core Methodology ...... 14 2.3.1 Measurements, Monitoring & Interpretation...... 14 2.3.2 Performance Overview ...... 15 2.3.3 Analysis & Results ...... 15 2.4 Case Structure ...... 16 2.4.1 Company Introduction...... 16 2.4.2 Data Sources ...... 17 2.4.3 Measurements, Monitoring & Interpretation...... 17 2.4.4 Performance Overview ...... 17 2.4.5 Analysis& Results ...... 17 2.4.6 Conclusions ...... 17 3 Network Infrastructure, Measurements Techniques and Background Literature 19 3.1 Network Infrastructure ...... 19 3.2 Network Measurements ...... 20 3.3 Measurement Metrics ...... 21 3.3.1 Metrics ...... 21 3.3.2 Possible Relations to Indicators ...... 22 3.4 Measurement Techniques...... 22 3.4.1 Active Measurements ...... 23 3.4.2 Passive Measurements ...... 23 3.4.3 Hybrid Measurements ...... 23 3.4.4 Vantage Points and Techniques ...... 24

viii Contents ix

3.5 Measurement Uses ...... 24 3.6 Network Measurements and Companies ...... 25 3.6.1 CDNs ...... 25 3.6.2 Traffic Monitoring ...... 25 3.6.3 Novel Use of Network Measurements ...... 26 4 Company Performance 27 4.1 Key Performance Indicators...... 27 4.2 Financial KPIs...... 28 4.2.1 Indicators & Sources...... 28 4.2.2 Considerations on the Proposed Indicators ...... 30 4.3 Webrelated KPIs ...... 31 4.4 General KPIs ...... 32 5 Case Selection 33 5.1 Considered Measurement Repositories ...... 33 5.2 Measurements on Companies ...... 34 5.3 Cases Selected ...... 34 6 Salesforce 36 6.1 Salesforce.com ...... 36 6.2 Data Sources ...... 37 6.3 Measurements, Monitoring & Interpretation...... 38 6.3.1 Collection Methodology ...... 38 6.3.2 Hosts ...... 38 6.3.3 Measurements Specifications...... 39 6.3.4 Measurements Interpretation ...... 40 6.4 Performance Overview ...... 40 6.4.1 Financial Performance...... 41 6.4.2 Significant News ...... 44 6.5 Analysis & Results ...... 44 6.5.1 Analysis ...... 44 6.5.2 Results ...... 45 6.5.3 Salesforce Timeline...... 48 6.6 Conclusions ...... 49 7 Netflix 51 7.1 Netflix, Inc...... 51 7.1.1 Netflix Audience ...... 51 7.1.2 International Expansion...... 52 7.1.3 Original Content ...... 52 7.2 Data Sources ...... 53 7.3 Measurements, Monitoring & Interpretation...... 54 7.3.1 Collection Methodology ...... 54 7.3.2 Measurements Monitoring ...... 54 7.3.3 Measurements Interpretation ...... 57 7.4 Performance Overview ...... 58 7.4.1 Financial Performance...... 58 7.4.2 Other KPIs ...... 62 7.4.3 Italian & International SVoD Market ...... 64 7.5 Analysis & Results ...... 64 7.5.1 Analysis ...... 64 7.5.2 Results ...... 65 7.5.3 Netflix Timeline ...... 68 7.6 Conclusions ...... 69 x Contents

8 Snapchat 71 8.1 Snap Inc...... 71 8.1.1 Snapchat...... 71 8.1.2 Demographics...... 72 8.2 Data Sources ...... 72 8.3 Measurements Monitoring & Interpretation ...... 73 8.3.1 Measurement Monitoring ...... 73 8.3.2 Measurement Interpretation ...... 76 8.4 Financial Performance...... 76 8.5 Relevant News and Information ...... 80 8.5.1 Snap Inc. IPO ...... 80 8.5.2 Instagram ...... 80 8.6 Analysis & Results ...... 81 8.6.1 Analysis ...... 81 8.6.2 Results ...... 82 8.6.3 Snapchat Timeline ...... 84 8.7 Conclusions ...... 85 9 Discussion 87 9.1 Summary of Results ...... 87 9.1.1 Literature Review Results ...... 87 9.1.2 Case Results ...... 88 9.2 CrossCase Synthesis ...... 88 9.2.1 Corporate Performance & Measurements ...... 88 9.2.2 Performance Prediction & KPIs...... 89 9.3 Social Implications ...... 90 9.3.1 Considerations for Stakeholders ...... 90 9.3.2 Considerations for Managers ...... 91 10 Conclusions & Recomendations 93 10.1Research Questions ...... 93 10.1.1Answers to SubQuestions ...... 93 10.1.2Answer to the Main Research Question...... 96 10.1.3Theoretical Cotribution ...... 96 10.2Suggestions For Future Research ...... 97 10.2.1Social Research Insights ...... 97 10.2.2Other Datasets & Measurements ...... 98 10.2.3Further Research...... 98 Bibliography 99 List of Figures

2.1 Methodology Structure ...... 9 2.2 General Report Structure ...... 10 2.3 Multiple Case Study Method ...... 12 2.4 Measurements, Monitoring & Interpretation ...... 15 2.5 Core Methodology ...... 16

3.1 Simplified Structure of the Network...... 20 3.2 An example of an Active Network Measurement ...... 23 3.3 An example of a Passive Network Measurement ...... 23 3.4 An example of Hybrid Network Measurement ...... 24

6.1 Salesforce Hosts ...... 39 6.2 Salesforce Revenue and Difference with respect to Forecast ...... 41 6.3 Salesforce GAAP Earnings Per Share and Difference with respect to Forecast ...... 42 6.4 Salesforce Non GAAP Earnings Per Share and Difference with respect to Forecast ... 42 6.5 Salesforce NPM ...... 43 6.6 Salesforce Net Income ...... 43 6.7 Salesforce Return on Equity ...... 43 6.8 Salesforce Return on Assets ...... 43 6.9 Salesforce Revenue Difference with Respect to Forecast ...... 47 6.10 Salesforce Hosts ...... 48 6.11 Salesforce Timeline ...... 49

7.1 Measurement Infrastructure and Processing Steps ...... 54 7.2 Netflix Total Download Traffic ...... 55 7.3 Netflix Total Upload Traffic ...... 55 7.4 Netflix Total Traffic Flows ...... 56 7.5 Netflix Total Users ...... 56 7.6 Netflix Download MB per User ...... 56 7.7 Netflix Upload MB per User ...... 57 7.8 Netflix Revenues ...... 59 7.9 Netflix Financial Ratios ...... 60 7.10 Netflix Earnings Per Share ...... 60 7.11 Netflix Operating Income & Net Income ...... 61 7.12 Netflix Subscribers ...... 63 7.13 Netflix New Original Series ...... 63 7.14 Netflix International Subscribers and Total Download Traffic ...... 66 7.15 Netflix International Results ...... 67 7.16 Netflix Total Download Traffic ...... 67 7.17 Netflix Timeline ...... 69

8.1 Snapchat Demographics ...... 72 8.2 Snapchat Total Download Traffic ...... 74 8.3 Snapchat Total Upload Traffic ...... 74 8.4 Snapchat Total Traffic Flows ...... 74 8.5 Snapchat Total Users ...... 75 8.6 Snapchat Download MB per User ...... 75 8.7 Snapchat Upload MB per User ...... 75 8.8 Instagram Download MB per User ...... 76

xi xii List of Figures

8.9 Snapchat Revenues and Average Revenue Per User ...... 77 8.10 Snapchat Net Income adn EPS ...... 77 8.11 Snapchat NPM ...... 78 8.12 Snapchat Return on Assets ...... 78 8.13 Snapchat Return on Equity ...... 78 8.14 Snapchat Daily Active Users ...... 79 8.15 Snapchat Stock Price ...... 80 8.16 Snapchat and Instagram Daily User Download Traffic ...... 82 8.17 Snapchat Total Download Traffic ...... 83 8.18 Snapchat Daily Active Users And Total Traffic ...... 83 8.19 Snapchat Stock Price with Total Snapchat Traffic ...... 84 8.20 Snapchat Timeline ...... 85 List of Tables

2.1 Core Methodology & Case Structure ...... 18

4.1 Financial KPIs ...... 30

6.1 Date References for Fiscal and Time Quarter ...... 37 6.2 Salesforce Revenues and Forecast ...... 41 6.3 Salesforce Yo.Y. Revenue Increase and Earnings Per Share ...... 41 6.4 Salesforce Financial Ratios ...... 43 6.5 Salesforce Prediction & Announcement Dates ...... 48

7.1 Netflix Market Penetration ...... 52 7.2 Netflix Revenues ...... 59 7.3 Netflix Financial Ratios & EPS ...... 61 7.4 Netflix OI & NI ...... 62 7.5 Netflix Subscribers ...... 63 7.6 Netflix Subscribers in Italy ...... 66 7.7 Netflix Original Shows & Key Dates ...... 67

8.1 Snapchat Revenues and Average Revenue Per User ...... 77 8.2 Snapchat Net Income adn EPS ...... 78 8.3 Snapchat Financial Ratios ...... 78 8.4 Snapchat Daily Active Users ...... 79

xiii Page intentionally left blank. 1 Introduction

1.1. Problem Introduction: Tumblr Loss of Users

Tumblr is a social media platform for microblogging where users can network and upload content [7]. It is a blogging website founded in 2007, that partly inspired the research project. After strong criticism, and its removal from Apple’s iOS App Store, on the 17th December 2018 the company decided to ban and remove NSFW and pornographic content [8]. This outraged its core users, as the app’s traffic started to decline sharply: pornographic content was clearly one of the main drivers on the platform. From 521 million monthly website visits in December 2018, the traffic was already plunging to 369.9 million visits in February 2019 [9], [10]. It is clear that the decision strongly affected its daily visits and its userbase. In this case, we had access, a posteriori, to its website traffic. We wonder how many actual users remained after such decision, and if it were possible to monitor them. This is insider knowledge of the company, and the question is if it would have been insightful to monitor its users. Being a blogging platform, its number of visits is a clear performance parameter, as it generates income based on ads. After such a decision was made, could have we predicted the sharp decrease in visits by monitoring the number of users? After all, how users reacted to such a decision, and how that changed their behaviour on the platform are valuable pieces of information. Tumblr was later sold for 20 million $ to Automattic Inc. (owner of WordPress): compared to the valuation of 2013, it was just 2% of its past value [8], [11]. The Tumblr case raises a few questions; it is clear that visits represent a way of establishing how well Tumblr is doing: they are, in this case, considered as performance indicators for the company. The assumption is that parameters like daily and monthly active users might have predicted the loss of visits by Tumblr. We question what more information could be provided by such monitoring that and if it could provide valuable information on the performance of the company: knowing this information before it was made public, provides a valuable market edge. Ultimately, would it have been possible to predict the loss of Tumblr visits before it was made known? So far, this is only a missed opportunity for research. Furthermore, if users were meaningful, how would it have been possible to obtain such data? Being an online platform operating on the Internet, network measurements and monitoring seem the best way to gather, collect and analyse this type of data. Network measurement can describe Internetrelated metrics like, for instance, traffic or users. This is the technology that can be used to collect and obtain the abovementioned information and is at the core of this thesis. The main idea is to verify if such data can be predictive of company performance. We question whether there is a connection between certain measurements and companies, as the Tumblr case suggests. By network measurements, we refer to the collection and gathering of empirical data describing a characteristic of the network.

1.1.1. Construct Introduction In the previous section we have linked Tumblr’s performance to its number of visits, and proposed network measurements as a means to collect data on its users; this, however, was a specific and

1 2 Introduction particular case. The next step is to introduce and define the two constructs that we focused on: network measurements and company performance. These are further discussed and identified in later chapters, but it is imperative to have an initial and working understanding of the two. To measure a network means collecting empirical data on the various computer networks that make up the Internet. Network measurement is the activity and process of measuring and recording Internet traffic and other metrics to be later analyzed. These could be its physical or soft characteristics, like quantity and speeds of the traffic passing through certain networks, or the names and addresses that computers use to exchange data. There are various kinds of measurements with different and multiple purposes, as networks are constantly measured and analysed, to understand their characteristics, usage, topology, relations, health, security and performance. We define network measurements as the “collection, interpretation and modelling of empirical Internet data”[12], p. 31. Next, we introduce what company performance is. Performance is the result of a given task or activity against preset and known standards; applying this concept to corporations, company perfor mance describes how well a company’s activity is faring. Ultimately, a company’s objective is profit; establishing how well its operations are performing can be a way to determine how many profits are expected. Activity is, however, a quite vague term: nowadays most companies have more than one business line. Furthermore, in the extremely diversified digital and analogue markets that exist nowa days, each company values its performance differently. For a company, performance is, therefore, the composite set of various parameters that evaluate a certain activity or aspect of its operations. Gener ally, these parameters can be Key Performance Indicators ( or KPIs) and are used as the main tool to understand a business and its performance. These can be financial or webrelated and are commonly used as a proxy of performance. However, there can be other “less quantifiable” parameters that a company can consider as performancerelated. For instance, the acquisition of new clients, a positive review or endorsement and M&As can be considered as part of the performance of the company. A more thorough discussion and stricter definition of performance and KPIs follow in Chapter 4.

1.1.2. Literature Review Having defined these two construct, we proceed with an initial literature review on networks. We proceed with analysing the current literature measurements, and see what is the current state of the art relating measurements and companies. In particular, we scan for literature that relates computer networks to company performance. The Internet is now part of most people’s lives. Started as a switching packet interface in the ‘60s in the United States [13], it was mostly used by researchers and developed further in the 70s. Its popularity drastically increased in the 80s and 90s, growing in connectivity and applications [14]: today, almost everybody in developed countries has access to it. Its reached mass adoption after the invention of the World Wide Web in 1991, at CERN, by Tim BernersLee. People use the Internet for work, interpersonal communication, international commerce, research and technological development [12]. The past decade displayed a never seen before the growth of the Internet: adoption rates, social and economic relevance and traffic volumes have nothing but increased exponentially [15]. Network measurements have received different levels of attention during the development and ex pansion of this global network [12]. In the last couple of decades, however, researchers scanned and measured networks with increased interest, developing new tools and techniques to perform measure ments. The wider networks became, the greater the need for them: “network measurement research has grown in scope and magnitude to match the Internet.”[16], p. 74, implying the need for larger network monitoring and research. Accurate, precise and sound measurements help network managers understand and spot errors, protect and maintain networks properly [17], [12]). Network managers base their decisions and operations on the current state of the network; it is, therefore, crucial for them to rely on sound and precise measurements. Much more so nowadays that such traffic volumes are becoming ever larger, as internet adoption increases across the world. Alongside the Internet, an entire scientific and business community grew and developed expo nentially. Businesses saw opportunities in computer networks and started shifting and reorganizing, partially or totally, their activities on the Web [18]. It is also now common for businesses to operate exclusively on the Internet and offer products that exist only through the Web. Given the relevance of networks in the Internet scientific community and business community, we would assume that they 1.2. Research Gap and Problem Statement 3 are important for the latter as well. This is not the case, as there is only some research correlating businesses to network measurements. Most research relates company operations and strategies on the network. Others consider Internet companies and the clientprovider relationships between them [19], [20]. While technically relevant, most research shows how things operate at the time of measurements, with little interest to future implications. From an initial literature analysis, we see how network measurements are used and for which pur poses; [21] use measurements to understand the validity of security protocols; [1] or [22] focus their research on network performance and its analysis. [3] address, study the topology and hierarchical characterisation in networks. Another recurring theme is verifying the deployment of new technolo gies, such as IPv6 deployment, to propose innovative measurement techniques, measurement tools or software for analysis [5]. The main topics in network research are performance, security, adoption of innovations, new measurement methodology, maintenance, topology and understanding the relations between network elements [2], [23], [24]. This type of research is extremely various and conspicuous, but, mostly, quite technical and focussed on the network. There is rather little research linking network measurements to other subjects. An interesting but rather isolated case is brought by [25], where they study and analyze the effects that increasing Internet traffic, data centres and distributed networks have on the electricity grid, showing how in London traffic peaks correlate to power demands. In general, there is little scientific literature relating to network measurements and Internet traffic to companies, which seems rather odd. Despite the extensive presence of businesses on the Internet, some of which are even defined by some as “Internet giants”, few companies are studied. [26] links Web metrics to ebusinesses, but is exclusive to Web activities and describes an environment that does not exist anymore, as it describes a 2001 context that has now radically evolved. Most of the relevant literature deals with Internet content providers, their traffic, and their relation to CDNs, Content Delivery Networks [27], [19], [28]. We are also surprised that researchers did not focus much on this subject, digging deeper into companies, as it is a very wide argument, in which various types of measurements could be performed. As many companies shift on the Internet, we assume that network measurements can provide insight into companies. In particular, there is no literature whatsoever, where measurements are used as a predictive tool for company performance. As we discuss in Chapter 3, measurements are used to study background activities of content delivery companies. However, these studies deal with technical issues and strategies that companies use, unknown to the public.

1.2. Research Gap and Problem Statement

1.2.1. Research Gap From the initial literature review in Section 1.1.2 and the Tumblr case, we see a research opportunity into companies by means of network measurements, since there is little literature relating the two constructs (more insights are provided in section 3.6). Generally, network measurements are used to study performance, network topology or internal operations of elements such as CDNs [27], [19], [28], [1], [22], [3]. In particular, there is no literature on measurements used as predictors of company performance. While this might be expected for “brick and mortar” stores, it seems unusual for e business and companies operating online. This lack of research also leaves room for our research: we use network measurements in a novel and exploratory way: predicting company performance. We question whether monitoring certain measurement can provide insider information on a certain company and predict its future performance: this means knowing something on the company before it is made public. For instance, knowing and monitoring Tumblr userbase might have proven insightful or useful in predicting its loss of visits There is no previous research on the matter and is the subject that we explore in this thesis. This is clearly a rather unexplored topic in network measurement re search (Section 1.1.2), and it requires further attention and investigation. The issue is understanding whether measurements can be used as a predictive tool, in which context, for which companies and for which indicators. Timing is essential as well: knowing the right information at the right time could be enlightening on the company and its current performance. Furthermore, establishing which kinds of connections can exist and which are invalid, is a meaningful addition to the current state of the art. 4 Introduction

Due to the lacking literature on this aspect, it is an issue worth exploring whose relevance still needs to be established. This type of research and correlation is meaningful for a broad spectrum of stakeholders, with companies at the forefront. Showing the existence of this type of correlation could help businesses understand themselves better, obtain useful information on their competitors and obtain a competitive advantage. Similarly, market analysts and investors can profit by such a relation, making better market predictions and investments decisions. The fundamentally untapped potential of this type of measure ments and analysis gives us room to research one particular aspect of this unidentified relationship between networks and businesses.

1.2.2. Research Objectives The highlighted gap in the literature clarifies the objectives of the thesis. The core objective of the project is to verify whether it is possible to predict changes in performance before it is made available to the general public by means of network measurements. The assumption from Tumblr’s case is that it a decrease in performance could have been predicted by a loss of users, and by monitoring the number of users it would have been possible to know this information before it was made public. It raises the question of whether a relationship between the two constructs exists and if one can predict the other; combined with the lack of literature, it provides the thesis with a meaningful research opportunity. This thesis studies network measurements in relation to companies and explores the relationship between them. In particular, we focus on what we can see from the measurements at a certain point in time, and what we already know on the company before this information becomes public. This is not a trivial task, as it also assumes that there is a connection of some kind between a measurement and a performance indicator. We want to see if network measurement can help forecast and understand company performance in some way. The nature of this research is therefore exploratory, as this would be the first time that network measurements are related to companies in this way. We structure the research on three case studies on Salesforce, Netflix and Snapchat with a multiple case study scenario [4]. The goal is not to prove a specific relation by itself: but to show the relevance and existence of such a relationship between a measurement and an indicator. Starting from the definition of company performance, we intend to show that its elements can be predicted by the measurements. This entails establishing why such indicators were predicted and why others were not. Ultimately, the goal of the thesis is to verify if such a relationship between network measurements and company performance exists and is worth exploring further.

1.3. Research Question, SubQuestions & Methodology Approach

Network measurements have not been thoroughly studied in all their possible fields. While we see them used for testing, topology and performance applications on measurements, they are not widely used in relation to companies. In particular, we highlighted an absence of research on the predictive ability of company performance. To address such a gap in the literature, the research focuses on the relation between measurements and company performance and to see if it is possible to predict it. The main research question of the thesis is the following:

Main RQ

Do network measurements predict company performance?

The nature of the question is explorative and aims to discover and explain the possible relationships and findings on the topic. The background idea to answer this research question is that it is possible to access, in some way, information on the company before this is made available to the public. This information, provided by monitoring the measurements, is assumed to predict changes in company performance with insider information that is still not available to the general public. In particular, it should clarify which indicators were predicted by the measurements and possibly why. Similarly, it should also clarify why certain indicators are not related to the measurements, i.e. a change in them 1.3. Research Question, SubQuestions & Methodology Approach 5 was not predicted. This way, it is possible to explain the valid predictions and implications made from the measurements on Salesforce, Netflix and Snapchat.

1.3.1. SubQuestions To thoroughly answer the main research question, we use three subquestion on which we structure our research. These are building blocks to guide the research towards a clear and outcome by defining the core elements of the research. The first subquestion is the straightforward identification of the construct company performance:

SQ1

What is company performance?

We define company performance with appropriate estimators that can be used as a proxy for perfor mance. We answer this research question by checking what estimators companies use in their reports and through literature analysis. While a temporary definition was already introduced further discussion is needed on the factors and indicators used for company performance. These factors, from the fields of economics and finance but not exclusively, have to be related to companies and their market. We use the same indicators that companies and the market use to evaluate their own performance; dif ferent factor reflects different aspects and perspective on the company. These indicators are grouped together and often called KPIs. Once the performance is defined, we need to find adequate measurements to analyze and to relate to companies:

SQ2

What are suitable network measurement sets and techniques to predict companies’ performance?

With this subquestion, we identify the types of measurement sets to predict company performance. Firstly, we define the requirements and parameters to be used during the measurements and techniques selection. Various network traffic measurements of variable nature can be found online, mostly from research institutes. We can open data repositories to obtain relevant datasets for our goals. The many measurement types, sources and techniques are studied and analysed. With this knowledge, we can then sift through and filter the considered repositories for possibly predictive measurements. The third subquestion brings together the previous two and provides insights on the predictability of performance:

SQ3

Which KPIs are predicted by network measurements?

The third subquestion adds two crucial elements to our research and combines the previous two. It brings together the construct of network measurements along with company performance. While closely related to the main Research Question, it serves as a foundation for further discussion and analysis, in order to find an answer to our research. In particular, we discuss why and how certain KPIs were predicted by the measurements and why others were not. This answer is found at the end of each case for each company considered. As we have said, we identify company performance with KPIs (a thorough discussion and justification are provided in Chapter 4). Through a future perspective, we see whether network measurements can anticipate changes in KPIs. By using historical data, we can study and monitor the measurements, and see if they anticipate and forecast changes in KPI or performance in some way at a certain point in time. We validate our findings in this way, by checking if crucial information on a company was made available through the measurements before it was known. This way, we can verify if at a certain point in time, with certain available information, it would have been possible to predict company performance. If our findings are found valid and insightful, they can prove the need for further research, as there it is a topic worth exploring. 6 Introduction

1.3.2. Methodology Approach

To answer the main Research Question and the Subquestion, we use an exploratory approach to our research. We use an exploratory approach as there is no available literature on the subject and no clear starting point for it. The exploratory approach is used when a few facts on the subject are available or known; it is appropriate to use it when little is known of a certain phenomenon or context. It allows the researcher to seek new insights, with qualitative and quantitative approaches. We need to use research tools that allow us to discover new findings on the topic, based on realworld experience. The first two subquestions are answered by means literature review on the subjects, to have a proper idea of performance, measurements and how to select measurements sets to use in the thesis. Then, we use a case study as a research methodology to answer our third subquestions and gather our results. We use a multiple case study scenario, following [4]’s design; a thorough discussion on the methodology follows in chapter 2. We built a case study per set of measurements and company available: the three cases are on Salesforce, Netflix and Snapchat. In each of this case, we study the measurements and the performance of the company, by means of KPIs and news. In the cases, we use historical data, measurements and news. This way, we see whether some insider information on the company was already available at a certain point in time, before it was announced and made public: it is a way to validate our findings. Basically, we intend to see if by monitoring the measurements it is possible to know in advance meaningful information on company performance. Once the study cases are completed, we group the main findings, discuss their implications and uses. Lastly, we answer our findings and make suggestions for future research.

1.4. Research Contribution

1.4.1. Scientific Relevance

In this context, we propose to use network measurements for less documented research purposes. The goal of the research is to prove the existence of a predictable relationship between company performance and network measurements. It combines the more technical topic of network measure ments literature with a business and social perspective. The scientific relevance of such goal can be summarized in the following points:

• In certain cases, network measurements predict changes in certain KPIs; secondly, changes in the measurements anticipate meaningful information and news on the company. This happened throughout the three cases, but each with different performance indicators. We discuss which KPIs are predicted and which are not, trying to understand why and their connection with the measurements. We discuss our findings and the possible social implications that they might have, discussing whether this subject needs further research and it is worth exploring.

• The research proposed has an explorative nature: it contributes to the scientific literature with novel information and findings. There is little literature linking company performance to net work measurement, and the research starts to fill that gap. Most network measurements deal with technical aspects of the Internet and its understanding. Some researches were published in early 2000, linking Web metrics to Webcompanies [26]; however, since then, the Internet has drastically changed and evolved, in a dramatic fashion for both Web and “brick & mortar” businesses.

• The research provides a new perspective on network measurements and companies; it discusses what are new implications found from the cases for the stakeholders involved. In particular, it highlights the fact that network measurements should not be overlooks by managers or analysis and that should be taken into account in a decisionmaking process. 1.4. Research Contribution 7

1.4.2. Management of Technology Affinity The thesis was developed as part of the final project to the Master of Science in Management of Technology conducted at the Delft University of Technology. The program is based on the idea that industries professionals able to implement new technologies and are acquainted with management practices. There is a strong correlation between the topic of this thesis and the ultimate goal of the programme. The MoT course aims to teach students how companies can use and manage technology to their benefit as a valuable corporate resource. This means preparing future managers, consultants and entrepreneurs to develop correct strategies to social, economic and technological changes. This includes understanding the nature of the technology, knowing how to retrieve it, use it at best in a company environment and what benefits it brings over competing technologies. Secondly, the pro gram teaches students how these technological innovations in highly competitive and fastdeveloping markets. What is their market lifecycle and how they can be used in the market to obtain an advantage over competitors are some of the main themes of the course. The multidisciplinary aspect of the re search fully reflects the technical and management perspective that the MoT curriculum aims to teach its alumni. . There are three MoT requirements that need to be met by students in their master thesis, and this project meets them as such:

“The work reports on a scientific study in a technological context” The research uses network measurements as technical context and foundation to the research. The technology is studied and analysed with realworld data and realworld scenarios like the Salesforce, Netflix and Snapchat cases. The technology is used in a new context and for new purposes, i.e. predicting company performance. The social context and implications for the main stakeholders are also discussed. The main actors are the public, companies, researchers, market analysts and investors. The technological context requirements are met thanks to the use and analysis of network measurements.

“The work shows an understanding of technology as a corporate resource or is done from a corporate perspective” The research shows that network measurements predict changes in company performance. The setting of this study is not a single company but three, creating a strong corporate setting and envi ronment. In this research, we discuss which performance indicators are predicted and why; similarly, we discuss which are not. This point requirement is met as the thesis thoroughly discusses the impli cations and findings from each case. These implication and novel findings show that managers can use network measurements in a corporate setting. In particular, we identify three key implications for managers. Firstly, measurement monitoring can be used by managers to improve and evaluate their short and longterm decisions. Then, network measurements can be used as a way to assess, predict and understand KPIs, and a way to identify new meaningful ones for a company. Lastly, managers can use network measurement monitoring to obtain a market edge and competitor advantage by learning from competitors and monitoring measurements. By predicting a company’s future performance, this technology can be validly used as a corporate resource.

“Students use scientific methods and techniques to analyze a problem as put forward in the MoT curriculum” The theoretical notions and knowledge provided by the MoT programme were used to develop the research and structure of the thesis. In particular, some exams provided the theoretical background on which the research is built. • MOT2312 Research Methods: the main research approaches to research were drawn from this course. • MOT1461 Financial Management : the basic and necessary knowledge for financial notions and tools were learned in this class. • MOT1435 Technology, Strategy & Entrepreneurship : this course provided the notions for understanding how network measurements can be used as a corporate resource for performance prediction. 8 Introduction

• SEN1631 Fundamentals of Data Analytics: the course was used as a basis to present and use data in a clear and effective way. • 53.561 Corporate Financial Policy: this course (attended abroad) provided further knowledge on establishing how companies define and evaluate performance.

1.5. Document Structure

The document is strucuterd in four main parts. The first part, from Chapter 1 through 2, contains an introduction to the research and the methodology developed for this research. Chapter 3 through 5 make up the second part and contain the necessary background knowledge to develop, select and properly analyze the study cases. The third part, Chapter 6, 7 and 8 contains the study cases analysed. The last part, Chapters 9 and 10 contain the conclusions and reflections drawn from the thesis. Lastly, the report contains the Bibliography used throughout the project. 2 Methodology

The goal of this project is to study whether network measurement can predict company performance. To answer the main research question and subquestions of the proposed research, we use literature analysis and case studies. Figure 2.1 shows the general structure an hierarchy for the methodology. First, a literature review serves as a steppingstone to answer the first two subquestions, providing the background needed for the research (Section 2.1). Then, we build case studies to tackle the third subquestion (Chapter 2.2). The case study contains the core methodology of the research, the main element to verify our findings (Section 2.3). In this Chapter, we provide insights into the background knowledge needed and on which topics, to thoroughly answer the first two subquestions. In Section 2.2 we then provide insights on study cases and how we selected them. Then, we provide a detailed description of the core methodology used to study network measurements. We decided to first describe the methodology and then explain how we implement it in the case study and its structure (Section 2.4). Lastly, we provide the structure we implemented in our case studies.

Background

Case Study

Core Methodology

Figure 2.1: Methodology Structure. Shows how the case studies are built on the background knowledge on the case. Secondly, the core methodology to study network measurements is within the sing study case.

A general structure of the report is provided in Figure 2.2. This is not a description of the content of each chapter (Section 1.5) but is closely connected to it, as it delineates the order in which we proceed to analyse this problem. It is used as a guideline for our research and shows where we retrieve the answers to our subquestions and main research question.

9 10 Methodology

Problem Introduction

Methoology

SQ1 SQ2 Network Company Literature Measurement Performance Review Research

Key Network Scanning for Performance Measurements Measurements Indicators and Companies

Selecting Study Cases Company Overview SQ3 Appropriate Case Study Network Measurements

Company Performance

Cross-Case Analysis

Discussion

Conclusions

Figure 2.2: General Report Structure. We also detail where we answer the subquestions to reach a more comprehensive overview on the issues discussed.

2.1. Background

To provide appropriate context to the case studies, we study ad review the current literature on the topics related to the research. In Chapter 3, we thoroughly describe the structure on networks and how network measurements are taken. This is a more technical section that helps us build the necessary knowledge on networks and its measurement. Throughout this chapter, we understand what metrics and measurements that can be relevant for companies and ebusinesses. In section 3.6 we discuss how network measurementrelated research interacts with companies, and what are the main topics of interest. In Chapter 4, we provide a description of company performance. We refer not only to scientific literature but to other data sources as well. We identify these data sources as company reports released 2.2. Case Study Review 11 every year or every three months. These reports provide insightful data on the company and are of exceptional relevance to define company performance, as they show realworld practical data. These reports often summarise the main statistics that the company and its investors are interested in and that use in realworld settings. We consider these as the real indicators for performance that can help us thoroughly identify company performance. The literature analysis is used to answer the first two subquestions, and to be the founding knowl edge on which to build the research. The selection, structuring and analysis of the study cases is based on the ground truth built in these sections.

2.2. Case Study Review

2.2.1. Fitness of the Case Study Methodology In research, each case study should be different, as it is moulded and adapted to the research question and the context of the study. There are however guidelines to give purpose and structure to the case. [4] defines a case study as ”empirical inquiry about a contemporary phenomenon (e.g., a “case”), set within its realworldcontext”, p. 4: the aim is to provide an indepth understanding and overview on a particular topic through one or few particularly relevant cases. Therefore, the research shouldn’t only focus on the object of interest, but on its context as well. As we know, our cases are mostly based on practical experiences and processes, with little interference from theory. Rather, the lack thereof is another drive for researching this topic thoroughly, as there is no bias towards a certain result. The study case methodology seems wellversed for our research. [4] suggests using study cases when answering explanatory questions that try to understand what is going on. Furthermore, the case study is most appropriate when considered within its natural settings. Our goal is to see whether network measurement can predict company performance and tell us more of this relation [4]. We explore and understand the relationships between measurements and company performance, and we want to achieve this through realworld data and news. It, therefore, seems that the means to our end and our goal itself align quite well with the case study scenario. Study cases would allow us to isolate a company and its measurements, analysing their relation within a bounded context. We can draw attention to a single company and a single measurement, and we can learn from them [29]. It allows us to look at both qualitative and quantitative sources in detail, deeply analysing the settings of the case from which we draw conclusions. In fact, we would need to study various data sources, from financial reports to companyrelated news. Each data piece contributes to the overall understanding of the phenomenon. The detail to settings we can have in a case study helps us create a good narrative on the case contributing to the quality of our findings [30].

2.2.2. Designing a Case Study There is a lot of available literature on cases studies and how to build them. However, most of them present different methodologies and structures that do not contradict each other but rather offer similar paths. [31] describes an eightstep guide to building a study case. After bounding and selecting the cases, the researcher should start to collect the data needed, analyze it and compare it with other cases and ground truth. Then it starts a process of hypothesis shaping and theory building, by analysing contrasting literature, until theoretical saturation is reached. As others, [31] preaches the bottomup approach of case studies, and how it is possible to build theories from practical experience and data [4], [30], [32]. For this research, we mainly focus on [4]’s work on case studies as a guideline for creating our own cases. While designing a case study we first have to define what our case is: it is the bounded entity on which we intend to focus [33]. Determining the unit of analysis helps the researcher define the topic and its context [4]. To do so, the researcher must have a clear and welldefined research question. The scope and the objective of the research question are to set the boundaries of the case, along with the breadth and depth of the study. For us, the unit of analysis is going to be a single company and its own set of network measurements. 12 Methodology

Next, we have to define the type of case study we use. [4] divides study cases into Exploratory, Descriptive and Explanatory, which can then be Single or Multiple; [29] divides them into Instrumental, Intrinsic and Collective. Each of these cases has its own goals, methods and implications, according to the needs of the researcher. Our research and needs prompt us to proceed with multiple units of analysis within one context and an Instrumental take on it. Our units of analysis, the companies and measurements, are embedded within our exploratory context. Multiple case studies are carefully selected to predict and replicate similar findings across cases [4], [33] and we hope to observe similar findings in our as well. An instrumental study case provides an indepth insight into a topic but goes beyond; the case is of secondary importance, as it helps us understand something else [29], [33]. We want to discover more on the relationship between company performance and network measurement; our goal is to discover and explore this topic. This sets the context and the lenses through which we are going to look at and analyse our case studies. In Figure 2.3 we present [4] structure for multiple case study. While we do not strictly follow all the steps presented, we use this as a general guideline on which we can build our research.

Multiple Case Study Method

Define & Design Prepare, Collect & Analyse Analyse & Conclude

Conduct 1st Case Draw Cross-Case Delevop Theory Study Conclusions

Select Cases Modify Theory

Conduct 2nd Case Delevop Theory Study Develop Policy Implications

Design Data Collection Protocol

Write Cross-Case Report Conduct Remaining Case Studies

Figure 2.3: Multiple Case Srudy Method [4]. Copyright by Robert K. Yin

The defining & designing section roughly corresponds to our background section, where we present our background knowledge and the methodology to select the various cases. Then, we present and conduct various case studies. Next, we provide an overall overview of the phenomenon and draw crosscase conclusions. However, for the stage of the research at this point, we believe that it would be too early to develop or modify a theory, or to develop policy implications: we are not including these steps in our research. We point out that the general report structure, previously presented in Figure 2.2, is based on this method and roughly follows its steps. Lastly, [4] prompts us to decide whether we implement theory in our work or not. Theory helps the researcher with preexisting notions and perspectives that could support the work or that need to be challenged during the case. In our research, we decide to use and implement the theory, and, especially, the lack thereof. In fact, from Chapter 3 through 5 we study the current literature on network and company performance on which we build the tehcnincal background for the cases (as 2.2. Case Study Review 13 mentioned in Section 2.1). However, we also highlight the lack of any relevant literature and theory linking measurements and performance. We exploit this hole in the literature as a driver for purpose and justification of our exploratory approach.

2.2.3. Case Study Selection Once the goal of the research is clear and the boundaries are set, researchers need to select which cases to use. [30] describes two categories of case selection, random and informationoriented, which are then divided into subcategories. By using different selection methods, researchers can obtain different results on the same topic. Selecting a case should reflect the needs and purposes of the research. We use an informational oriented approach to select our cases, according to data availability. With a multiple case scenario, we need to select multiple cases to analyse. These cases need to be “chosen carefully so that the researcher can predict similar results across cases.”[33], p. 548. The goal is to find a connection between network measurements and KPIs across various cases. While a single case on a company could be incredibly insightful on its operations, it can result as isolated. If multiple cases are found, suggesting a similar trend, it would be possible to obtain a more general overview of our findings. Selecting appropriate cases for our research is not an immediate task. We believe that three re quirements are necessary to properly select a case for this research. Firstly, we need a company for which network measurements can be relevant, which is most likely a company operating through the Internet. Secondly, there needs to be financial data available on the company, to use it as a valid proxy for performance. The third requirement has to deal with measurements and their appropriate ness. While the first two requirements can be easily fulfilled, the third can be more problematic and important when selecting a case. Hence, three guidelines criteria and steps are defined and needed to judge whether a measurement set is worth researching and analysing. We proceeded in a linear way: if a certain measurement passes the first selection criteria, it goes forth to the next one. The steps are the following:

• Step 1: The measurements are collected and measured on a network connected to the Internet. We require network measurements as the main source of data. Web metrics are accepted as complementary to network measurements but are not our object of analysis. Chapter 3 discusses the type, kind, and appropriateness of the usable network measurement on which we based our first selection criteria.

• Step 2: The measurement set has to be potentially interesting toward the goal of the research i.e., provide some type of information regarding a specific company. The measurement should represent a certain metric related to a company and be historical. By historical we mean that the measurement is concluded in time, with a welldefined start and end date. The measurement is not ongoing anymore and there has been enough time between the measurement and the time of writing of this thesis. This allows us to have a future perspective that allows us to validate the findings. This information gathered through network measurements can provide internal information on the company and its practices or its clients. For certain companies, clients can simply be endusers.

• Step 3: Verifying that there is enough financial data and news on the company. This way, we target and study company with open and verifiable financial data and news on it.

By combining the technical background on the measurements and the company performance back ground, we expect to be able to select the cases that respect the three criteria. This selection and filtering process is crucial to only consider measurements that can have significance for our research. It is important to choose the right tools to make this selection. Once a company is selected and considered viable, we can then proceed and build a study case around it. During our research, establishing which measurements to use has also proved challenging. Another option is to collect measurements ourselves during the time period of the project. However, after appropriate reflection, this would have created other time and verification issues. Firstly, establishing what to measure, how to measure it and building the appropriate infrastructure would have proved 14 Methodology quite challenging. Secondly, the measurements would have been collected for a rather short time frame within the graduation project, after which the results should have been analysed. We could have only seen the predictive ability of the measurements in the short term, which might not even produce significant results. Losing a future perspective for the analysis and measurement does not permit for proper verification of the results of the forecasts, limiting the scope of the research. The decision was then to choose and use historical measurement, allowing us a broader and wider perspective on the case.

2.2.4. CrossCase Conclusions Our goal is to find how and if network measurement predicts company performance and prove that a connection exists between them and that is worthy of further research. This step corresponds to the ”Develop Theory” and ”Draw CrossCase Conclusions” steps that [4] suggests in Figure 2.3. Here, the researcher examines whether the findings and implications could be replicated across cases, to find a silver lining connecting and explaining them. While case studies allow a good level of generalizability of their findings [4], [30], we believe that a theoretical generalisation is beyond the scope of our research. That does not mean, that we cannot provide a valid and overall overview of the topic. From here, we hope for more and further research to happen and to prove insightful. Summarizing case studies is difficult due to the complexity of the subject [30]. Case studies often make use of various data sources and triangulation methods for analysing and verifying results of vastly different types. Bringing together complex results from case studies can make a significant contribution to the scientific literature on the topic. As we mentioned in Section 2.2.2, while it is interesting to observe the single outcomes from each study case, their goal is to contribute to a wider description of a phenomenon. The study cases proposed to contribute in their own way, with validated findings, to the validation of our assumptions and answering our research question. Since there is still little theory linking company performance and measurement, the combined outcome of the study cases should prove the need for it. The final and comprehensive overview of the topic should finally answer the research question bringing the thesis to closure. Answering the main research question does not mean providing a yes or no answer, but to provide a thorough description of this unexplored phenomenon in the world of networks. Along with a possible answer to the main research question, we share the lessons learned on network measurements and their relationship with companies. We then propose further research topics and the shortcomings of the cases analysed. Lastly, we discuss the possible social implications of this type of research, along with other possible issues, legal or ethical, on the topic.

2.3. Core Methodology

We describe here the methodology to be used in the study cases to analyse the relevance of the measurements and their predictive ability. Our goal is to find our if network measurements are insightful to understand and anticipate the company’s KPIs and to what degree. We need to develop a core method to analyse the company with the measurements and to verify the results. We propose a linear and simple way to analyse our findings and discoveries. With the help of historical data and a future perspective, we bring validity and relevance to our exploration on the subject. The Core Methodology roughly corresponds to the ”Study Case Analysis” element in Figure 2.2.

2.3.1. Measurements, Monitoring & Interpretation In the second section, we present the measurements, their origin and collection methodology; in Section 2.2.3, we already provide the selection criteria for the measurements, which then need to be interpreted. In this section, we monitor, interpret and then discuss the measurements. We remind that the measurements are collected from timepoints T1 to T2, which differ for each case. Once we have presented a specific measurement set, the next step is to observe and interpret it. We start by observing the measurements, their trends and unexpected behaviours. However, by themselves, all measurements are just quantitative observations of a phenomenon with a measure ment unit characterizing its value. The next step is to interpret them, figuring out what has been 2.3. Core Methodology 15 observed, what has caused it and what is its intrinsic meaning. The interpretation of the data should explain the patterns and trends that the measurement describe [34], while the subjective bias threat is nullified by scientific peer review. In our case, the measurements are analysed and interpreted through the lenses of the study case. The two cooperating and contributing factors to a proper interpretation framework of the measurements are its technical specifications and the study case context. Firstly, we need to understand what is being measured and when, along with the collection methodology used, to understand what phenomenon the data represent; we, therefore, provide the necessary background on network measurements in Chapter 3 to technically and safely understand the measurements. The second step is to consider the measurements in relation to the company; it entails understanding the company business, operations and activities, to determine the relevance and significance of what is being measured in such context. This double sociotechnical perspective should allow us to safely interpret the measurement and the degree of relevance and significance for the company. The inter pretation of the measurements is to be used as an input for company performance analysis. The core methodology (Section 2.3) is, therefore, a test to see whether the measurement interpretation is a valid input for company insight and to what degree. Lastly, according to our interpretation of the measurements, we discuss their implications and in which way they could improve our understanding of the company. In the thesis, we continue to refer to the network measurements as predictive of company performance. However, we remind that is their interpretation within a companycase study that is being tested as insider information. A thorough and correct interpretation of the measurements is essential and at the core of the case study, as we see in the next chapter; Figure 2.4 provides a brief overview of the interpretation framework used for the measurements.

Technical Specifications

Collection Methodology

Measurement Origin

Metric Measured

Collection Period

Measurement Measurement Monitoring Interpretation Measurement Observing the Measurement Significance for the Company Monitoring as Input for Analysis Analysing Trends and Insider Information Unexpected Behaviours on the Company

Case Study Perspective

Company Value Proposition

Relation to the Metric Measured

Company Operations & Activites

Figure 2.4: Measuremens, Monitoring & Interpretation.

2.3.2. Performance Overview In this section, we collect and present the relevant KPIs to use as a proxy for company performance and analyse it within the time frame. We select these indicators as they are believed to be relevant for the company and possibly predictable by the measurements set at hand. In this section, we present the indicators used as proxies for performance (discussed in Chapter 4) and other relevant information and news on the company that we might retrieve. This way, we create a clear understanding and overview of the company to later analyse and study through the lenses of the measurements, to predict company performance.

2.3.3. Analysis & Results As [4] suggests, we follow a patternmatching logic for analysing our findings: ”... possibility is to stipulate some pattern of expected findings at the outset of your case study” [4], p. 16. This means 16 Methodology looking through our data searching for the existence of the predictions we are looking for. The last sec tion is the most insightful for each case, as in this section observe and notice connections between the measurements and the company’s performance. In this section, we thoroughly analyse the relation ship between the measurements and the gathered information on the company. Through our future perspective, we discuss and analyse any possible predictions that can be made from the measurements on company performance. In the Analysis section, we consider all the previously presented KPIs and information on the com pany. One by one, we analyze it through the lenses of the measurements and see if there is a predictive connection between the two. This means, being able to determine whether, at a certain point in time, the considered KPI could have been predicted by the measurements. Similarly, we also look at the other factors, such as news, presented and see if the measurements could have predicted them at a certain point in time or not. As mentioned, in this section of the case study we proceed to establish the possible connections between measurements and company performance and rule out the KPIs or information that is not predicted by the measurements. In particular, we look for patterns in the mea surements that could, in some way, anticipate and forecast meaningful events or changes in KPIs. It is precisely at that point in the methodology that we see which the measurements predict which KPIs. This way, also through our future perspective, we question the validity of such relation and verify it against reality [4]. In the Results section, we verify our findings thanks to our future perspective and we present the predictive relationships that we have found. In particular, we point out what can be predicted and at which point in time. With our future perspective, we can observe and verify our findings, case by case, to ultimately provide a comprehensive overview of the phenomenon. This overview ultimately supports the existence of a predictability relationship and its worthiness of further research. The whole process and core methodology is described in detail in Figure 2.5.

Core Methodology

Overview Analysis & Results Measurement Monitoring & Interpretation Compay Overview Measurements and Measurements Specifications Company Activities and Performance Comparison Monitoring Business Relevant News Interpretation Performance Overview Existing Predictions Other Relevant Information Validating & Presenting The Findings

Figure 2.5: Core Methodology.

2.4. Case Structure

Case study flexibility and malleability is a great asset and effective quality for the type for the exploratory research we proceed with. We exploit this flexibility in building and designing our case studies, es pecially in Chapter 8. Once a measurement set and a company are selected for a study case on a certain company, we can start working on a study case. We adapted our structure to the needs of our research, keeping in mind the considerations and limitations discussed in the previous sections. This is the main structure that we follow during the development of the case study; small and minor changes can be implemented if they were to produce a more comprehensive and insightful case study. We divide the study cases into six Sections.

2.4.1. Company Introduction The first section is set up the case and its narrative. In this Section, we provide a brief history & description of the company, its operations, an analysis of its business and other relevant information. 2.4. Case Structure 17

We describe providing the reader with key information on the company for a complete overview before going through the case. We describe its products, it’s market, its value proposition and their target audience. We also provide key dates relevant to the case, such as company announcement or IPOs dates that are relevant from the analysis. This way we intend to understand its position as an e business, its strengths and weaknesses.

2.4.2. Data Sources The next step is to describe the data sources used for the case study. We describe where and how these sources were gathered, put together and represented. We consider all the gathered statistics and useful reports on the company during that specific timeframe. Next, describe also the source of the measurements used and how we obtained them. Lastly, the main sources of the news on the company used and consulted.

2.4.3. Measurements, Monitoring & Interpretation Next, we dive into the measurements themselves. We provide a thorough description of the collection method used by researchers to collect the measurements. We provide a description of the raw format in which we received these measurements and how we proceeded to analyse them. Here, we follow the guidelines described in Section 2.3.1, by monitoring and interpreting the measurements.

2.4.4. Performance Overview Firstly, we present the performance indicators of the company from the starting point of the thesis until the measurement collection begins. This Section aims to provide an overview of the company and its position before the measurements begin and are analysed. We also discuss why such indicators were chosen and why they are relevant to that company. Lastly, in this section, we include any possibly relevant news on the company before the measurements are taken.

2.4.5. Analysis& Results It is the core section of the study case and the section from which we intend to extract relevant findings. We start by presenting the previously discussed financial performance indicators and any relevant information after the measurement collection starts. Then, we discuss the measurement interpretation and see if we can find, discover and infer any meaningful relationship between measurement and performance. In this section, we explore any other meaningful findings we might make during the study case. Luckily, case studies are a rather adaptable research methodology and we intend to exploit this opportunity to enrich our overview on the topic [4].

2.4.6. Conclusions In the concluding section, we aim to wrapup our main findings from the case, We validate and verify our findings by analysing what really happened after the measurements were collected. This way we intend to validate the relevance of network measurement as a helpful predicting factor. We then wrapup our case conclusions and expressing what new knowledge and implications we have learned from the case, and how we can use it. The findings and conclusions from a single case contribute to the bigger picture that we to obtain from the cases. We might also discuss how generalizable these findings are for other ebusinesses and companies in the industry or within a similar sector. 18 Methodology

Case Section Core Methodology Contents Comapny overview, its business Company Introduction Overview and any relevant information Description of the main data sources from the case: Data Sources Overview financials, measurements and relevant news Description, monitoring Measurements, Monitoring Measurement Monitoring and interpretation of & Interpretation & Interpretation the measurements Overview of the recent financial Performance Overview Overview performance, other KPIs and influencial information on the company Performance after the measurement, disussion and Analysis & Results Analysis & Results analysis on relationships and findings Conclusions Results Conclusions drawn from the case

Table 2.1: Core Methodology & Case Structure. The contents of each case Section are here summarised. 3 Network Infrastructure, Measurements Techniques and Background Literature

In this section, we start to provide the necessary background on which we base our research. First, we provide technical and indepth on the structure of networks, along with the methods and relevance of measurements in the scientific literature.

3.1. Network Infrastructure

Networks are now made of many elements and complicated hierarchical structure, made of various elements connected with each other. To understand the measurements and the object of measure ments, we provide a brief description of the structure of the internet. The Internet is made of various computer networks connected with each other, exchanging information under the digital form of data packets, loosely referred to network traffic. The Internet is built on the main network backbone that runs all over the world, to which Internet Service Providers connect their own networks; these providers can be at a national level (NSP) or more regional (ISP). These providers are connected to the main network through Points of Presence. They provide a connection to endusers through routers: these can be governments, corporations or everyday users, as Figure 3.1 shows [35]. The main network backbone is a connection tool between the more local ISPs, providing access to individual access to users through Points of Presence (PoPs) and then routers. There is, indeed, a sort of hierarchy to the structure of the Internet. Various other elements contribute to the proper functioning and operation of the Internet. One of these is Internet Exchange Points, (IXPs), which manage connections between the major networks, sustaining the Internet backbone. Servers contribute to the correct functioning of the Internet in many ways, from resolving addresses and connection to hosting and providing content. In these categories fall DNSs (Domain Name Servers) and Content Delivery Networks (CDNs), now widely by ebusinesses. Endusers can easily access the network through cable or wireless connections, and through this struc ture are redirected where needed. Network measurement happens on these networks, and through the various elements and nodes that make up this structure.

19 20 Network Infrastructure, Measurements and Collection Techniques

WMI

3.2. Network Measurements

There is no unilateral and unified definition of what network measurements are, as there is an open discussion on the matter with different perspectives and interpretations [36]. Without getting into the more abstract implications, we begin with the measurements and the act of performing a measure ment: we identify three elements that characterize any measurement. The first the object of interest that needs to be measured. The second is a measurement scale with which the measurement can be described, through a standard unit. Lastly, we need a measurement tool that allows us to phys ically make and record the measurement. Through these three elements, we identify the concept of measurement and see how it relates to the measurement of networks and their characteristics [37]. Network measurements have as an object of interest the network, or, one of its measurable char acteristics (described in the next section). Next, the scale is defined by the network metrics measured: for instance, for network traffic, GigaBytes or MegaBits are valid scales that could be used. Lastly, for network measurements, we not only need a measurement tool but techniques as well. Traffic could be measured through sniffing, SNMP (Simple Network Management Protocol) or WMI (Windows Manage ment Instrumentation), along with other techniques described in Section 3.4. Researchers, however, quite often develop and produce their own hardware and software as measurement tools for networks. The main issue with a sound and unequivocal definition of network measurements are that various numerous metrics, connections and relations exist between the elements in the network. We already defined in Chapter 1 networks measurements as a “collection, interpretation and modelling of empirical Internet data”[12], p. 31. We further expand and the latter with the knowledge that these measure ments are as well “a collection of network data describing finite metrics and relations between network nodes”. This data collection is often accompanied by relevant metadata such as timestamps and traces. A certain measurement, in fact, has a lifetime value, beyond the initial purpose of the measurement; the data could turn out to be useful with a different perspective in mind later on for which metadata is essential [38]. This more comprehensive and inclusive definition tries to capture the wide range of possible measurements. In fact, various measurement specifications, common practices and methods exist, one for every single metric, record or characteristic. Network measurements started to become relevant in the scientific community when two conditions in the early days of the Internet were met. The first was met when the Internet was large enough to produce interesting and significative measurements. The second, when networks were so many that a more accurate and local perspective on the network was needed to detect errors and failures, or to plan for future development and deployment [14]. This happened in the early ‘90s when the scientific community started understanding the potential of measurements and the regulatory authorities, such 3.3. Measurement Metrics 21 as the IETF or NWG (Internet Engineering Task Force and Network Working Group) started address ing this issue. Requests For Comments (RFCs) were and still are the official guidelines for correctly operating and deploying elements on the Internet, and we turn to them to thoroughly define network measurements. When measuring networks there are also other considerations to be made. RFC 1262 warns against privacy breaches and unethical measurement collection, where security and privacy of the unaware enduser could be compromised [14]. The measurements shouldn’t also put an inappropriate amount of stress on the network and is less invasive as possible when collecting data. Proper measurements have to be precise, accurate, rich in metadata and avoid wrong interpretations [38]. Another issue may arise when choosing the appropriate vantage point, as it can significantly skew measurement interpretation and produce untruthful results. Lastly, [38] warns against unprecise calibration of the instruments, which can result in nonrepresenting outliers and spikes. Careful network measurement is important, for the validity of the findings and the interpretation of the results [12].

3.3. Measurement Metrics

On a network, there are many types of elements that can be measured, each with different techniques, hardware, software and purposes. In this section, we provide more technical background on the object, use and methods of measurements. Overall, various metrics and statistic can be measured on a network using its various connections, protocols and elements. These are just some of the main objects of measurements in network research, showing what are the possible metrics that can be used in our case studies. We assume that most of these, or a variation, could be used in relation to a company for the goal of this thesis.

3.3.1. Metrics Latency Latency is the time that a data packet takes to travel from the sending to the receiving host. Various other metrics could be included in this category, such as oneway packet loss described in RFC 2679 [39]. The most notable is the roundtrip time (RTT), which is the twoway trip of a packet [39]. There are various types of delays in a network (processing, transmission, propagation, queuing), and RTT can help establish its causes [40], [41]. Assessing RTT is useful to establish the causes of endtoend faulty applications, the minimum delay due to propagation a and transmission, or assessing path congestion.

Throughput It can be defined as the actual quantity and raw amount of data actually transferred through a link, node or path at a certain point in time [40], [41].

Bandwidth Bandwidth is the maximum volume of information per unit of time that a transmission medium can handle [42]. By medium, we intend an Internet connection, either physical or wireless. It uses bits per second, b/s as a unit of measurement; 60 Mbps or 60 Mb/s, for instance, are speeds that represent 60 million b/s. There are three metrics that can be measured associated with bandwidth. These are capacity, or the maximum possible bandwidth through a certain path; available bandwidth is the unused capacity in a link; lastly, bulk transfer capacity is the achievable throughput of a TCP connection [40]. However, measured bandwidth is important for downloading and uploading files, and a clear understanding of bandwidth requirements from a user has been proven often useful. [43] describe in RFC 7594 how bandwidth between two users can be measured by placing between their traffic flow a measurement agent connected to a controller and a collector, in order to store and send the data to an analysis tool. 22 Network Infrastructure, Measurements and Collection Techniques

Packet Loss Rate Measures the rate of packets that, while travelling from point A to B, do not reach the destination, and it can happen for several reasons, such as corrupted (missing) packets [41]. It is accurately described in RFC 2679 [39] with similar implications to RTT. These packets are lost in transit, which is often quite different in opposite directions, can greatly affect applications like video streaming [40].

Intrusiveness This is a more peculiar type of metrics. Some network measurement, as described briefly later, create unwanted load on the network that can perturb it and affect measurements [41]. However, it is a rather unexplored metric and new methodologies on how to reduce intrusiveness should be developed.

Traffic Traffic can be seen as the network’s activity. Traffic is an observable stream of packets passing through an observation point in the network at a certain time interval [43], and RFC 2722 describes how to effectively deploy a traffic measurement architecture [44]. Traffic is measured to see how much data is going through computer networks. The abovementioned metrics, such as bandwidth, throughput or packet loss can be used to perform traffic analysis, for instance, [40]. Knowing traffic composition helps in planning for capacity or security. Network traffic can be of various types: Web, peerto peer applications, DNS lookups, email, content streaming, automated network operation, encryption certificates such as SSL, games, chat or multicast traffic.

3.3.2. Possible Relations to Indicators In this section, we discuss how the abovementioned metrics could be related to indicators and statistics used by companies and in relation to them. Most significantly, these metrics could be related to the way customers or users interact with the company’s service online. For instance, latency between users and a company’s website or service could be extremely high or low and have very different consequences and implication. If latency is low, the client is probably satisfied with the speed of the response, while if very high, it might be deterred to reuse that service. A large dataset on latency could provide information on average time users have to wait, showing which users have higher latencies, where and when. This could prompt companies to improve their infrastructure in order to retain customers and not lose them to the competition. Or it could highlight a serious problem of the company to investors ad analysts, changing their opinion on the company’s value proposition and operations. Another quite insightful metric could be traffic, as it could provide information on both users and providers. Traffic monitoring could show how many users a service has, how many users and how much they use a specific service. Traffic could tell investors a lot about user behaviour: for instance, it could immediately show if users are operating and using a new feature or if it went unnoticed. Traffic could also show how much time is spent on a specific platform or which company is attracting more traffic. We believe there is indeed potential for these metrics to be linked to companies and users, and provide meaningful insights on their business and operations.

3.4. Measurement Techniques

Once these metrics have been defined, there are various techniques to collect them for further analysis. Internet measurements are of three types, passive, active and hybrid. To describe these techniques, we use as the main source the Request For Comment 7799, the official guideline and definition provider for the structure of the Internet [45]. The literature firstly refers to an “Observation Point” or “Measurement Point” from which the packets (Internet data) exchanged can be observed, and therefore has to be an identified point in the network path. 3.4. Measurement Techniques 23

3.4.1. Active Measurements This technique makes use of probes to generate data which is then sent to a receiving host with the required measurement. These packets have dedicated fields for the measurement, often a sequence with a timestamp, which is then stored and analyzed [41]. The origin source and destination are usually known a priori, which both often already know what kind of packet to expect; in Figure 3.2 [41], p. 1376 we can see an example of active measurement. Lastly, these measurements affect network performance by creating an unwanted load, and such influence should be minimized or considered.

Figure 3.2: An example of an Active Network Measurement [41]. Adapted from “Active and Passive Network Measurements: A Survey” by V. Mohan, Y. R. J. Reddy, K. Kalpana, 2011, International Journal of Computer Science and Information Technologies, Vol. 2 (4), p. 1376. Copyright by Mohan, Reddy, Kalpana.

3.4.2. Passive Measurements First Observation Points are selected and then monitoring devices are put in place; the measurements only observes the undisturbed and unmodified packet stream of interest. They record packets with specifications that match their filter, without perturbing the network. The packets of interest are then sent to a collector, which in some cases, unfortunately, can produce unwanted noise in the network. Realtime analysis can be performed, even though passive techniques are more challenging than active ones [40]. Measurements are filtered, classified and then sampled for further analysis, and the data collected is often much greater than active techniques (Mohan, Reddy & Kalpana, 2011). In Figure 3.3 [41], p. 1375 we see how recording probes send the observed traffic to a database from which it is then collected and analysed. The measurement is then stored for analysis and later on collected.

Figure 3.3: An example of a Passive Network Measurement [41]. Adapted from “Active and Passive Network Measurements: A Survey” by V. Mohan, Y. R. J. Reddy, K. Kalpana, 2011, International Journal of Computer Science and Information Technologies, Vol. 2 (4), p. 1375. Copyright by Mohan, Reddy, Kalpana.

3.4.3. Hybrid Measurements These measurements use a combination of active and passive methods. For instance, a specific packet stream can be actively generated (from probes) and then it is passively observed which has been modified by the presence of the load. In general, it combines the active generation and the passive observation of a certain stream, that might have been modified or treated on the way. In Figure 3.4 [41], p. 1376 we present how active packets are sent from a probe and the traffic is passively monitored on its journey to the destination. However, they have the same problematics of active and passive techniques 24 Network Infrastructure, Measurements and Collection Techniques

Figure 3.4: An example of Hybrid Network Measurement [41]. Adapted from “Active and Passive Network Measurements: A Survey” by V. Mohan, Y. R. J. Reddy, K. Kalpana, 2011, International Journal of Computer Science and Information Technologies, Vol. 2 (4), p. 1376. Copyright by Mohan, Reddy, Kalpana.

3.4.4. Vantage Points and Techniques These techniques use specific software and hardware designed for measurement creation, capture and further analysis. The most common method for network measurements collection is through a vantage point: an advantageous access point to networks that allow measurement collection. Hardware and software are placed in such vantage points to perform measurements or observe traffic. A vantage point is often a network element and hardware such as an IXPs (Internet Exchange Points), a DNSs (Domain Name Systems) and, more recently, a CDNs (Content Delivery Networks). These vantage points have proven essential for network research but are often expensive to obtain or have limited accessibility. Computer networks are growing in size and relevance, along with measurement tools and related technologies, like probes [17]. The most wellknown tool is ping, available on most machines, which measures connectivity, RTT and packet loss by sending requests to the selected host. The second, and builtin every operating system, is traceroute, which resolves the path between two hosts [40], [41]. Bandwidth estimations are made through various techniques like VPS (Variable Packet Size), PPTD (Packet Pair/Train Dispersion), SLoPS (SelfLoading Periodic Streams) and TTOP (Trains of Packet Pairs), which we do not mean to discuss further [40]. Network delays of various types can be resolved with WAMP (Oneway Active Measurement Protocol) and TWAMP (Twoway Active Measurement Protocol). Other tools, that operate at deeper layers are Ethernet OAM, UDLD, Link Layer, MPL/LSPPing [41].

3.5. Measurement Uses

Apart from effective measurement techniques and practices, what is most relevant to us is to un derstand for what purpose these measurements are taken, and what do researchers do with them. Measurements are taken when a researcher needs to explain something, or to make decisions based on the measurement. Accurate and welldesigned measurements help researcher achieve their goals: which, as the current literature shows are extremely varied in nature with different implications. A brief categorization of the main themes helps us understand the common trends in research and where in stead there is a clear gap. One of the most recurrent themes is topology, which aims at describing and characterizing the structure, organization and hierarchy of networks. Researchers study and monitor Autonomous Sys tems (ASs), router traffic and hosts to determine and explain network infrastructure and relationships [46], [47], [3]. Another recurrent theme is security, from spotting weaknesses, estimating DoS (Denial of Service) attack or suggesting improved security systems and authentication methods [48], [49]. Presenting their measurement tools and their operative capabilities is another common trend is for re searchers. Without digging deep into the literature, we find examples in Tstat [2], EVA [22], KaTaLyzer [17], Sting [50] or NIMI, Cap, Iperf, Cisco NetFlow or Sting just to mention some [40]. These tools can 3.6. Network Measurements and Companies 25 capture most metrics, from bulk transfer capacity to traffic flows and bandwidth [41]. These metrics are then used for the last category of network research: performance and monitoring the network. From here, it is possible to have a better understanding of networks, traffic paths and characteristics used for network diagnostic and analysis. These are crucial for the longterm health and performance of the Internet, as operators need to know how and where to improve and upgrade the network. Pre cise, accurate and rich measurement is needed for this purpose [38]. Which is not a simple task at all due to the global scale of the internet and that Internet protocols and applications were not built to support and capture such data [12].

3.6. Network Measurements and Companies We then proceeded to look for research closer to our case, on how network measurements are used to reveal something about companies, without the specific purpose of predicting performance. Most of the literature deals with companies whose main value proposition is on the Web or comes from Web usage. Most of the literature deals with discovering their operations and activities on their back end or understanding their traffic.

3.6.1. CDNs There is a large section of the literature dealing with companies and Content Delivery Networks. These are elements in the network designed for efficient delivery of Web content, where the CDN manager is the provider and the content provider is the client [51]. CDN companies sell space on their servers to Webcontent providers, bringing their content closer to users; they improve performance but do not eliminate the need for Webhosting. For instance, [19] show how Netflix’s Web section is provided with both internally and by Amazon Web Services, but videos are distributed from three different CDNs. Their rise in use and importance mark a significant shift in the internet industry and ecosystem, as companies rely on them to deliver quickly online content [52]. CDN providers are studied, such as Akamai or Limelight, to discover how they improve the quality of their product, where they place their CDNs and their service performance [27], [28], [20]. Researchers study technical aspects and practices on these companies, on how their system redirects traffic through the shortest paths, measuring delays, uptimes and availability. [24] for instance, show and describe the topology and reachability of Akamai’s CDNs and ASs. CDNs are often coupled with their clients: [19] also show how Netflix optimized their content delivery through a dynamic CDN assignment for every customer. Before each video is loaded, Netflix performs a lightweight measurement that chooses the best performing CDN for that user and considering that they use three different CDN provider, this is a nontrivial endeavour. Most studies on companies and CDN are based on IP addresses and use DNS records (such as CNAME and IPs) to study these internet players. A wider perspective is taken by [27], describing how the CDN landscape is constantly evolving and how, for proper monitoring, recharting should happen every couple month.

3.6.2. Traffic Monitoring Another theme in the literature uses company traffic as the main measurement source; quite often, this traffic is usergenerated as it is the easiest to capture and monitor. Early research from [53] de scribes how higher website traffic leads to higher returns compared to competitors for many kinds of companies. However, their return is almost twenty years old and deals with a Web still in its infancy. They also consider only website traffic, which makes it limiting, at the time of writing, as a reference for company and networks. An interesting example comes from [54] where they provide an extensive study on the Italian Internet. From passive traffic measurements, they study FTTH and ADSL traf fic composition, establish the popularity and the quantity of traffic from the major Internet players. [55] reveal YouTube’s delivery infrastructure, after Google’s acquisition and restructure. Other studies show how from IP traces it is possible to characterize and study new multimedia traffic and therefore trends, such as VoIP and IPTV [56]. This is a closely related theme to companies, as it could show the emergence of new trends. Others bring various players together, such as YouTube, Google and European ISPs, analysing how this multiactor stage deals with loadbalancing from the video provider [57]. YouTube also draws attention on its traffic, measurement patterns, file properties, popularity and 26 Network Infrastructure, Measurements and Collection Techniques referencing characteristics, and transfer behaviours and suggest different caching practices to improve performance [58]. We see then that traffic is also analysed to study how companies operate and manage operations on their side.

3.6.3. Novel Use of Network Measurements We here described the main themes and purposes of network studies related to companies. Re searchers try to identify characteristics of the architecture and structure of Internet players; specifi cally, the only companies considered are among the largest that operate on the Internet. These large companies have more complex practices and behaviours to discover and leave a much larger footprint on the Internet, making itself more discoverable. Most network measurements try to understand what happens “behind the scenes” and show how these companies are upgrading, improving and changing their services. It is interesting to research for companies in the same market, to have an idea of other companies’ best practices and strategies, on how they obtained a considerable market advantage. We have already seen in Chapter 1 and 3 the main uses of measurements, from describing perfor mance to security issues, network topology and addressing error or liabilities on the network. When then used in relation to specific companies, we see how there is a tendency toward the technical analysis on the operations of these companies and how they use and operate the Internet. We have then identified a clear knowledge gap in the literature. With the evermore growing im portance of the Internet and the networks on which it runs, there is little relation to the companies operating on them. The previously mentioned studies are indeed a first step towards creating a con nection between ebusinesses and network measurement. However, we believe there could be many more topics from on which network measurements could be used as a knowledge creation tool. In this research, we decided to tackle the company performance issue. We see that these measure ments are not used nor considered by investors or market analysts, and there is no relation between measurements and company performance. The literature describes technical practices with few impli cations for markets, users, investors and other social topics, without attempts to fill this gap. Research shows that measurements are used for various purposes, even with the connection to Internet com panies and ebusinesses, but it is still an almost unexplored section. Many topics could be explored regarding network measurements and companies. While some literature deals with network and com pany services performances [27], [28], [52], we propose a novel and different perspective. Using networks as predictors for company performance is a new and unexplored use of measurements. We see whether there is a relation between the two constructs and possibly build a general theory around it. Ours is also the first attempt to fill this gap, and it is intended as a stepping stone towards further research. Being able to show the existence of this relationship helps us closing this gap. 4 Company Performance

In this Chapter, we attempt to define company performance and expand our understanding of it from Chapter 1. We attempted to find and come up with a onesided and unambiguous definition, but the result was disappointing and unsatisfactory. Rather, we had the certainty that company performance is a composite assessment that needs to be looked at and understood from different perspectives. This definition leaves room to interpretation and adaptation to each company and its performance requirements. First of all, performance could represent and display how well a company is doing towards its goals and the market. Each company operates in different markets and has different goals that can be measured in different ways. Different metrics are used to measure success, according to what is most relevant to the party involved or to its business model, line and market type. For instance, a social media app values greatly its number of users, while a car manufacturer counts how many vehicles it sells. It is clear, then, that company performance is not a single statistic or metric, but a miscellaneous assessment of various parameters that are considered together each time a company is analysed. Traditionally, the main parameters considered for performance are financial and marketrelated, but the range of business analytics for company evaluation is exceptionally wide. Other factors can include social responsibility, executive pay, goodwill, product or service quality, research and innovation, customer or employee satisfaction. These are not strictly financial indicators but parameters that have acquired increasing importance over the years: they are both qualitative and quantitative. Company performance has to look at the company as a whole, which means considering the various factors influencing it and the parameters it values most. Each business estimates its performance differently from each other, according to its business model, market, product and financial position by means of various and specific indicators. The literature identifies these indicators as Key Performance Indicators, or KPIs, that managers and companies can use to evaluate their business [59], [60]. Representing a whole company with a single indicator is extremely reductive and rather ineffective. Various indicators need to be considered according to a company’s activity and each one needs to be given a different weight for each company. In the following section, we describe the sources and the main indicators identified for performance in this research.

4.1. Key Performance Indicators

In the previous section, we have therefore identified the construct of company performance with the concept of a set of KPIs. It is, therefore, necessary to understand what are KPIs and how they can be monitored and measured. [61] reminds us that ”an indicator system should provide a measure of current performance, a clear statement of what might be achieved in terms of future performance targets and a yardstick for the measurement of progress along the way”. We can the define KPIs as a ”benchmarking system to effectively assess and evaluate a certain feature or metric”. This simple solution is used as a definition for KPIs through the course of this thesis. These indicators are values used to measure company effectiveness and monitor the progress towards a certain goal or objective

27 28 Company Performance in a quantitative way [62], [63], [64]. However, there are various types of KPIs, depending on their object of interest and the level of objective they refer to. For instance, some KPI could refer to the company as a whole (macroscopic) while other could focus on a single product or business line (microscopic), they can be leading or lagging [63], [62]. There are therefore various kinds of KPIs and each company gives a different weight to different KPIs, which can itself change in time as markets evolve. Analysing current literature, websites, and company quarterly reports, we identify factors, statistics, metrics, and elements that can be useful for better understand what network measurement tell us about company value, behaviour, and strategy at a certain point in time. These can both be strictly financial and nonfinancial metrics, as it emerges from an industry analysis. This is no easy task, as the literature clearly agrees on one fact: different ebusinesses use different performance metrics [65], [66], [67], [68]. Each business embraces various indicators and values, from the traditional financial ratios to newer custommade indicators for such a certain market segment. Companies identified and developed new indicators relevant to their ebusiness market, and they are willing to embrace them. Guidelines to choose, create and use KPIs exist and it is then up to the manager establish which and how to use: all effective KPI should go through a design phase. For instance, [63] provide the SMART criteria: a KPI has to be Specific, Measurable, Assignable, Realistic and Timebound to be used effectively. The company use KPIs as they are easy to monitor, implement and use effectively: this makes them a great tool to look at when assessing performance and progress. In our research and cases, we deal with ebusiness of various sizes operating on the Internet but in highly distinct industries and sectors. Therefore, we provide an analysis of various types of KPIs that could be used in our cases. We divide KPIs into 3 distinct categories. The first is Financial KPIs, the most traditional ones, that include standard financial metrics to look at the company from a macroscopic perspective. The second category presented is made of Webrelated KPIs, were special attention is given to those metrics used by companies that mostly or entirely operate online. Lastly, we include in the General KPIs section all those KPIs that do not neatly fit in any of the two abovementioned categories and that do not have an immediate relation between them.

4.2. Financial KPIs

Accounting and financial information can be of various and different kinds, from ratios to financial state ments to analyst predictions. They generally refer to the financial wellness or distress that the company is in, providing an overview of their properties, assets and liabilities. For Internetbased firms, this in formation is sometimes less relevant or indicative: “Internet companies have few tangible assets and their massive expenditures on the allimportant intangible assets are generally expensed rather than capitalize”[69], p. 352. Using financial literature to company reports and financial statements from big Internet companies, we extract what these companies and the investors report and find relevant to state, analyze, mention and predict. Quite often these statements are accompanied by an open letter to stakeholders, which might also contain information for future strategy. Since various documents are produced by companies at different times, it is imperative to clarify terms and differentiate them properly.

4.2.1. Indicators & Sources • Annual Reports: Released from the company at the end of the fiscal year, these are one of the main ways of the company to communicate with investors. They contain management reports from top executives (CEOs), describing and explaining what they have achieved, financial metrics such as sales growth and discussing what future plans and intentions might be. It has now be come increasingly popular for tech companies to live to stream a conference call (and then upload a transcript) or interview and discuss the company, while investors and analysts can follow live. Lastly, at the core of the report is a financial statement containing instruments like the balance sheet, the statement of cash flows, the retained earnings statement and the income statement 4.2. Financial KPIs 29

[70]. These reports are often mandatory and follow the GAAP standard (Generally Accepted Ac counting Principles), providing information on the state of the company and its business [71]. Next, to it, companies may provide financial analysis and an auditor’s report. • Quarterly Reports: Issued every three months, with the latest company results. Similarly to annual reports, they include financial company data accompanied by a presentation where man agement presents key performance data from the past three months to investors and analysts. These reports are crucial as they often include estimates of key metrics for future quarters: the goal of a company might be to beat its own estimates of growth. These reports are usually less extensive than annual ones. Not all companies release these reports at the same time, due to differences in fiscal years.

We considered some quarterly reports from companies coming from various Internet markets: Al phabet, Salesforce, Netflix and Snapchat, which are publicly available online [72], [73], [74], [75]. Reading through company reports, executive letters and announcement we decided to select some possible estimators, or factors, that companies use. We describe the selected estimators here, with the complement of the definitions found in the book “Corporate Finance” by and Berk & DeMarzo [76].

• Sales: sales are defined as the raw earnings directly coming from the transactions that a certain company has made in a certain period, without considering costs. Some companies also use net sales as a metric, which refers to sales minus price reductions that customers have access to (allowances, discounts, and returns). In general, there can be various types of sales defined in a company report, coming from different sections, departments or geographical areas. • Revenues: another of the most often used metric is revenue, which is quite similar to sales. However, the latter refers directly to the profits made by direct sell. Revenues instead are more comprehensive; it indicates the aggregate money received by a company through various sources, sales included. While sales and revenues are indeed related, they are not the same thing, and for different companies, according to their business model and incomes, these factors can have different relevance. • Operating Income: OI is defined as the profit that the company made stripped of operating expenses, such as selling, marketing, distribution, and development. Stripped of costs, it re flects quite closely the “cash on hand” that firms have, before taxes and interest. It’s growth percentages, both quarterly and yearly are also relevant. • Net Income: stripped of all costs and expenses, this is the actual earnings that a company made over a period. As mentioned above, growth percentages are indeed relevant for the intent of this thesis. Net income, operating income, and sales are a direct way to check company earning in a broadway. For our analysis, these “cash” metrics could be indicative, but there is an obvious main issue. Depending on the company size, products, and business model, these metrics might not have at all a relation with the measurements we are considering. For instance, tech companies are usually quite short on cash, as they often reinvest their resources to grow. • Assets: assets is a particularly broad and general financial term that includes all different kinds of resources that a company owns. These could be cash, investments, supplies, buildings, equip ment, vehicles, and technology. Most of the companies that we consider also have extremely valuable but intangible assets such as intellectual capital [77] and knowledge expertise. It is problematic as such assets are unquantifiable and are not reported in the financial reports. On the other hand, assets could be less explanatory, as they do not reflect the full value of certain ebusinesses that are more focused on software or online services. • Financial Ratios: another possible performance indicator is made up of the financial ratio cate gory. Ratios can provide businesses with valuable tools to evaluate and analyse a specific aspect of the business and are made by dividing one financial figure with another. They are used for an overall financial analysis of the company and are of various nature: profitability, liquidity, leverage, efficiency or risk ratios. We use the most significative for each company, if relevant; candidate ratios for our research return on investment or net profitability, while we do not think debt or asset ratios to be insightful to represent and predict performance. 30 Company Performance

• Earnings Per Share: EPS are calculated as net income minus preferred dividends divided by average outstanding common shares and signal is a good measure for the health of the company. It is a financial ratio signal that a company can pay out money to its investors and investors usually look at this parameter across a certain industry. It influences a stock price and shows how profitable a company is. It is, however, a rather specific ratio, as certain industries such as the IT sector do not, for instance, give out dividends but prefer to reinvest the money. Companies can rework and reorganize the number of their shares from one quarter to another, by stock splits or rebuying share, which can influence this parameter in certain occasions. Often described in quarterly reports, unexpected sharereorganizations are often explained in detail by the company.

4.2.2. Considerations on the Proposed Indicators These are the main financial indicators considered for our research and to see if they can be, at least partly, predicted through network measurements. We decided to discard some other choir relevant indicators in finance that are however not appropriate for the scope of our research. One of the purposely excluded factors is stock prices. Stocks reflect the value the market values a certain company and, consequently, the performance of the company [78]. It would appear to be the most immediate performance estimator, and, in a certain way, it is; but there are multiple reasons for our decision. If we were to consider stock prices as estimators, an almost infinite list of other parameters and influencing factors should be considered. We would need to consider the general trend to the market, company correlation to the market [79], investor sentiment, unexpected news that might swing stock prices, investor sentiment and many other factors. Company performance is therefore only partly reflected in such a parameter, drastically diluting the link between measurements and performance. Due to all the social, financial and economic implications that stocks have, we discard this parameter as too problematic and not relevant enough. If used, as it could have a limited degree of significance, it is going to be done in a careful way and mindful of the above considerations. Traditional financial measurements have long been criticized to be too shortterm oriented, reflecting past performance and not the current one [66]. However, there is not a tendency to overhaul more traditional performance metrics [67], [68], rather, to use newly developed metrics as a complement in strategy and decisionmaking process. Despite their shortcomings, traditional performance metrics are still widely used, still remarking their relevance, while new indicators are used to capture other aspects of the business. We then consider both traditional performance metrics (financial and accounting ones) as well as new ones (Web metrics or KPIs) as possible proxies for performance. When conducting the case studies, we link these to network measurements, according to their types of businesses and operations.

Performance Indicator Formula Use Sales Number of Units Sold Measures total sales Direct cash coming Revenues Number of Units Sold x Sales Price to the company, signals growth Cash coming to the company,stripped Operating Income Revenue Direct Costs Indirect Costs of most expenses, signals profitablity Net cash coming to the company, stripped Net Income Revenues Total Expenses of all expenses, signals profitablity Cash + Investments + Equipment Everything the company Assets + Inventory + Real Estate + owns with monetary value Accounts receivable + Intangible Assets Can signal profitability, Financial Ratios Indicator 1 / Indicator 2 liquid cash availability, growth or other factors Earnings Per Share Total Earnings / Outstanding Shares Profits per share basis

Table 4.1: Financial KPIs. 4.3. Webrelated KPIs 31

4.3. Webrelated KPIs

The knowledgeintensive type of businesses that operate on the Internet, standard and traditional accounting measures can fall short and explain only part of company performance. This is why other adhoc metrics were developed by managers to monitor their business performance and make decisions on their future strategy. These ebusiness related indexes are often called Web Metrics and KPI’s, Key Performance Indicators. Web Metrics include indicators strictly related to website performance, which is indeed relevant for ebusinesses whose main value proposition is delivered through the website, such as ecommerce platforms. KPIs are similar metrics used in the ebusiness market, but they are not websiteexclusive: they are used for apps, services from a wide variety of companies. KPIs are more businessspecific and have a narrower scope that captures a particular aspect of an Internet related business. They can capture the financial, customer or internal ratings of a company. There is an overlap between KPIs and Web Metrics, as certain factors could belong to both categories, but it should not be an issue for representing company performance over time. We then proceed to identify some of the most relevant Web metrics for website businesses. Early research by[69] uses the Nielsen NetRatings to understand the performance of a website. [65] takes a more casual perspective in analysing which factors influence website performance, while [80] indicates factors to monitor for the longterm success of a website. The main Webratings that can represent website and company performance are: • Unique Audience: the number of unique Web surfers who have visited the Web property during the month. This index can also be identified as a number of unique visitors. • Universal Reach: the estimated percentage of the universe of Web surfers who have visited the Web property during the month at least once. • Customer Number: the number of users and subscribers that use the website or its services. • Web Sales: the number of goods sold through the website, or the amount of money made through it. It measures performance through a monetary index. • Website Traffic: it is a quite undefined metric for websites, as it can be made up of new users, subscribers or page visitors over a time period, but is often regarded as one of the most immediate indicators. It could be measured hourly, daily or monthly. • Average Time: indicates the average time spent on the Web property per person per each visit. • Bounce Rate: is a percentage that shows how many visitors leave a website immediately after landing on the main page. This entails not exploring the content offered and the failure to retain the visitor. • Conversion Rate: is the opposite percentage to bounce rate. It shows the percentage of visitors from potential users become actual subscribers; in this case, the website succeeded in obtaining a customer and retaining a visitor. • Website Satisfaction: this metric indicates how satisfied visitors are on a certain website, and it could point out to desired or unwanted features on a website. These metrics are significantly website sensitive and not relevant or applicable to every website in today’s Internet but can be applied and used in various types of companies that partially or totally rely on the Web. [69] further advance their study by defining three other characteristics: reach, stickiness, and loyalty to a certain website. They represent the ability to attract new users to a website, to keep them on the page during the visit and to generate repeated visits, respectively. With today’s Internet giants, we understand that website metrics are far from explaining company performance. Despite still being relevant for certain websiteexclusive enterprises (such as video streaming platforms or ecommerce), these have lost part of their role. Eenterprises have been diversifying, expanding and acquiring value through other means. These Web Metrics are not, as mentioned already, network measurements, but it could be possible, from measurements to estimate some of these Webrelated statistics. These Webmetrics are rising in importance and therefore considered as possible performance estimators. 32 Company Performance

4.4. General KPIs

KPIs have similar implications to Web Metrics but can also be used for other types of ebusiness and often have a narrower scope. As already mentioned, some KPIs can also be considered as Web Metrics, and there is a sort of grey line between them. KPIs are sometimes directly taken from traditional financial metrics. One such example are conversion rates, profit measures or sales, but with a focus on the company activity; for the sake of brevity, we do not repeat them. These metrics can be financial, customerrelated, operational or can come from marketing; KPIs need a measure, a target, a data source and a reporting frequency, daily or monthly for instance [81]. The following are some of the most relevant KPIs that can be used [81], [82], [63], [62]:

• Customer Lifetime Value: CLV measures the value of a customer in the long term. It helps narrow down channels and means to obtain the most profitable customers.

• Customer Acquisition Cost: CAC is calculated by dividing the number of new customers by the acquisition costs in the relevant time frame. These costs often help identify effectiveness and weaknesses in marketing campaigns. • Net Promoter Score: NPS represents long term company growth by identifying how much customers are willing to recommend such a service. Increases or decreases in such KPI are valuable to understand how much a company value proposition reflects the market demand. • Customer Support Tickets: this KPI is quite specific to only some businesses, but helps identify how often customers require support and how effective it is. • Active Users: this KPI can be considered over any interval of time, and counts how many users use or take advantage of a certain service. The time frame and the minimum activity to be considered as active are defined by the KPI monitor. • Employee Satisfaction: satisfied employees are more productive and positively contribute to the longterm growth of a company, it is also a great metric to attract prospective employees. • Subscribers: is a quite specific metric that might be used by almost ever ebusiness category, but only relevant for some, depending on their business model. It is clearly closely related to a number of customers but can also help capture other aspects of company performance. For companies with a monthly subscription fee, for instance, proving that a certain measurement can predict an increase MAUs, would be extremely meaningful. Such a metric should be considered when coupled with a meaningful business model.

These are but some of the KPIs that are being used, or at least complimentary, as proxies for company performance. One of the main concerns with KPIs is that is mostly used by SMEs and might be unrepresentative for bigger companies or corporations. This is mainly a cause of the higher number of products or services that bigger companies might offer, and therefore a single metric might be limited in representativeness. However, using KPIs at a microscopic and macroscopic level, across a corporation, might provide a solution to the issue. This varied list of indicators tells us that each company might find one parameter more indicative that another, and therefore understand more about it. While we do not use all of the abovementioned metrics, it is indeed a practical guideline to identify company performance in the ebusiness market. 5 Case Selection

Following the methodology, we proceed to select the cases for further analysis. Firstly, we present the measurement datasets considered for the first step.

5.1. Considered Measurement Repositories

Various measurements sources were scanned for and analysed, referring to the background knowledge on ebusinesses and network measurement that we built in Chapter 3, 3.6 and 4 . Almost as many were discarded, as ”The world is full of observations that can be made, but not every observation constitutes a useful piece of data.”, [34]. A large portion of the considered datasets come from research institutions and published papers. Below here we briefly describe the measurements considered and its source; most of this data is open source, and the link to the data is provided in the reference.

• CAIDA: The Center for Applied Internet Data Analysis, provides a wide dataset of geographically and topologically diverse locations [83]. They provide both open and closed datasets of ongoing, complete and onetime measurements. Their repository includes active and passive network measurements on AS and their relationships, measurements on topology, security and DDoS attacks, IP topology and census, IXP traffic and anonymised Internet traces. Most of these measurements are about Internet topology, anonymized traffic or infrastructure. Despite the various gathering methods, this type of measurements are abundantly technical and describe the structure of the Internet. There is little to no relation or mention to companies in the data. • WIDE Project: The Japanese project WIDE and it’s working group provide various traffic traces of IXPs, root servers response times and loss rates from various probes all over the world [84]. These traces are 15 years old and characterize the amount of traffic considered. • RIPE NCC: Réseaux IP Européens Network Coordination Center is a research platform and in stitution, with active probes installed all over the world. They offer open datasets [85], mostly on the state of IPv6 deployment or other rDNS measurements that proved to be uninteresting. They also set up the Atlas project, an open platform for active measurements with probes for active measuring all over the world [85]. It provides, mostly, metrics and statistics on network performance, and is a great tool for researchers to use but that publishes few measurements on its own. • OpenINTEL: This platform provides various applications for network security and research. It contains open DNS records of the top 1 Million ccTLDs and gTLDs [86]. • CNetS: The Center for Complex Networks and Systems Research, offers some Internet traffic metrics [87]. Despite containing some measurements on social media, such as Twitter, their datasets, which looked interesting at first, mostly contain data relevant for more social network and mapping research.

33 34 Case Selection

• Hurricane Electric: Their webpage contains various measurements on the state of the Internet and a repository of all IXPs in the world [88]. Despite the possibility to check the traffic going through many IXPs all over the world, it is impossible to differentiate, as most of the statistic describes ASs and IPs. The traffic provided is therefore too wide and not specific enough.

• Stanford Dataset: Stanford University provides a large repository of datasets on Internet mea surements, considering companies such as Twitter and Amazon, along with websites such as Wikipedia [89]. This was another possibly interesting measurement repository, however, the measurements had other social purposes. These measurements mostly deal with social inter actions and circles, characterized by nodes and node social networks on these websites, which question their value as network measurements. From connections on social media to node net works of recommended products on Amazon, these measurements were taken in 2002.

• Tstat: Is a measurement tool that allows to passively capture Internet traces [2]. For instance, they use this tool to capture Skype traces or IP TV measurements [90].

• SmartData@Polito: Polytechnic of Turin provides, through its network association Smart Data@PoliTo, passive measurements from the Italian Internet, retrieved with the above men tioned Tstat tool in another setting [6]. These measurements [54], provide information on the situation of the Italian Internet traffic, such as popularity and traffic percentage of the major Internet players. Furthermore, there are available datasets on total and peruser upload and download Internet traffic of many companies.

A great variety of sources were considered, with many different network measurements and cap turing techniques. Few were selected for a firstlevel inspection, and fewer were brought even further to the second step. It is a further confirmation of something already found in the literature analysis section. Network measurements are used for many purposes, mostly regarding structure and topology of the Internet, which are widely researched by the academic community. These types of measurement are, mostly, not used to analyse and understand companies, but for other purposes. This is, for us, both positive and negative, and the main implication is that, in the literature, there is little evidence in the relation between network measurements and company performance. On one side, this is a negative fact, as it does not give us any benchmark on which to build our analysis. Not having any reference, however, gives us a quite high degree of freedom, as we are able to make this investigation on multiple levels in the study cases. Once a measurement is found, then it could be related to various factors and indicators, that might be connected to such measurement.

5.2. Measurements on Companies Of all the above measurement sets considered, few of them related to companies in some way. [2] and [54] present various datasets on various ebusinesses and Interner players. [2] uses Skype traces and Fastweb (an Italian ISP) IPTV traces. The SmartData@PoliTo group [54] shows traffic of companies like Facebook, Snapchat, Instagram, YouTube, Netflix and many other companies operating in Italy. We, therefore, proceeded to check the availability of financial data for each of these companies and proceeded to select the cases for our research.

5.3. Cases Selected Firstly, we remind that we build one of the study cases on the already available Salesforce.com datasets, as they were the datasets that provided inspiration for the thesis. This is a privately collected set of vantagepointfree measurements by the researcher Tobias Fiebig, giving us an insight into the number of hosts deployed by the company Salesforce.com [5]. The measurement represents the number of hosts deployed, deal with a significantly large cloud ebusiness that publishes large amounts of financial data for its investors and that is largely covered by business news outlets. The choice of selecting this case was therefore immediate and almost automatic. From the various data sources considered, we identified only valid measurement dataset, the one from Turin at the SmartData@Polito networking group [54]. From this measurement dataset, we 5.3. Cases Selected 35 had a look at all the measurements and the companies they refer to. We, therefore, studied the measurements per each company and selected the measurements that seemed the most interesting or that could reveal something. Lastly, we checked if the measurement referred to a company with large financial data and news available online and finally selected the case studies. We decided to use the Netflix and Snapchat measurements to perform two separate case studies. For both companies, we have possibly meaningful datasets and all the financial data and news necessary to hopefully create an insightful case study. We also believe that other measurement from other companies, such as YouTube and Instagram could contribute to each of these cases in some way and might be used. Other companies, such as Facebook, were for instance discarded, as the financial data available also include revenues and costs from other major services (such as Whatsapp or YouTube), undermining the possible relevance of the single measurement. Hence, we decided to build three study cases, one for Salesforce, Netflix and Snapchat respectively, and see how they can contribute to a general overview and explore our topic. 6 Salesforce

The first company approached as a study case is Salesforce.com, Inc., an American public company listed on the NYSE as CRM. We selected this company to perform a study case for three reseasons.

• It is a company that largely operates through the Internet and perfectly matches the definition of ebusiness. • The measurement found on the company has a direct link to their business model and operations. • There is a large amount of financial data and news available on which it is possible to build a study case.

With the measurements on this company at hand, the decision to select this case was quite imme diate. We proceed with this study case according to the structure proposed in Section 2.4; Figure 6.11 shows a timeline of the case.

6.1. Salesforce.com

Salesforce.com is a digital Customer Relationship Management (CRM) enterprise that offers a cloud based application for sales, services, marketing and more. It was founded in 1999 and went public in 2004: a young company that grew in parallel to the digital transition of businesses. It is now part of the S&P 500 index, which groups the stocks of the 500 biggest American companies. They are one of the top 5 fastest growing software enterprises and can be defined as an Eservice and SaaS provider company with a B2B model, counting up to 35.000 employees [91]. Through their platform, they sell company management tools, like sales, analytics or productivity [92]. They assist other companies with cloud platforms and services for marketing, industry, com merce and online payments with premade software. They also offer innovative and newly developed products such as AIbased tools [92]. The company offers software and professional services as a value proposition to its customers. They offer monthly subscriptions to their clients with premade tariffs and also offer custommade products and services that they require specific payments. Some of their biggest clients are Amazon Web Services or Adidas. When a company commits to using Salesforce’s services, it invests on its financial and intellectual capital, as the workers need to learn and integrate it with their workflow. The adoption of a subscription model then seems appropriate for the company. They are also one of the biggest companies and most successful companies in their sector and have been steadily growing in the last years, as more and more SME (Small and Medium Enterprises) turn to the cloud to scale up their business and operations. The majority of their customers, around 71%, comes from the domestic Americas region, while the rest is international. The company also reports being the market leader with the largest market share, as it kept steadily growing in the past few years [93]. There is also information on the revenue streams from the company, over a couple of years, where Cloud revenues account for Sales, Service, App & Other and marketing altogether. Both internally and

36 6.2. Data Sources 37 externally, the company reports constantly increasing revenues [73], [93]. This sustained growth is not unexpected form such a large company in such a new, evolving and upcoming sector. Secondly, we also saw from the previous paragraphs that the company steadily increased its market share and beat its competition, and the figures just reflect its external performance. We also see that the company reports the revenues for their fiscal years: Q1 2016 is released on the 20/05/2015, considering it the first report of that fiscal year. Table 6.1 reports clearly this issue. In the rest of the report we do not use a financial timescale but a yearly timescale: when we mention a quarter, it refers to the data of that same year and not the previous one.

Date Fiscal Quarter Time Quarter 20/05/2015 Q1 2016 Q1 2015 20/08/2015 Q2 2016 Q2 2015 18/11/2015 Q3 2016 Q3 2015 24/02/2016 Q4 2016 Q4 2015 18/05/2016 Q1 2017 Q1 2016 31/08/2016 Q2 2017 Q2 2016 17/11/2016 Q3 2017 Q3 2016 28/02/2017 Q4 2017 Q4 2016 18/05/2017 Q1 2018 Q1 2017 22/08/2017 Q2 2018 Q2 2017 21/11/2017 Q3 2018 Q3 2017 28/02/2018 Q4 2018 Q4 2017

Table 6.1: Date References for Fiscal and Time Quarter. Adapted from Salesforce.com. Retrieved from https://investor. salesforce.com/financials/default.aspx, Copyright by Salesforce.com.

6.2. Data Sources

We then describe the source of the data used to build this study case. The main data used are financial data, the measurement dataset, and financial news and information relevant to the case.

• Financial Data: The company reports useful information on its business, activities and market share in their quarterly and yearly reports, from which we have recovered most of the data used in the case. The reports are publicly available on the company’s website, at https:// investor.salesforce.com/financials/default.aspx [93]. Other information on the company and its activities has been collected on their website as well. These financial metrics are used as performance indicators for the analysis of this case study. For our analysis, we only selected the quarterly reports in a specific time frame. We consider all the quarterly reports from May 2015 to February 2018; respectively, we consider the 12 quarterly reports from Q1 2015 to Q4 2017 included. This is also used as the selected timeframe for the case study. While not part of the overall analysis, we also used some data reported on the Q4 2014 report release, for forecastrelated figures. Figure 6.2 through 6.6 were made using the data in these reports; we also report the financial data gathered and used in Tables throughout the case [93].

• Measurements: The measurements used were collected by the researcher, Dr.Ing. Tobias Fiebig, in the context of a broader research topic, between 20016 and 2017. The measurements were gathered in the context of a wider research project described in the paper “Something from Nothing (There): Collecting Global IPv6 Datasets from DNS.”[5].

• Financial News: We also report useful information regarding the company’s business and ac tivities during the selected time period. As a source for relevant financial news, we referred to the Reuters news organization (https://www.reuters.com/). We scanned their website for articles relevant to Salesforce and its operations in that time frame for additional information.

These sources are used as ground truth and as the practical experience that case studies require. With our future perspective, we are able to validate findings against reality, and these sources are the reality against which predictions are verified. Due to their importance and relevance in the context of this study case and research, these data sources were selected for their validity and reliability, as it directly correlates to the value of the findings and results. We stress how we used various types of 38 Salesforce data and sources (news, measurements and financial data), as the literature suggests and supports it [31]. Constantly checking and establishing converging lines of evidence [4] is defined as triangulation, and is at the core of case study analysis. Using various sources should help validate and increase the findings reliability.

6.3. Measurements, Monitoring & Interpretation

6.3.1. Collection Methodology The measurements selected for this section were gathered by the researcher [5] between 2016 and 2017 and access was provided by the researcher. The measurement begins on the 12th of September 2016, while the measurement stopped being collected on the 28th of March 2017. The paper describes the techniques used for collecting these measurements (”Something from Nothing (There): Collect ing Global IPv6 Datasets from DNS”, [5]). The measurements were collected with the toolchain de scribed in the paper, available on GitHub at https://gitlab.inet.tuberlin.de/ptr6scan/ toolchainin. The researchers used a vantage pointfree, rDNS active measurement technique to collect existing IPv6 addresses datasets. Through PTR records and rDNS, the algorithm crawls for all the company IPv6 host addresses, by seeding the algorithm with possibly existing bases such as ip6.arpa zones. The methodology implemented is able to reach and recover a large number of addresses. For instance, with their algorithm, they also reach possibly address zones by hierarchically higher DNS servers. They approach this problem by seeding the algorithm with possibly existing bases such as ip6.arpa zones. Furthermore, they also use publicly available Routevies and RIS BGP tables, which are openly available [5]. Since this active measurement technique could create unwanted load on the reverse DNS zone, they limit their outbound throughput. For the measurements regarding Salesforce hosts, the researchers collected the IPv6 addresses that the company deployed in that time frame. They do so by specifically selecting the company’s IPv6 addresses announced on bgp.he.net. This way, the researchers collected and registered for four months all the IPv6 addresses of the company. The host records were daily gathered from early September 2016 until March 2017. By limiting the total output to 10 Mb/s and to 2 Mb/s for every single target, they encourage best practices and reduce possibly disturbing load.

6.3.2. Hosts The toolchain obtains IPv6 addresses operated by Salesforce’s hosts: each host has his own IP, pro viding us insider information on the company and its operations. Each address collected possibly represents a host operated by the company. For the sake of the case we to define what “hosts” are. We considered early RFCs, which are now considered slightly outdated, but still provide a workable definition of hosts; we considered RFC 871,112, 1123 and 1127 [94], [95], [96], [97]. Hosts a defined as “computer operating systems”, and at the time researchers were still setting up the infrastructure of networks that now makes up the in ternet. They were mostly defining the types of protocols through which machines could communicate. Later on, when specifying the requirements of internet hosts, they more thoroughly defined. “A host computer, or simply ”host,” is the ultimate consumer of communication services. A host generally exe cutes application programs on behalf of the user(s), employing network and/or Internet communication services in support of this function. An Internet host corresponds to the concept of an ”EndSystem” [97], p. 6. these kinds of hosts can have various sizes, purposes, speed and computing power: from microprocessors to supercomputers. Hosts can be considered the single node that makeup the networks connected to the internet. Software capabilities and hardware were subsequently defined. Today, many electronic devices can be considered hosts; personal computers, servers, and routers can be hosts, as long as they have an IP address connected to them. 6.3. Measurements, Monitoring & Interpretation 39

6.3.3. Measurements Specifications

We received a measurement for each day in the of the collection period, categorized by a Unix times tamp. The measurement contains, along with the IPv6 addresses, IPv4 PTR addresses that we did not consider for the measurement. Each day, the toolchain outputs a JSON file, containing the IPv6 records, one per each line. On each line, there is a record that presents the query sent by the toolchain and the response received, which happens when a valid record is found. Each line is made up of various strings one after another; we can see a query example: {”ARPA”: ”9.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.0.1.1.0.2.0.0.0.0.8.1.6.5.0.6.2.ip6.arpa.”, ”QUESTION”: ”9.0. 0.0.0.0.0.0.0.0.0.0.1.0.0.0.0.1.1.0.2.0.0.0.0.8.1.6.5.0.6.2.ip6.arpa. IN PTR”, ”AUTHORITY”: ””, ”CNAM E”: ””, ”runid”: 1475181481, ”error”: false,”ANSWER”: ”9.0.0.0.0.0.0.0.0.0.0.0.1.0.0.0.0.1.1.0.2.0.0.0 .0.8.1.6.5.0.6.2.ip6.arpa. 3600 IN PTR eth1491–asw1e0401dfw.net.sfdc.net.”, ”PTR”: ”eth1491– asw1e0401dfw.net.sfdc.net.”, ”metadata”: {”rcode”: ”NOERROR”, ”opcode”: ”QUERY”, ”id”: ”58104 ”, ”flags”: ”QR RD RA”}. For each day the number of records found corresponds to the number of hosts found. We then counted day by day the number of records found each day, i.e. IPv6 addresses, that were collected by the toolchain. We then graphed out the results in Figure 6.1, where the numbers of hosts are represented for each day. It is not possible, from our data to immediately distinguish between servers or routers, as the measurements are made by recording their IP addresses. However, considering the kind of ebusiness and services the company operates, we can make some assumptions on these hosts. We believe that it is safe to assume that a good portion of their hosts is made up of servers, to maintain and expand their growing cloud services.

# IPv6 Hosts 90000

85000

80000

75000 Hosts

70000

65000

60000 12/09/201621/09/201630/09/201609/10/201616/10/201623/10/201630/10/201606/11/201613/11/201620/11/201627/11/201604/12/201611/12/201618/12/201625/12/201601/01/201708/01/201716/01/201723/01/201730/01/201706/02/201713/02/201720/02/201727/02/201706/03/201713/03/201720/03/201727/03/2017

Date

Figure 6.1: Salesforce Hosts [5]. Adapted from ”Something from Nothing (There): Collecting Global IPv6 Datasets from DNS” by T. Fiebig, K. Borgolte, S. Hao, C. Kruegel, G. Vigna., Passive and Active Measurements Conference 2017. Copyright 2017 by T. Fiebig.

The measurements show three welldefined trends.

• In just a couple of weeks in September 2016, they deployed around 10.000 new IPv6 hosts to their stock.

• They then deployed, more gradually and slowly, over a 3 months period, another 10.000 hosts.

• From the end of January onward, the number remained constant at around 86.000 hosts.

By analysing the company before the measurements are taken, we see whether the increase in IPv6 host addresses would change the outlook on the company or anticipate unexpected results. 40 Salesforce

6.3.4. Measurements Interpretation We need to interpret the measurements and understand what they can tell us, following the proposed framework 2.3.1. By looking at them, and their timings, we see that the increase by 10k hosts in September could be reflected on its own quarter or the following one. Similarly, 10k more hosts are deployed between October and December, which could influence company performance on later quarterly reports. While we can monitor the deployment of the hosts, we do not know exactly how and when the company used them, but they must be there to be integrated into their operations at some point. From this, we can infer that the company is expanding its operations and investing in its core infrastructure: servers and routers. The company is expanding its capacity of traffic, storage and computational power, expecting to acquire and service more clients. These investments create higher costs, that need to be accounted for in the short term. The sudden increase is hosts can reflect an increase in demand, and the company consequently expanded its business and operations by integrating new hosts. Salesforce wants to benefit from their expansion in the long run, due to their subscription business model. This investment can also be interpreted as an increase in demand and its number of customers for the company, and in the short term, the company’ profitability might remain low. However, if they decided to expand their operations, then it means that they want to profit from it, at some point in time.

6.4. Performance Overview

In this section, we start with the analysis methodology proposed in Section 2.4. The first step is to select the point in time in which we place our perspective. In this Section, we present prominent KPIs for the company and relevant news. As previously mentioned, the measurements were taken from September 2016 until March 2017; this corresponds, roughly, from Q3 2016 to Q2 2017, roughly represented on the graphs as two crosses, as these are the start and end date of the measurement collection. The KPIs here presented and graphed were selected for two reasons. The first is their general importance as a financial benchmark, the second is their relevance to Salesforce.com. Most of the presented indicators and ratios are because of a multinational corporation such as Salesforce values such indicators. In particular, we did not find other specific KPIs that the company values in its quarterly reports apart from the financial ones. We present revenues and profitability ratios, as a listed company greatly focuses on such indicators. Considering their B2B model as well, we identify the number and the type of customer as extremely relevant for the company. As we have seen, their subscription model focuses on longterm profitability through long contracts with their clients. Therefore, we can clearly see how the size and longterm stability of their customer is crucial to their business. Secondly, Salesforce, in its reports, makes a forecast of these KPI for their next quarter. This way, we were able to collect the quarter result and its forecast from the previous quarterly report. These differences are calculated by subtracting the forecast to the actual performance: if the difference is positive, it means that the company beat its own estimations and overperformed, the opposite if negative. This might help us see whether measurements could justify, with hindsight, unexpected behaviours or beyond forecast results, along with assessing how accurate the company is with its forecasts. We also decided to consider these proxies as they are widely accepted KPIs on the market. Furthermore, these factors were also identified in Chapter 4 as valid traditional financial indicators. We also consider possible relevant news or market opinions on company performance. We also need to make a remark on Earnings Per Share and Revenue forecasts that the company makes (Figures 6.2, 6.3, 6.4 and Tables 6.3 and 6.2). The company forecasts a range in which they expect their revenues and EPS to fall. In each quarter, they forecast an upper bound and lower bound in which the result for the future quarter is expected to fall. However, the difference between these two bounds is, in each quarter, 10 million$ for revenues and 0.01 $ for EPS We decided to take a simple average between the lower and upper bound as the forecast for the next quarter and consider that as the forecast for the next quarter. From that forecast average, we computed the difference with the actual results. Consequently, the YearoverYear revenue forecast percentage presents a 0.5% error. 6.4. Performance Overview 41

6.4.1. Financial Performance We started by graphing out the total company revenues and then computed the difference between the actual revenue and the forecast revenue from the previous quarter, which we can observe in Figure 6.2. Both figures are in thousands of $, revenues represented on the left yaxis and difference on the right yaxis. We can see that revenues increase steadily, an average of 105 million $ per quarter, reaching 2800 million $ in early 2018. For the forecasts, we see that the company beats his own estimates by at least 22 million $, with peaks above 40 million $ in late 2015 and early 2017.

Actual Revenue and Difference to Forecasted Revenue

2800 45

2600 40

2400 35

2200 30 2000 Revenue, million $ Difference, million $ 25 1800 20 1600

15 20/05/201520/08/201518/11/201514/02/201618/05/201631/08/201617/11/201628/02/201718/05/201722/08/201721/11/201728/02/2018 20/05/201520/08/201518/11/201514/02/201618/05/201631/08/201617/11/201628/02/201718/05/201722/08/201721/11/201728/02/2018

Date Date

Figure 6.2: Salesforce Revenue and Difference with respect to Forecast [93]. Adapted from Salesforce.com. Retrieved from https://investor.salesforce.com/financials/default.aspx Copyright by Salesforce.com. Date Total Revenues Next Q. 20/05/2015 1,511,180 1,595,000 20/08/2015 1,634,665 1,695,000 24/02/2016 1,809,438 1,890,000 18/05/2016 1,916,610 2,010,000 31/08/2016 2,036,638 2,115,000 17/11/2016 2,144,794 2,272,000 28/02/2017 2,294,037 2,345,000 18/05/2017 2,387,571 2,515,000 22/08/2017 2,561,590 2,645,000 21/11/2017 2,679,810 2,806,000 28/02/2018 2,851,003 2,930,000

Table 6.2: Salesforce Revenues and Forecast [93]. In Thousands of $. Adapted from Salesforce.com. Retrieved from https: //investor.salesforce.com/financials/default.aspx, Copyright by Salesforce.com.

The second KPI considered is the YearoverYear revenue increase. This percentage shows how much the revenues grew with respect to the same quarter from last year. The company also makes predictions from for the coming quarter. This way we are able to see by how much the company improved over a year. As mentioned in Section 6.4, for revenues, we allowed a 0.5% error in the YoY prediction. We observe in Table 6.3 that the value floats around 25% and that the difference with from the last quarter prediction is generally around 2%, without showing particular or interesting trends. This value reflects revenue increase with respect to the previous year and signals a steadygrowing company given the yearlength perspective.

Date YoY Inc % YoY Inc % Projection EPS Projection NonGAAP Projection 20/05/2015 23.0% 21.0% 0.010 0.000 0.160 0.175 20/08/2015 24.0% 22.5% 0.000 0.015 0.190 0.185 18/11/2015 24.0% 23.5% 0.040 0.085 0.210 0.185 24/02/2016 25.0% 25.0% 0.040 0.050 0.190 0.235 18/05/2016 27.0% 23.0% 0.060 0.035 0.240 0.245 31/08/2016 25.0% 23.5% 0.330 0.045 0.240 0.205 17/11/2016 25.0% 25.5% 0.050 0.095 0.240 0.245 28/02/2017 27.0% 22.5% 0.070 0.025 0.280 0.255 18/05/2017 25.0% 23.5% 0.010 0.005 0.280 0.315 22/08/2017 24.0% 23.5% 0.020 0.045 0.330 0.365 21/11/2017 25.0% 23.5% 0.07 0.035 0.39 0.325 28/02/2018 24.0% 23.5% 0.09 0.095 0.35 0.435

Table 6.3: Salesforce Yo.Y. Revenue Increase and Earnings Per Share [93]. In $. Adapted from Salesforce.com. Retrieved from https://investor.salesforce.com/financials/default.aspx, Copyright by Salesforce.com.

The third performance indicator we considered is Earnings Per Share, which the company forecasts 42 Salesforce from quarter to quarter. This allowed us to similarly graph the actual EPS and the difference with the forecast from the previous quarter. Salesforce reports and forecasts both GAAP (Generally Accepted Accounting Principles) and nonGAAP. The company also describes how they compute nonGAAP EPS in its quarterly reports and explains its decisions. To find nonGAAP EPS, they exclude amortization (of purchases and debt discount), stockbased expenses, gains or losses on sales of strategic investments and adjustments of income taxes. EPS and non GAAP EPS (Table 6.3) are in $ and have a similar forecastrange issue as we described above, and therefore we mention a 0.005 $ error in the forecast that directly reflects in the difference. The left yaxis is for the EPS result while the right yaxis is a scale for the prediction difference. From GAAP EPS we notice a generally negative trend close to 0 $, until an unexpected increase to 0.30 $ per share, on Q2 2016 (Figure 6.3). However, in that report, Salesforce.com announces that they benefit from a tax valuation allowance release of around 266 million $ from the acquisition of Demandware [73]. This unexpected result was not already forecast and accounted for, as the difference with respect to the forecast follows the same curve. It means that the company, in the previous quarter, did not expect such a rise in EPS. It explains the uncharacteristic rise in EPS and difference, which usually floats around 0 $. We can also notice that the difference is quite often negative, meaning that the company underperformed with respect to its own forecasts. Ultimately, EPS start to grow above negative values but only in late 2017, from Q2 2017. NonGAAP EPS present a rather different behaviour (Figure 6.4). It is growing rather steadily and showing how the company usually overperforms with respect to its estimations and shows steady growth.

Actual GAAP EPS and Difference to Forecasted EPS 0.35 0.30 0.30 0.25 0.25 0.20 0.20

0.15 0.15

0.10 0.10 GAAP EPS, $ Difference, $ 0.05 0.05 0.00 0.00 0.05

20/05/201520/08/201518/11/201514/02/201618/05/201631/08/201617/11/201628/02/201718/05/201722/08/201721/11/201728/02/2018 20/05/201520/08/201518/11/201514/02/201618/05/201631/08/201617/11/201628/02/201718/05/201722/08/201721/11/201728/02/2018

Date Date

Figure 6.3: Salesforce GAAP Earnings Per Share and Difference with respect to Forecast [93]. Adapted from Salesforce.com. Retrieved from https://investor.salesforce.com/financials/default.aspx Copyright by Salesforce.com.

Actual non-GAAP EPS and Difference to Forecasted EPS 0.40 0.035

0.030 0.35 0.025

0.30 0.020

0.015

0.25 Difference, $ 0.010 nonGAAP EPS, $ 0.005 0.20 0.000

0.005 0.15 20/05/201520/08/201518/11/201514/02/201618/05/201631/08/201617/11/201628/02/201718/05/201722/08/201721/11/201728/02/2018 20/05/201520/08/201518/11/201514/02/201618/05/201631/08/201617/11/201628/02/201718/05/201722/08/201721/11/201728/02/2018

Date Date

Figure 6.4: Salesforce Non GAAP Earnings Per Share and Difference with respect to Forecast [93]. Adapted from Salesforce.com. Retrieved from https://investor.salesforce.com/financials/default.aspx Copyright by Salesforce.com.

The last KPIs considered are the Financial Ratios, Figures 6.6, 6.7, 6.8, 6.5. We present Net profit Margin (Net Income divided by Revenues), Return on Assets (Net Income divided by Assets) and Return on Equity (Net Income divided by Stockholder’s Equity). NPM shows how much of a dollar in revenue translates into real profit. It has a rather stable trend except for a sudden and sharp increase in Q2 2016, but as mentioned already, the company explains as a tax allowance from an acquisition. ROA shows how profitable a company is relative to its assets, showing how well the company profits from 6.4. Performance Overview 43 its investments and maintains a rather stable trend in this case. Lastly, ROE shows how effectively the company is being managed and how much cash it needs from investors to operate effectively. Similar to ROA, this percentage stagnates around 0%. Closely related to these ratios, we also present Net Income independently, in Figure 6.6, which can be understood as the ultimate goal of the company, as its profit benchmark. We see, that except the unexpected revenue source of Q2 2016, NI stagnates and sometimes signals losses for the company. All these ratios only change their negative after Q2 2017, where we can notice an increasing trend very similar to the one made by GAAP EPS: this is rather expected, as these financial KPIs are strictly related with each other within a corporation.

NPM Net Income

25 200000

20 150000 15 100000 % 10 million $ 50000 5

0 0

5 50000

Date Date

Figure 6.5: Salesforce Net Profit Margin [93]. Adapted from Figure 6.6: Salesforce Net Income [93]. Adapted from Salesforce.com. Retrieved from https://investor. Salesforce.com. Retrieved from https://investor. salesforce.com/financials/default.aspx Copy salesforce.com/financials/default.aspx Copy right by Salesforce.com. right by Salesforce.com.

ROA ROE 1.50

1.25 3 1.00

2 0.75 % % 0.50 1 0.25

0 0.00

0.25

Date Date

Figure 6.7: Salesforce Return on Equity [93]. Adapted from Figure 6.8: Salesforce Return on Assets [93]. Adapted from Salesforce.com. Retrieved from https://investor. Salesforce.com. Retrieved from https://investor. salesforce.com/financials/default.aspx Copy salesforce.com/financials/default.aspx Copy right by Salesforce.com. right by Salesforce.com.

Date NI NPM ROE ROA 20/05/2015 4,092 0.27% 0.10% 0.04% 20/08/2015 802 0.05% 0.02% 0.01% 18/11/2015 25,157 1.45% 0.53% 0.23% 24/02/2016 25,509 1.41% 0.51% 0.20% 18/05/2016 38,759 2.02% 0.69% 0.30% 31/08/2016 229,622 11.27% 3.74% 1.60% 17/11/2016 37,309 1.74% 0.55% 0.26% 28/02/2017 51,440 2.24% 0.69% 0.29% 18/05/2017 9,027 0.38% 0.11% 0.05% 22/08/2017 17,736 0.69% 0.21% 0.10% 21/11/2017 51,394 1.92% 0.58% 0.29% 28/02/2018 67,555 2.37% 0.72% 0.32%

Table 6.4: Salesforce Financial Ratios [93]. In Thousands of $. Adapted from Salesforce.com. Retrieved from https:// investor.salesforce.com/financials/default.aspx, Copyright by Salesforce.com. 44 Salesforce

6.4.2. Significant News To also have more external perspective on the company, we scanned for and collected business news regarding the company operations in the months before and after the measurements were taken. The company made various announcements, regarding its future operations and prospects in 2016. Investors responded positively to the company explicit intention of integrating artificial intelligence for longterm robust growth, a sector in which they are lacking compared to competitors such as Oracle and SAP. The company was also active in the M&A section, successfully acquiring Demandware, a provider in the ecommerce field [98]. The company also created a lot of attention on itself by making bids to buy companies like Twitter or Linked in the same year, [99]. However, both acquisitions did not happen, as target companies suffer from this missed opportunity [99]. On the 17th November 2016, the release date of their Q3, Salesforce’s reported revenues were some 30 million $ higher than estimates, and their stock price rose by 4.8% [100]. Furthermore, Salesforce’s CEO also announces its intention to expand its Cloud and AI services, already seen as a company point of strength [100]. On the same day, during the classic investor conference call, they announced a stronger relationship with Amazon and the acquisition of new topclients. These clients are banks such as Citigroup and PNC Bank, beauty products manufacturers such as Shiseido or technology companies such as KONE and Telcom Argentina [101][102]. We stress the fact that in the previous quarters, the company did not disclose any information regarding the possible acquisition of such large clients. We then see that the increase in September of IPv6 hosts could have signalled the acquisition of these new clients or it could have been related to the new and stronger alliances with AWS (in December 2016) or Cisco (in late September 2016), [103], [104], [105]. Similarly, the steady increase in hosts up to January could be related to the company acquiring these new clients. Furthermore, we also see that, through the course of 2017, Salesforce announces strategic partnerships and agreements with IBM, Dell Technologies and Google [106], [107], [108]. Then, as of Q4 2017, the company announced strong profits and growth thanks to its cloudbased sales and marketing software: traders appreciated the company’s 18% market share and its stock rose by 14 % over the year.

6.5. Analysis & Results

To establish whether there is a link between measurements and company performance, it necessary to analyse whether the measurements forecasted unexpected factors or behaviours. We are able to validate and verify our results thanks to our future perspective. We attempt to present the evidence as clearly as possible because the case should ”allow readers to judge independently your later inter pretation of the data.”[4]. This is the most important step of the case study and the one that needs the most attention, as we triangulate and combine together all data presented so far. We intend to see whether there is a connection between the measurements presents in section 6.3 and the KPIs presented in section 6.4.1, along with the news discussed in section 6.4.2.

6.5.1. Analysis In this section, we need to remind ourselves that Salesforce:

• Added 10k new IPv6 hosts in a couple of weeks in September 2016.

• Added around 10k more IPv6 hosts in the next few months, until January 2017.

In our analysis, while considering the relevant KPIs and measurements, we constantly remain aware of this fact and ask ourselves whether a certain trend or pieces of news could have been predicted by this increase of hosts and demand. In other words, could a certain behaviour, trend or KPI result have been predicted by such increase in hosts? We need to be particularly careful to general trends and outliers and unexpected results in the data. In particular, we intend to see if, at a certain point in time, with the knowledge of the measurements, it could have been possible to gain meaningful information on the company and predict its future results. If these measurements can predict KPIs or suggest relevant new, then we would have found a clear connection. 6.5. Analysis & Results 45

• We start by looking at Revenues and the difference in results with respect to the previous quarter forecast in the specific time frame selected. We see that Revenues keep growing rather steadily, without unexpected or unprecedented behaviours when the new hosts are deployed. It is instead worth noticing how the difference to its forecasted revenues behaves right after the hosts are deployed. In January 2017 Salesforce finishes deploying hosts: as already discussed, the mea surements signal an increase in demand from the company: Salesforce is preparing to supply its increase in demand and sustain its expansion in the cloud business. As we can imagine, it takes time from deployment to utilization of servers, especially considering the extremely diversified value proposition that Salesforce has. In particular, we see that four months after the new hosts are deployed, the company announces revenues above a forecast of 22, 42 and 46 million $ for the following 3 quarters. This means, that throughout 2017, the company had much larger rev enues than its own forecast on its own business. The assumption is that the company deployed its hosts in order to supply its increasing demand, which turned out to be even more profitable than expected. We believe that this steady increase in Revenues and the unexpected beyondforecast revenues are predicted by the increase in hosts as explained above. If we placed our perspective in time, after the measurements are taken, we would correctly predict beyondforecast revenues for the company for the next few quarters.

• Next, looking and YoY growth and its future forecast, we notice very stable trends and values. It does not seem insightful and the measurements would not predict different trends or unexpected behaviours, as there simply are not.

• Considering GAAP EPS and nonGAAP EPS, we already knew that the sharp increase in Q2 2016 is due to the acquisition of Demandware. Apart from that, we see a slight growth of GAAP EPS after Q2 2017, but quite modest, while nonGAAP EPS maintain their growth rather steadily. Furthermore, the differences with respect to the forecast are always within 0.05 $, which is quite negligible. Ultimately, we do see a small GAAP EPS positive growth after the deployment of the hosts, but EPS is a parameter that heavily relies on financial markets and company policies such as dividends, and therefore could be much harder to validly predict from the measurements.

• The last financial KPIs considered are Net Income and the related profitability ratios. Like Rev enues and EPS, Net Income (and consequently, the other ratios, as they are computed from Net Income) starts to grow after the deployment of hosts. These ratios start to grow from Q2 2017, but so does Revenues: NI is a function of revenues, and since we see them growing from that point on, we are not surprised to see NI grow as well.

• Lastly, by monitoring the news related to Salesforce, we see that the first batch of measurements is deployed in September 2016. Shortly after, in November at the Q3 2016 release, Salesforce announces the acquisition of major clients like Citigroup, PNC Bank, Shiseido, KONE and Telcom Argentina [101][102]. Similarly, in September and December 2016, they announce stronger alliances with Cisco and AWS [103], [104], [105]. Lastly, we see that after the second batch of hosts is deployed in January 2017, they announce partnerships with Google, IBM and Dell Technologies[106], [107], [108]. We are quite convinced that the deployment of hosts predicts the acquisition of such clients and the expansion of their partnerships. We can definitely say that building up and increasing their number of hosts is done to comply with the demand that they expected to have, but not made public yet.

6.5.2. Results

In this section, we discuss and highlight the results obtained in the previous section highlighting how and why certain aspects of company performance were predicted by the measurements and why some were not. In general, various parameter and KPIs were not considered predicted by KPIs. Revenues and nonGAAP EPS do not show abnormal or unexpected behaviours before or after the deployments of the hosts, as they both maintain an upward trend showing how the company is growing. Similarly, 46 Salesforce

YoY revenue growth remains rather stagnant and shows how much the revenues have grown with respect to the previous year, which is ultimately deemed uninteresting. GAAP EPS, NI and its related profitability ratios all show a steady growth after the hosts are deployed. However, this is line with the growth of revenues in those quarters, and do not show abnormal results or behaviours. While indeed the measurements signal an increase in demand, the result for these are in line with the growth in Revenues and could have also been influenced by dividend policies or cuts in costs to become more profitable. It seems that the measurements do not predict or particularly relate to them. On the other hand in Q1, Q2, Q3 and Q4 2017 the company outperformed its own revenue expec tations by more than 30 million $. In Figure 6.9 we provide an overview of how the beyondforecast revenues are timed with the deployment of hosts, and that they predict and anticipate such a posi tive result before the company announces it. At the end of January 2017, the company has deployed 20000 new IPv6 hosts, crucial for a cloud software provider that offers online services and servers to companies. From 65000 hosts, they jumped to more than 85000 hosts in just a few months: they increased the hosts available to them by more than 25 %. At this point in time, at the end of January 2017, it is clear that the company is expanding its operations, and that it is expecting an increase in clients and hopefully profits. In the same quarter as the host deployment (Q4 2016), the unexpected revenues only exceed forecasts by 22 million $. However, right after the hostdeployment quarter, the company reports higher than expected profits for a whole year. In particular, for a whole, the company actually beats his own revenues estimated by 30 and 40 million $. We consider such KPI as predicted by the measurements for a couple of reasons; firstly, there was no expectation from the company of such high revenues, secondly, the measurements show that the company earned even more than they expected: the deployment of such hosts and the increase in demand was more meaningful than expected. While reading through the company’s quarterly reports and conference calls transcripts, we do not find hints on future performance. These unexpected results reinforced the expectations of the company and the markets noticed it positively [109]. However, the company is able to supply more clients, and end they end up profiting even more that they expected. It is therefore easy to associate the increase in hosts to the large increase in demand that the company experience in 2017, leading to unexpected profits. The measurements anticipate an increase in demand, forecastable at the end of January 2017 and first announced in May 2017, going throughout the whole year of 2017. To sum up, Figure 6.9 provides an overview of the results and findings. The measurements predict an increase in demand; after the deployment of the hosts, it is possible to predict and anticipate an increase in revenues i.e. unexpected revenues. This prediction point is selected in January 2017, after the company increased its number of hosts by more than 25 %. This point is chosen as it immedi ately follows the deployment of hosts, i.e. an unexpected change in the considered measurement. Four months later in Q1 2017, and for the rest of the year, Salesforce announced aboveexpectations revenues: thanks to the measurements, it could be possible to know this months in advance. The announcements of higherthanforecast revenues are identified all throughout the 2017 quarters when the information becomes public knowledge. 6.5. Analysis & Results 47

Difference to Forecasted Revenue

45 10k Hosts 10k Hosts 40

Difference, million $ 25 Q1 Q2 Q3 Q4 2017 Higher Than Forcast Revenues

Prediction Point 15 20/05/2015 20/08/2015 18/11/2015 14/02/2016 18/05/2016 31/08/2016 17/11/2016 28/02/2017 18/05/2017 22/08/2017 21/11/2017 28/02/2018

Date

# IPv6 Hosts 90000

85000

80000 10k Hosts 75000 Hosts Increase in Demand 70000 Higher Revenues Prediction 65000 10k Hosts

Date

Figure 6.9: Salesforce Revenue Difference with Respect to Forecast.

Next, we have highlighted how the deployment of hosts anticipates, in the short and long term, the acquisition of major clients and stronger strategic alliances or partnerships with major companies. The newly acquired clients are Citibank, KONE or PNC Bank, while the partnerships were announced with Google, AWS, Cisco and IBM. Clearly, an outsider would not have access to such knowledge before it is made public by the company. This is insider information gathered from the measurements, unaccessible in another way and providing the observer with a market edge. Instead, thanks to the measurements, the observer knows that the company is expanding and increasing its number of hosts. It is possible to make such prediction in a specific point in time, as shifts in the measurements have already been observed and there is enough data to support them. For these predictions, we highlight two prediction points, placed after the noticeable changes in the measurements. Figure 6.10 shows the first prediction point right after the first deployment of hosts, at the end of September 2016, and forecasts the acquisition of new clients. The increase in hosts is a clear signal that the company is expanding: this concretizes in the acquisition of major clients by the next quarter. Also, strategic alliances are announced with Cisco and AWS, in September (right after the first host deployment) and in December 2016, respectively. This acquisition is then announced in November 2016, at the release of Q3 2016, but could already be forecasted a couple of months before. Very similarly, after the second batch of 10k hosts is deployed, the company expanded its operations again and is expecting an increase in demand: hence, January 2017 corresponds to the second prediction point for an expansion of their operations. Salesforce later announces strategic alliances throughout 2017 with Google, IMB and Dell Technology. In the first case, the change in measurements anticipated the announcements by just a couple of months; for the second prediction, the time varies, as Salesforce announces its new alliances throughout 2017. Ultimately, the change in the measurements effectively predicted an increase in demand hence an increase in clients and partners for Salesforce. To conclude, Figure 6.10 provides an on overview of the abovementioned events on the hosts 48 Salesforce graph. We consider the two prediction points, each after the 10k hosts are deployed, as a clear shift in the measurements can be observed and is indicative of the future acquisition of clients. These are firstly announced in Q3 2016, a couple of months after the hosts, and later on throughout 2017.

# IPv6 Hosts 90000 AWS Strategic Alliance

85000 Cisco Strategic Alliance

80000

75000

Hosts Second Prediction Point Q3 2016 70000 Acquisition of Major Clients 65000 First Prediction Point 60000 12/09/201621/09/201630/09/201609/10/201616/10/201623/10/201630/10/201606/11/201613/11/201620/11/201627/11/201604/12/201611/12/201618/12/201625/12/201601/01/201708/01/201716/01/201723/01/201730/01/201706/02/201713/02/201720/02/201727/02/201706/03/201713/03/201720/03/201727/03/2017

Date

Figure 6.10: Salesforce Hosts [5]. Adapted from ”Something from Nothing (There): Collecting Global IPv6 Datasets from DNS” by T. Fiebig, K. Borgolte, S. Hao, C. Kruegel, G. Vigna., Passive and Active Measurements Conference 2017. Copyright 2017 by T. Fiebig.

Prediction Point Date Performance Indicator Announcement Date 30/09/2016 New Clients 17/11/2016 31/01/2017 Clients & Partnerships Throughout 2017 31/01/2017 Unexpected Revenues Q1, Q2, Q3, Q4 2017

Table 6.5: Salesforce Prediction & Announcement Dates

6.5.3. Salesforce Timeline In Figure 6.11, we constructed a timeline of the measurements, the events and the results obtained from this case. We divided the timeline per year, month, and relevant quarterly report reported as QX in the Figure. Furthermore, we represent in cyan the relevant measurement period and collection. Through this view, we hope to highlight how the measurements can anticipate the Salesforce acquisition of major clients and their future longterm profits. Furthermore, we also point out in cyan the previously discussed prediction point, i.e. where there is enough data from the measurements to predict the acquisition of major clients and the unexpected revenues. We believe that a timeline like this can help us better understand how measurements are used to obtain insider information on a company. In the figure, we point out in red where and which KPIs the measurements predict. Timings are also important in this case, and this Figure 6.11 shows how it can be meaningful to always monitor measurements. 6.6. Conclusions 49

Q1 Salseforce Q2 Timeline Relevant Measurement Q3 Period 10k Hosts Deployed Q4

2016 y v eb ug ep Jan F Mar Apr Ma Jun Jul A S Oct No Dec Jan

Cisco Strategic Alliance

First Prediction Point Clients & Higher Revenues PNC Bank, Citigroup, Kone, Shiseido, Telcom Argentina 10k Hosts Deployed AWS Strategic Alliance

2017 y v eb ug ep Jan F Mar Apr Ma Jun Jul A S Oct No Dec Jan

Google Strategic Revenues Partnership Second Prediction Point 35 M $ Higher Than Forecast Higher Than Revenues Forcast Revenues 22 M $ Higher Than Forcast IBM Strategic Alliance

Dell Strategic Agreement

Revenues 46 M $ Higher Than Forecast

Figure 6.11: Salesforce Timeline.

6.6. Conclusions

In the conclusion section, we present and report the findings from the case, as [4] suggests doing for each case. First of all, we highlighted, through news and KPIs what the company considers important as performance. In particular, being a large corporation, Salesforce indeed values profit: in particular, this is highlighted by the relevance that the news gives to the announcement of higher revenues [109]. Secondly, according to the composite view on performance we developed, the performance of the company is positively influenced by the announcements of new clients, partnerships and alliances. For a B2B cloudsubscription based company like Salesforce, acquiring new clients and developing their relationships with partners is indicative of long term profit and positive performance. In later chapters, we proceed to put together and combine the findings for each case. In this case, we have seen that network measurements do predict company performance: the measurements forecast higher than expected revenues and an increase in demand, concretized as the acquisition of major clients. The main findings are two: the measurements used (6.1) anticipate the acquisition of major clients and anticipate a beyondexpectations revenue for that quarter.

• After the first batch of 10k hosts is deployed, the company acquire a few major new clients, among which Citigroup, PNC Bank, Kone, Shiseido and Telcom Argentina. Later on, new cooperations and strategic alliances with AWS, Google, IMB and Dell Technology are announced too. The first increase in hosts happened on September 2016, predicting the client acquisition announced in November 2016 [101][102]. The second batch of 10k hosts was deployed by the end of January 2017, that anticipates the announcements of new alliances, as the company is preparing 50 Salesforce

to cope with the possibly increasing demand. In fact, the acquistion of new strategic partners and alliances was not made public until late 2016 and into 2017 [103], [105], [106], [107], [108]. We identified the increase in IPv6 hosts as an increase in demand for the company and its services. It is very likely that the actual acquisition happened much earlier, and that Salesforce deployed its new IPv6 hosts to cope with the increasing demand, but that it was only announced later: this is why we select the prediction points right after the deployment. However, the acquisition of partners and alliances is not, by all means, a KPI. Nevertheless, the measurement anticipates very important information that very positively affects company performance. • The measurements anticipate a beyondforecastedrevenues for the quarters after the two batches of hosts are deployed, as Figure 6.2 shows. While we see a constant increase in profits, the com pany reports revenues beyond its own forecasts. In particular, after the company deploys 20000 new IPv6 hosts, by January 2017, the company keeps announcing revenues beyond their own forecasts. In particular, for Q1 2017 in May, they announce revenues 42 million $ higher than their own forecast. This trend continues for the next three quarters, where revenues are at least 35 million $ higher than forecasts. In this case, we can see how the increase in measurements, interpreted as an increase in demand, can predict higher revenues both in the short and in the longer term. It is a clear case where a sudden change in the measurements and a constant increase, predict the increase in demand for the company, i.e. revenues. These findings show that the measurements quite directly predict and anticipate changes in a rather meaningful KPI for the company.

The acquisition of these results from and through practical experience and data is extremely relevant for a study case of this kind. Understanding and verifying realworld phenomena against reality can help the research and its validity. Having observed these network measurements and the financial data retrieved from Salesforce, we have found and verified a connection between the two. From this case, we see that it is possible to predict company performance through the use of network measurements. The case showed that the measurements show a crucial increase in demand for the company, a crucial piece of information that can provide a meaningful market edge. It is a clear instance where measurements predict an unexpected increase in revenue, an important KPI for the company and the acquisition of major clients. While this cannot be properly linked to a specific KPI, it is a strong signal of positive performance for Salesforce’s business. We highlight as well the points in time when it is possible to acquire such knowledge on the corporation before it is made public. The measurements anticipated the acquisition of clients before it was made public and an increase in demand and capacity leading to higherthanforecast revenues. This study case showed us how a relationship between network measurements and KPIs exists and is valid to a certain extent. We had a deeper look at cloud service providers and how their performance is related to network and the Internet. This can become the starting point for a larger study relating company hardware to financial performance. We see that host monitoring could be quite insightful in predicting the acquisition of new customers or increase in profits in the cloud business. We identified when there is enough data and is possible to make such predictions. 7 Netflix

The second case study considered for this research deals with Netflix, Inc., an American corporation listed on NASDAQ as NFLX. We selected this company for the following reasons:

• The company almost completely operates through the Internet, fully embracing the ebusiness type and model. • The measurements found on the company show user traffic consumption and behaviours, and from those, we hope to extract insider information on the company. • The measurements show the launch of the service in a new market. • There is a large amount of financial data and news available on which it is possible to build a study case.

7.1. Netflix, Inc.

Netflix, Inc. is an American company in the Entertainment sector and is considered the largest media service provider in the world. It was founded in 1997 and was able to launch its website the following year. It started as a subscriptionbased DVDbymail enterprise, whose main competitor was Block buster at the time, a brick and mortar retailer. In 2002 the company went public and started to expand their operations until they hit the 1 million subscriber mark in 2003. With a B2C business model, their growth followed closely the expansion of the Internet towards the wider public. In 2007 Netflix started delivering online video content to their subscribers. Netflix produces and outsources content of various types and genres: series, movies, cartoons and documentaries. Their browser divides content into categories, has a search function, allows to save shows to watch later and has a builtin Recommended System that suggests content based on the user’s preferences. The content is often translated in other languages, either through dubbing or subtitles. From a webexclusive platform, it expanded to mobile devices, gaming consoles and smart TVs.

7.1.1. Netflix Audience Netflix sells its services for a monthly fee, with different pricing strategies, but is, in general, accessible to a wide audience. Its demographics show how the majority of its users comes from average education levels and is well distributed among population ranges [110], [111]. Before 2015 the majority of users were under 34, while after 2017 subscribers are more distributed among the population, showing the increasing popularity of the service and its appeal to most categories [110]. The service is not limited to the U.S., but is available in many countries, creating a wide variety of subscribers, with different interests. The service presence and penetration highest in Englishspeaking countries or countries

51 52 Netflix where the English language is well known in general, despite providing subtitles and dubbing. In 2018, 4 of the top 10 countries with the highest user penetration have English as their mother tongue, while the other countries are from Northern Europe, where English literacy level is particularly high [112]. In fact, Netflix provides most of its content in English, and we clearly see how countries reacted and related to this value proposition. From this, we can see and understand why Netflix creates original content in nondomestic languages, as they clearly know how the country reacts to content in their own language.

Top 10 Coutries by User Penetration 2018 Country % of digital video viewers United States 64.5% Norway 62.4% Canada 56.3% Denmark 54.9% Sweden 50.2% Netherlands 43.6% Australia 42.2% Finland 39.7% Germany 35.5% United Kingdom 33.8%

Table 7.1: Netflix Market Penetration [112]. Adapted from eMarketer,Top 10 Countries, Ranked by Netflix User Penetration, 2018, retrievd from https://www.emarketer.com/Chart/ Top10CountriesRankedbyNetflixUserPenetration2018ofdigitalvideoviewers/220373 Copyright by eMarketer.

7.1.2. International Expansion In 2010 the company decides to expand internationally, starting in Canada, then moving to Latin America and then reaching Europe in 2012. They firstly targeted Englishspeaking countries such as the UK, Canada, and Ireland, then slowly started to move to other European countries. Next, Netflix expanded to other developed countries such as Japan and South Korea to then massively expand to multiple countries at once. [113] defines it as “exponential globalization”, as they first started slowly, hitting certain countries to plan the correct approach and learn from it. Nowadays, Netflix is available in almost every country in the world (only 4 missings). There are now more subscribers abroad than in the US: 70 million over 130 total, as they were able to expand to 190 in just 7 years [113] . The number of subscribers is an extremely important KPI for the company. This growth in users and relevance made its service into one of the largest players on the network. Netflix also shows how in North America its traffic percentage went from 21% to 37% in just 5 years (20102015) chipping away other traffic sources [74]. The company has a large quantity of traffic that has to be handled properly. We already explored in Section 3.6.1 how Netflix manages to effectively, thanks to multiple CDNs, to deliver highquality content to its increasing number of subscribers [19].

7.1.3. Original Content Its originally produced and created content proved to be a strong and successful feature of the plat form. In fact, Netflix produces its own original TV series and movies that are only available on its platform. We group these series and movies together as original content. We mention the original content because the company seems to give it a lot of attention. In their quarterly reports, they dis cuss the upcoming series to their service, with particular attention to their originals. Furthermore, they also discuss whether the public appreciated or not their shows [74]. Netflix started to produce original content in 2012; this proved to be a winning strategy, especially by creating original content in the original language for the new markets in which the company launched its service. It would become routine for the company to launch its service in a country coupled with an original series in the local language; the foreign languages supported are now 23. The number of original TV series, in foreign languages as well, along with renewed seasons, kept increasing the more the company acquired sub scribers and went international. Original content has probably proven the most successful strategy for the company, as many competitors, such as Hulu, YouTube and Amazon Prime followed suit creating their own original series [114]. 7.2. Data Sources 53

In this introduction, we tackled various aspects of Netflix’s operations: original content and interna tional expansion of American multinational abroad. Here, we are already introducing various aspects that are going to be relevant later on for our case. In fact, while triangulating various data sources in Section 7.5 we what it is possible to predict from the measurements. This is possible thanks to our exploratory and instrumental approach to each case and the act that the case study scenario allows for such a flexible structure [4],[30], [32] From this brief introduction of the company, we understand how expanding its operations internationally became a turning point for the company. In this study case, we present one of such cases, Netflix’s launch in Italy, to understand its significance and to see how network measurements predicted how outstandingly the company performed.

7.2. Data Sources

We here describe the sources of the data used for the previous company introduction and the case analysis that is going to be performed next. Due to the nature of the measurement considered, we are going to consider data for the company, globally as a whole, and the data for a local market, the Italian one.

• Financial Data & KPIs: The company uses its website (not his streaming platform) to describe its activities and its business, for users and investors. The financial data and indicators used in this research have been mostly recovered from its quarterly reports. These reports are publicly avail able on their website, https://www.netflixinvestor.com/financials/quarterlyearnings/ default.aspx [74]. The financial metrics and indicators used in this case analysis were found in the company’s quarterly reports. For our analysis, we only selected the quarterly reports in a specific time frame. We consider all the quarterly reports from April 2014 to January 2019; respectively, we consider the quarterly reports from Q1 2014 to Q4 2018 included, respectively, for a total of 5 years and 20 trimesters. While not part of the overall analysis, we also used some data reported on the Q4 2013 report release, for forecastrelated figures. Figure 7.12, and 7.8 through 7.11 were made using the data in these reports; we also report the financial data gathered and used in Tables 7.12 through 7.3 [74]. These indicators, when possible, are split between the international and the domestic market, as the company provides data for both sections. Secondly, since we are also considering a smaller and national market tried to obtain the financial data for the company for its Italian segment. This has proved lengthy, and most of the material we use has been retrieved online from analytics services. The indicators used are financial, such as revenue, and KPIs, such as the number of subscriptions. • Measurements: The measurements are collected by the research group SmartData@Polito, the network association of Polytechnic of Turin. The researchers Trevisan, Giordano, Drago, Mellia & Munafo describe their techniques and findings in their paper “Five Years at the Edge: Watching Internet from the ISP Network”[54], and they have kindly granted us access to their collec tion of data. Some of these data and their findings are available on their website at https:// smartdata.polito.it/fiveyearsattheedgewatchinginternetfromtheispnetwork/. The measurements are part of a larger and broader study in the context of traffic analysis and monitoring, but we can use them to characterise and understand Netflix’s traffic and, possibly, performance. • Financial News: We also report useful information regarding the company’s business and activ ities during the selected time period. As a source for relevant financial news, we referred to the Reuters news organization (https://www.reuters.com/). As keywords, we used “Netflix” and “Italy”. Other news and content have been retrieved from various newspapers, such as The Guardian, or from business data platforms such as Statista (https://www.statista.com/) to complement the case with the needed data.

For this case, we gave equal importance to both the Italian market and the global company. The gathering of sources reflected the importance of having relevant information and metrics on the Italian market, the international segment of the company and its global operations. bigskip 54 Netflix

7.3. Measurements, Monitoring & Interpretation

7.3.1. Collection Methodology The researchers have monitored and collected for around 5 years the download data from one of the biggest ISP providers in Italy, from early 2013 to late 2017. They placed two passive probes inside two PointsofPresence in one of the largest Italian cities. Based on their previous works and studies, we assume the measurements to have taken place in Turin and the ISP to be Fastweb, one of the largest Internet provider in the country. They measured the (anonymized) traffic of 10000 ADSL and 5000 FTTH subscribers. We consider this as a good proxy for most of the Italian traffic. Figure 7.1 shows the methodology and structure used for collecting the Internet traces that make up the measurements.

Figure 7.1: Measurement Infrastructure and Processing Steps [54]. Adapted from Five Years at the Edge:Watching Internet from the ISP Network, by M. Trevisan, D. Giordano, I. Drago, M. Mellia, M. Munafo, 2018, p. 2. Copyright by M. Trevisan, D. Giordano, I. Drago, M. Mellia, M. Munafo.

While monitoring both ADSL and FTTH traffic, these probes would then send the packet information (records containing classic traffic monitoring fields such as IPs, port numbers and packet size) to their custommade traffic analyzer, Tstat. Despite some probe outages, the complex process of data identifi cation and the big quantity of data gathered (31.9 TB), their measurements are truly rich and insightful. Their research analyses the evolution and maturing of the Italian web services market, confirming how companies are moving their CDNs closer and inside ISP’s PoPs. They have passively monitored all the traffic passing through their probes and categorizing it by IP addresses. [54] grouped together all the different IPs of a company and associated them through the DNS system. For each company, the selected sets of IP address traces and is considered as traffic from that company. Characterizing website traffic through user data, the paper provides various types of data. We need to mention that the graphs and our datasets present missing data characterized as 0. This is because the researchers mention hardware failures and were not able to collect and analyse the data at that time [54]. The researchers also filtered unwanted traffic noise, considering subscribers, called active subscribers, that have only generated 10 traffic flows, with at least 15kB in download and 5kB in the upload.

7.3.2. Measurements Monitoring The researchers report the findings from the measurement in their paper and on their website, where they report per service breakdown of the measured traffic, displaying how the main webplayers shared the traffic [54]. Similarly, the researchers present the popularity of various players in the Italian Inter net, showing how Netflix starts, timidly, to show its presence on the network [54]. We were granted access to a particular dataset from [54], providing us with insightful data on how and by how much is Netflix consumed in Turin. The measurements show the daily traffic from a 7.3. Measurements, Monitoring & Interpretation 55 certain service, in our case, Netflix. We were granted access to their data and obtained information on the traffic in download, upload, the number of observed subscribers and the number of TCP/UDP flows. Netflix launched in Italy on October 2015, along with Spain and Portugal, and therefore there is almost no traffic available before that date. The announcement came earlier that year, on the 6th of June 2015 [74]. These measurements then represent an important step in the company’s expansion strategy, hence their possible relevance. Firstly, we have a look at the total traffic in download and upload recorded by the researchers, in GB/day, in Figure 7.2 and Figure 7.3 [54]. The download data represents the total quantity of bytes streamed by users, while the upload data are the requests to Netflix’s servers sent from users. We see that both upload and download traffic keep increasing quite steadily, with a couple of spikes in April and October 2017. Next, Figure 7.4 and Figure 7.5 show the total number of users and flows recorded, which follow a growing path as the total traffic. It is noteworthy to see how users have a distinctly sudden increase as soon as the service launched, to then decrease to a more stable trend. In these figures we also notice a small amount of traffic and users in 2013 and 2014, even if the service had not launched in Italy: these users were probably using VPNs to access the service.

Total Netflix Download Traffic 400

350

300

250

200 GB/day 150

100

18/03/201317/05/201316/07/201314/09/201313/11/201312/01/201413/03/201412/05/201411/07/201409/09/201408/11/201407/01/201508/03/201507/05/201506/07/201504/09/201503/11/201502/01/201602/03/201601/05/201630/06/201629/08/201628/10/201627/12/201625/02/201726/04/201725/06/201724/08/2017

Date

Figure 7.2: Netflix Total Download Traffic [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchinginternetfromtheispnetwork/ Copyright by D. Giordano.

Total Netflix Upload Traffic 6

3 GB/day

Date

Figure 7.3: Netflix Total upload Traffic [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchinginternetfromtheispnetwork/ Copyright by D. Giordano.

s 56 Netflix

Total Netflix Daily Flows

250000

200000

150000 # Flows 100000

50000

Date

Figure 7.4: Netflix Total Traffic Flows [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchinginternetfromtheispnetwork/ Copyright by D. Giordano.

Total Netflix Daily Users

1750

1500

1250

1000

# Users 750

500

250

Date

Figure 7.5: Netflix Total Users [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchinginternetfromtheispnetwork/ Copyright by D. Giordano.

Lastly, each day, the measurement represents the average daily streamed or uploaded MB per user. We have then graphed the results for Netflix, both for upload and download. In Figure 7.6 [54] we report the average downloaded quantities per user over the measurement period, from Netflix’s launch until the termination of the measurements. Excluding equipment shortages, a user downloaded 669.34 MB of data on average, with a standard deviation of 134.47 MB. Similarly, in Figure 7.7 [54], we can see the uploaded traffic towards Netflix per user, on average 1.34 MB with 1.82 MB in standard deviation.

Netflix Download per User

800

600

400 MB/day

200

Date

Figure 7.6: Netflix Download MB per User [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchinginternetfromtheispnetwork/ Copyright by D. Giordano. 7.3. Measurements, Monitoring & Interpretation 57

Netflix Upload per User 6

3 MB/day

Date

Figure 7.7: Netflix Upload MB per User [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchinginternetfromtheispnetwork/ Copyright by D. Giordano.

7.3.3. Measurements Interpretation We now start again in October 2015 and see whether by observing the measurements in Figures 7.6, 7.7, 7.2, 7.3, 7.4 and 7.5. The first step is to observe the measurement interpret them, as proposed in Section 2.3.1. From Total Download and Upload traffic (Figures 7.2 and 7.3) we see how the total traffic steadily increase in Turin, with traffic spikes in late 2017. The total number of Flows (Figure 7.4) shows a similar behaviour, while Figure 7.5, showing the number of Daily Users, shows more interesting behaviour. We can see how there are much more spikes and the trend is less linear; one of these is right in October 2015, right after the service launched, before acquiring a more stable trend. This probably shows the excitement of early adopters to access the platform. Similarly, there are various spikes and less stable trends all across 2017. We can see how consistently the total traffic and the total number of users kept increasing in Turin, which represents the Italian population to a large extent. From the download peruser data, we can see a small initial trend at around 200 MB/day per user, probably representing the test phase of the service in Italy. Then, we see the quantity increase and stabilize at around 650 and 700 MB/day. This trend and average would remain the same for the whole measurement period. A quite different pattern can be observed in Figure 7.7, representing the average upload per user, which steadily increased from 2 to 6 MB/day in just 2 years, from 2015 to 2017. These measurements, contextualized with the company in a study case, can be interpreted and provide us with information on Netflix and its usage. Downloaded bytes per user represent the video content that users stream to their own devices, and we can see that peruser consumption did not decrease but stayed the same for two whole years. The 700 MB streamed daily, per user, can roughly correspond to streaming a movie or a couple of episodes of series. We can that on average, each user spent between one or two hours on the platform each day. Considering that this quantity remains constant over 2 years, we can clearly see how subscribers, once subscribed, tend indeed to use the platform and integrate it in their routine. The user is committed to the platform and is proof that Netflix is fully able to supply the Italian demand for video content. The upload measurements (Figure 7.7) are less straightforward to interpret, as Netflix does not allow users to upload videos like other VoD video providers like YouTube. However, on the Netflix Web page, IPTV or mobile app, users can browse and perform researches for shows, and whenever a user does so, it uploads browsing and searching requests to the Netflix servers. This means that the more requests a user sends, the more time it spends browsing and looking for shows or keep browsing the platform and sending Netflix’s servers requests. We can then infer that subscribers spend more time browsing and looking for shows and movies of their liking. This user behaviour has two possible causes that can directly influence future company performance. Firstly, users finish shows and start browsing for other shows. Secondly, that there is more video content available on the platform, that users scan for shows of their likings. Overall, the measurements provided very insightful information regarding consumer behaviour and the increase in Netflix’s subscriptions and usage of the service. The constant interest and commitment that user behaviour shows through these measurements is a highly positive sign for the platform in Italy. Secondly, considering the upload data (Figure 7.7) the company is expected to increase its release 58 Netflix and production of original content, especially in Italian. It signals that the platform has been positively received and that it’s a valid option among the competition.

7.4. Performance Overview

In this section, we introduce the most relevant KPIs for the company. This case study couples measure ments from a single country with a global company. The measurements represent only Italian traffic, while the financial data recovered from the quarterly reports represents the company as a whole, di vided between its domestic and international segment. This gives us the opportunity to see how Netflix performs abroad in a foreign market, and what prediction can be made from such measurements. The Italian launch date corresponds, roughly, to the release of the Q3 2015, on the 14th October 2015. Unfortunately, there is not available measurement before that date. The Italian launch and the end of the measurements, are represented, respectively, as two black crosses in Figures 7.12 and 7.8 through 7.11, for clarity purposes. As we describe in Section 7.2, in its quarterly reports the company provides financial indicators for both its domestic and international segment. This way, it is possible to study the company’s inter national performance independently from the domestic one. Here, we see a great study opportunity, to research and study in more detail Netflix’s international market, which is much more diverse from its domestic one. This gives us the chance to use these more local measurements as predictors for a much larger section of the company.

7.4.1. Financial Performance In this section, we present and discuss what we believe are the main KPIs for a company like Netflix. It’s necessary to consider the whole performance of the company in the previous quarters and understand the company’s objectives and strategies. While certain performance indicators, such as Net Income or Earnings Per Share, refer to the whole performance of the company, when possible we have divided these performances into domestic and international. We also provide the case with quarterbyquarter forecasts that the company makes for each of its segment. Netflix’s U.S. market is already developed, and it serves as a benchmark for new and emerging international markets such as the Italian one. However, in general, indicators like revenues and profitability ratios are crucial for a quick and effective analysis of such an international corporation. Since it’s a subscriptionbased webcontent provider, we see, however, how greatly the company values such indicators. Some of the most relevant KPIs are the local number of subscribers and revenues. Firstly we observe Revenues, both international and domestic, along with the forecast for the next quarter. We see in Figure 7.8 how, up to 2015, domestic and international revenues follow a similar trend, with a 600 million $ difference. However, by the end of 2018, the company reports higher international revenues than domestic ones. In general, we notice how revenues reflect the number of subscribers quite closely. Total revenues and total subscribers show a correlation coefficient of 0.98, which is unsurprising, considering Netflix’s business model. The company also makes forecasts for the next quarter, and we can see how their forecasts, up until October 2015, are only 10 million short, usually, of the actual result, both for domestic and international revenues (Figure 7.2). We consider revenues to be a quite relevant index for a public company like Netflix, as it is indeed a good proxy for market share, userbase and sustained growth. Studying Netflix’s quarterly revenues over a 10year period, it is possible to notice interesting trends: while revenues were below the 1 billion $ marks for many years in a row, we notice how trends changed, around 2012. The revenue increase changed from linear to exponential after 2012, with the company reaching 4 billion $ in revenues in 2019 (Figure 7.2). Not surprisingly, this change of trend happened when the company decided to expand internationally and to produce its own original content. 7.4. Performance Overview 59

Revenues

2000 2000

1800 1750

1600 1500

1250 1400

1000 1200 Domestic, million $

International, million $ 750 1000 500

800 250

21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019 21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019

Date Date

Figure 7.8: Netflix Revenues [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https://www. netflixinvestor.com/financials/quarterlyearnings/default.aspx

Revenues Date Domestic International D. Forecast I. Forecast million $ 21/04/2014 799 267 835 304 21/07/2014 838 307 877 347 15/10/2014 877 346 917 388 20/01/2015 917 388 973 425 15/04/2015 985 415 1024 450 15/07/2015 1026 455 1069 524 14/10/2015 1064 517 1101 566 19/01/2016 1106 566 1160 653 18/04/2016 1161 652 1210 754 18/07/2016 1208 758 1309 846 17/10/2016 1304 853 1397 947 18/01/2017 1403 948 1471 1045 17/04/2017 1470 1046 1499 1141 17/07/2017 1505 1165 1553 1306 16/10/2017 1547 1327 1616 1553 22/01/2018 1630 1550 1807 1780 16/04/2018 1820 1782 1898 1943 16/07/2018 1893 1921 1893 1970 16/10/2018 1937 1973 1995 2119 17/01/2019 1996 2106 2064 2350

Table 7.2: Netflix Revenues, in millions $ [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https: //www.netflixinvestor.com/financials/quarterlyearnings/default.aspx

In Table 7.5 (presented later) and 7.2 it is possible to observe the actual results per each quarter and the forecast for the next one, both for subscribers and revenues. Unsurprisingly, with a monthly subscription fee of around 10 $ and the abovementioned results from revenue differences, the forecast error does not exceed the 1 million subscriptions. The error is limited and we can see that Netflix knows its market and growth expectations quite well. These two proxies, being among the most relevant for the company, show little error and a strong relation between each other. Figure 7.9 presents profitability ratios that are usually used in financial analysis as functions of Net Income, some of which were already discussed in the previous chapter, in Section 6.4.1. Return on Assets (ROA) is computed by diving Net Income by Assets, showing how profitable a company is relative to its assets, showing how well the company profits from its investments. We can see ROA to barely surpass the 0% mark, which is rather low. We consider comparisons with other industries carefully as these ratios can also be computed using a twelvemonth trailing Net Income index. However, in general, it is reported that Netflix underperforms. This lowlevel signals how the company’s profitability is driven low due to the large investments the company is taking, from expanding internationally to its originals strategy. Return on Equity is obtained by dividing the Net Income by stakeholder’s Equity, showing how effectively the company is being managed and is a widely regarded as a relevant index to forecast and plan sustained growth for a company. It shows how much revenue a company can make without requiring more funds, provided by the stockholders. Similarly, ROE is slightly below 5% trending towards 1%. The last presented ratios are the domestic and international Net Profit Margins, dividing Net Income by the respective revenues. This measure shows how much of a dollar in revenue translates into real profit. These two follow similar trends, but we can see how these margins, while still quite high for the international segment, start to decrease in 2015. These indicators are showing 60 Netflix a company that is notable, at the moment, to optimize the profit from its revenues.

Financial Ratios

20.0 22.5

17.5 20.0

15.0 17.5

12.5 15.0

12.5 10.0 Domestic NPM, %

International NPM, % 10.0 7.5

7.5 5.0

5.0 2.5

1.6 8

1.4 7

1.2 6

1.0 5 ROE, % ROA, % 0.8 4

0.6 3

0.4 2

0.2 1 21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019 21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019

Date Date

Figure 7.9: Netflix Financial Ratios [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https://www. netflixinvestor.com/financials/quarterlyearnings/default.aspx

We next consider EPS, as another relevant financial ratio for performance, and the difference be tween the actual result and the forecast, which follows similar trends (Figure 7.10). This financial indicator signals the company profitability for investors and stockholders. We see a sharp decrease between the first and second quarter of 2015, which are however due to a 7to1 stock split and its adjustment in price. We see this adjustment as a positive sign, as by increasing the number of shares, the company allows more investors to trade company equity, invest, and participate in corporate gov ernance. As of October 2015, we see EPS to be just slightly above 0 $.

EPS 1.4

1.2

1.0

0.8 $ 0.6

0.4

0.2

0.0 21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019

Date

Figure 7.10: Netflix Earnings Per Share [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https: //www.netflixinvestor.com/financials/quarterlyearnings/default.aspx 7.4. Performance Overview 61

Date EPS Domestic NPM International NPM ROA ROE 21/04/2014 0.86 6.63% 19.85% 0.88% 3.59% 21/07/2014 1.15 8.47% 23.13% 1.12% 4.41% 15/10/2014 0.96 6.73% 17.05% 0.87% 3.42% 20/01/2015 1.35 9.05% 21.39% 1.18% 4.47% 15/04/2015 0.38 2.44% 5.78% 0.26% 1.26% 15/07/2015 0.06 2.53% 5.71% 0.27% 1.28% 14/10/2015 0.07 2.73% 5.61% 0.29% 1.34% 19/01/2016 0.1 3.89% 7.60% 0.42% 1.93% 18/04/2016 0.06 2.41% 4.29% 0.25% 1.21% 18/07/2016 0.09 3.39% 5.41% 0.35% 1.70% 17/10/2016 0.12 3.99% 6.10% 0.42% 2.06% 18/01/2017 0.15 4.78% 7.07% 0.49% 2.50% 17/04/2017 0.4 12.11% 17.02% 1.24% 5.99% 17/07/2017 0.15 4.39% 5.67% 0.40% 2.12% 16/10/2017 0.29 8.40% 9.80% 0.77% 3.91% 22/01/2018 0.41 11.41% 12.00% 0.98% 5.19% 16/04/2018 0.64 15.93% 16.27% 1.44% 7.21% 16/07/2018 0.85 20.29% 19.99% 1.69% 8.54% 16/10/2018 0.89 20.81% 20.43% 1.72% 8.05% 17/01/2019 0.3 6.71% 6.36% 0.52% 2.56%

Table 7.3: Netflix Financial Ratios & EPS, in $ [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https://www.netflixinvestor.com/financials/quarterlyearnings/default.aspx

Lastly, in Figure 7.11 we can see Operating and Net Income (OI and NI). These are some crucial financial KPI for most companies, as Operating Income represents the profit from companies opera tions, contributing to describing company efficiency and is often considered as one of the main KPIs: it trends, quite steadily, around 100 million $. Net Income can represent the company’s bottom line for revenue purposes, and we can see its decreasing trend of around 50 million $. After 2017, however, OI quadruples in five years, going from 98 million $ in Q1 2014 to 481 million $ in Q3 2018. Similarly, NI increases significantly as well, going from 24 million $ in Q1 2015 to 403 million $ in Q3 2018.

Net Income and Operating Income 500 400

350 400 300

250 300

200 NI, million $ OI, million $ 150 200

100 100 50

Date Date

Figure 7.11: Netflix Operating Income & Net Income [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https://www.netflixinvestor.com/financials/quarterlyearnings/default.aspx 62 Netflix

Date OI NI OI Forecast NI Forecast millions $ 21/04/2014 98 53 125 69 21/07/2014 130 71 103 55 15/10/2014 110 59 57 27 20/01/2015 65 83 79 37 15/04/2015 97 24 59 16 15/07/2015 75 26 81 31 14/10/2015 74 29 49 10 19/01/2016 60 43 50 11 18/04/2016 49 28 47 9 18/07/2016 70 41 64 22 17/10/2016 106 52 126 56 18/01/2017 154 67 239 165 17/04/2017 257 178 120 66 17/07/2017 128 66 204 143 16/10/2017 209 130 238 183 22/01/2018 245 186 362 282 16/04/2018 337 290 349 358 16/07/2018 462 384 420 384 16/10/2018 481 403 205 105 17/01/2019 216 134 400 253

Table 7.4: Netflix OI & an NI, in in millions $ [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https://www.netflixinvestor.com/financials/quarterlyearnings/default.aspx

The stock split in Q2 2015 is a sign of confidence from the company in the future growth and profitability of the company. Similarly, the revenues growth rate increased slightly but remained sub stantially unchanged. While we are aware of the expansion programme that Netflix is going through and reaching a wider international audience in Europe and Asia, there are some possibly worrying signs. We see a stagnating Operating Income, Net Income, both around 50 million $ for the past year and decreasing. At the same time, there are decreasing ROA, ROE and NPM. This indeed signals how the company is focusing on expansion rather than profitability, from which we can assume that the future of the company depends on the success or failure of this strategy. Furthermore, in 2015, the company still has a moderate international audience, mainly from Latin America and northern Europe, which all grouped together still forms a much smaller audience than the domestic one. It signals a smaller and weaker market penetration abroad, probably still due to the youth of the market.

7.4.2. Other KPIs In this section, we consider other KPIs that are relevant for a SVoD platform like Netflix. In Figure 7.12 we present the domestic and international subscribers over the considered time period. We see how international subscribers started to grow steadily, following Netflix’s expansion strategy, as mentioned in Section 7.1.2. We have to consider that the company is still in the middle of its expansion strategy when the measurements are taken, as the two black crosses represent. So, the company is investing large sums in infrastructure for its expansion in the international markets and its production of original content. However, we can see the steady growth in subscribers in the past few years and the moment when international subscribers surpass domestic ones. 7.4. Performance Overview 63

Subscribers

55 70

60 50 50

45 40 Domestic, millions

International, millions 30 40 20

35 10 21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019 21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019

Date Date

Figure 7.12: Netflix Subscribers [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https://www. netflixinvestor.com/financials/quarterlyearnings/default.aspx Subscribers Date Domestic International D. Forecast I. Forecast millions 21/04/2014 35.5 12.5 36 13.5 21/07/2014 36 13.5 36.5 16 15/10/2014 37 15 39 18 20/01/2015 39 18 41 20.5 15/04/2015 41 20 42 22 15/07/2015 42 23 43.5 25.5 14/10/2015 43 26 45 29.5 19/01/2016 48 30 46.5 34 18/04/2016 47 34.5 47.5 36.5 18/07/2016 47 36 47 38 17/10/2016 47.5 39 49 43 18/01/2017 49.5 44 51 48 17/04/2017 51 47.5 51.5 50.5 17/07/2017 52 52 52.5 55.5 16/10/2017 52.5 56.5 54 61 22/01/2018 54.5 63 56 68 16/04/2018 56.5 68 58 73 16/07/2018 57 72 58 77 16/10/2018 57 78.5 60 86 17/01/2019 58 80 60 88

Table 7.5: Netflix Subscribers, in millions [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https: //www.netflixinvestor.com/financials/quarterlyearnings/default.aspx

Secondly, we also consider the development and release of the original content. It is a strategy that Netflix is using to gain share both in domestic and international markets is the production of original content. Already in 2015, we can see that the company is the largest producer of exclusive content (Figure 7.13,[115]).

Figure 7.13: Netflix New Original Series [115]. Adapted from Parrot Analytics, The Global TV Demand Re port, 2018, retrieved from https://insights.parrotanalytics.com/hubfs/Resources/whitepapers/ Parrot%20Analytics%20%20The%20Globa%20TV%20Demand%20Report%202018.pdf?hsCtaTracking= d6e808ee731e40198ad3650e9425f54c%7C497e4b088afb4c7ab746ff160885da2c Copyright by Par rot Analytics.

And from Figure it, we see how in 2015 Netflix started releasing much more original content, and competitors followed suit on this strategy [114], [115]. The release of all these original series happened almost at the same time as the international expansion process started a few years earlier, along with the launch in Italy. 64 Netflix

7.4.3. Italian & International SVoD Market To complete an overview of the company and what the measurements predict, we also need to contex tualize the measurements used. Since the measurements were take in Italy, we introduce briefly the Italian media service market. The past 5 years have seen a decline in traditional television and the rise of Over The Top media services in Italy. Webenabled televisions, computers, and internetenabled devices are the main means for OTT services. The other competing technology is payTV: these two can be considered substitute products. OTT and payTV are then differentiated in 2 categories, TVoD and SVoD: Transactional Video on Demand, where a onetimefee is paid for limited exclusive con tent, or, Subscription Video on Demand, where a monthly fee is collected instead; Netflix is an OTT SVoD. In 2016, SVoD users were only about 2 million [116], and revenues were modest: payTV is still dominating the market in Italy. The Italian OTT market, however, is a quite peculiar one, for two reasons. The first is the rather slower adoption rate of innovations and disrupting technologies in Italy [117]. In general, it takes longer for new services to penetrate, especially in the digital section; we could identify this as a general distrust to embrace the new. Digitalization in all sectors, from entertainment to bureaucracy, is happing at a slower pace, as Italians still have to be convinced of the convenience of such means [118]. The second reason is an already wellestablished OTT market against which Netflix has to compete. There is a free section that includes global names like YouTube or Italian broadcasters that make available their content on the web such as RaiPlay, Mediaset.it, DPlay or La7.it [116]. Paid services are Tim Vision, Infinity, Chili TV, DAZN, Now TV and Amazon Prime [116]. However, the ondemand payTV segment was an already established one that many users might not be willing to abandon. It is also interesting to notice how many of the abovementioned services are Italian and mostly offer content in the local language. Globally, a similar trend can be observed. [119] shows how global SVoD market users are expected to increase in the next years; similarly, SVoD revenues could to increase much faster than other VoD services. Only in 2017, revenues increased by almost 4000 million $, while users increased by 50 million in the SVoD market. From the measurements considered and the global results observed, the SVoD market is growing all over the world, and in Italy as well.

7.5. Analysis & Results

The goal is to see what the measurement can predict on the company, its performance or provide insider information before it is made public. We combine the considerations and the interpretations of the measurements made in section 7.3.2 and the observations on company performance made in section 7.4. We check each KPI and see it could be predicted by the measurements at a certain point in time. We do so by triangulating our data with the key assumptions of our research (Chapter 1), and proceed with the patternmatching logic suggested by [4].

7.5.1. Analysis We start by considering revenues, and in particular, international ones. From Netflix’s launch in Italy in late 2015, we see them quadruple from 500 million $ to roughly 2 billion $, in about three years. The measurements show us that Netflix users in Italy are increasing and remain committed to the platform (Figures 7.5 and 7.6). At the turn of 2017, the company really started to profit from its international expansion. In Q4 2017, the company starts to acknowledge their own results: ”We believe our big investments in content are paying off”[74]. International revenues in Q4 2018 (Figure 7.8), hit a total of 2106 billion $, surpassing domestic ones. Constant growth and expansion of the company towards its international segment are clearly noticeable. Despite possibly being slightly slower in innovation adoption, Italians are switching to services like Netflix increasingly. The crucial question is whether the growth in Italy can predict such growth in revenues too. Next, we consider the abovementioned profitability ratios, Net Income and Operating Income: we do so because, financially, they are all functions of net income. After the measurements are taken, we see these ratios slightly increase while we monitor the measurements and then experience 7.5. Analysis & Results 65 a very meaningful growth ins 2018, after the end of the measurements. By that time, it is clear that the company is recovering its expansion costs and is starting to profit greatly from it. Net Income, representing profits, starts to increase at the beginning of 2017, or Q1 2017 (Figure 7.11). Following this indicator, other financial ratios start an upward and positive trend (Figure 7.10 and Figure 7.9). international and domestic NPMs surpass 15%, while ROE goes beyond 5%. But our measurements only represent the Italian traffic, and these indicators are from the global performance of the company. We question if these measurements are predictive enough of already developed SVoD markets like Netflix’s domestic one in the U.S. The last financial KPI considered is EPS which follows very similar trends to the ones mentioned above. After modest growth in 2017, they start to increase steadily throughout 2018. Again, EPS is a rather influential KPI that can be subject to corporate policies and it seems hard to predict from measurements coming from a new market. Another crucial KPI for the company are subscriber, and in particular, we look at international sub scribers. They follow very closely the results obtained by international revenues, which is unsurprising considering Netflix’s business model and the 0.98 correlation between the two. Like revenues, interna tional subscribers surpass domestic ones; it happened in Q3 2017, as the Netflix traffic keeps growing in Italy. Similarly, can these measurements predict such an increase in international revenues at a certain point in time? It would seem so, considering that the Italian one is quite hard to penetrate, due to the slower adoption ratios of innovations. We would expect, therefore, Netflix to perform much better in other international markets that it plans to enter after the Italian one. Lastly, while not being exactly a KPI, we had a look at Netflix’s original content and how it could relate to the measurements. As mentioned, Netflix gives a lot of attention to its own original content released on the platform in its quarterly reports [74]. The company values how well these series are received, as this way it learns how better it can supply its subscribers, knowing what they might enjoy watching and hopefully increasing their subscribers. While researching original content and observing the measurements, we have noticed that Total Download Traffic presents two noticeable traffic spikes in 2017. They are perfectly timed with the release of a specific series. The April 2017 traffic spike is probably due to the release of the teen drama 13 Reasons Why, released at the beginning of the month [74]. Similarly, the second noticeable traffic spike in September 2017 coincides with the release date of the first Netflix Italian Original, Suburra: Blood on Rome [120]. The measurements, therefore show us that these two shows were highly appreciated by the Italian public. Furthermore, this information was not made available until the next quarterly reports.

7.5.2. Results Considering the previously made analysis, we have found two instances where the measurements proved relevant and insightful to predict the performance of the company. We have established that there is a strong correlation between revenues and subscribers and that the measurements predict an increase and growth of both, internationally. The data shows that To tal Traffic, Users and Flows increase throughout the measurement period, while the per User Traffic shows how users remain committed to the platform and keep using it. The measurements clearly hint ant predict that Netflix will increase its number of subscribers and therefore revenues, as already dis cussed. In Figure 7.14 we show its number of international subscribers and the total download traffic. We highlight a red in the section that corresponds to when the measurements were taken. We can clearly see that the International subscribers keep growing steadily along with Netflix’s traffic. We see total download traffic, and the other measurement considered in the analysis, predict an increase in international subscribers and revenues for Netflix, during and after the measurements are taken. In the graph, we highlight Netflix’s year and a half in Italy when the traffic kept growing throughout the whole measurement period. We select this point as, by April 2017, there is more than enough mea surements and traffic to clearly see that Netflix is going to steadily increase its subscribers, in Italy and internationally. We select this time in point as the prediction point for this result, because by that time, there is already a year and a half data of measurements predicting Netflix’s strong success abroad, in terms of subscribers hence revenues. At this point in time, it is possible to know beforehand the company’s success in the international market, before the company announces it in its future quarters. 66 Netflix

International Subscribers and Total Traffic 400 80 1.5 Years of Sustained Traffic Growth 350 70 300 60 250

50 200

40 150 GB/day, millions Subscribers, millions 30 100

50 20 1.5 Years of Sustained Traffic Growth 0

21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019 18/03/201317/05/201316/07/201314/09/201313/11/201312/01/201413/03/201412/05/201411/07/201409/09/201408/11/201407/01/201508/03/201507/05/201506/07/201504/09/201503/11/201502/01/201602/03/201601/05/201630/06/201629/08/201628/10/201627/12/201625/02/201726/04/201725/06/201724/08/2017

Date Date

Figure 7.14: Netflix International Subscribers and Total Download Traffic [74], [54]. Adapted from Netflix, Finan cials Quarterly Earnings, retrieved from https://www.netflixinvestor.com/financials/quarterlyearnings/ default.aspx. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https://smartdata.polito.it/ fiveyearsattheedgewatchinginternetfromtheispnetwork/ Copyright by D. Giordano.

In fact, looking at the international and Italian results, the measurements correctly anticipate Net flix’s strong success abroad, verified by the company’s reports in 2017 and 2018. Table 7.6 [121], [122] shows how Netflix performed in Italy in terms of yearly revenues and subscribers. From a mod est number of users in 2015, the platform reached more than 800.000 subscriptions in 2017 [116] and expected to hit 2 million in 2019: Netflix managed to become a market leader in just a few years, obtaining one million subscribers in about two years. Similarly, in Figure 7.15, international subscribers and revenues kept growing as well, as predicted by the measurements. In the quarters after April 2017, Netflix keeps announcing, in its reports, higher revenues and a higher number of subscribers. Ultimately, as we see in Figure 7.15, Netflix’s international subscribers surpass its domestic ones in July 2017, almost 2 years after the launch in Italy and the start of the measurements. After that, interna tional revenues also surpass domestic ones, in July 2018, exactly one year after the subscribers. This validates the predictions that the measurements already made in 2017, as the growing Italian traffic is followed by the international segment outperforming the domestic one. These announcements are made in Netflix’s quarterly reports, but the Italian measurements already show such trends. The pre diction point was chosen because there were already enough measurements to predict Netflix’s growth internationally and Italy in the next few years. At this point in time, it is possible to know Netflix’s future increasingly positive performance in the international segment. At that point, we could correctly expect the international segment to become more profitable than the domestic one, which then happens in about one year.

Italy million $ Year Revenues Subscribers 2011 0 0 2012 0 0 2013 0 0 2014 0 0 2015 0 0 2016 22.5 0.441 2017 70 0.899 2018 119.6 1.347

Table 7.6: Netflix Subscribers and Revenues in Italy [121], [122], [123]. Adapted from Nakono, Netflix Streaming Subscribers 2019, retrieved from https://fusion.nakono.com/data/netflixstreamingsubscribersitalyannual and https://fusion.nakono.com/data/netflixstreamingrevenuesitalyannual Copyright by Nakono 7.5. Analysis & Results 67

International Results

80 2000 1.5 Years of Sustained Traffic Growth 70 1750

60 1500

1250 50

1000 40 Revenues, millions 750 Subscribers, millions 30 More International Than Domestic 500 20 More International Than Domestic 250 10 21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019 21/04/201421/07/201415/10/201420/01/201515/04/201515/07/201514/10/201519/01/201618/04/201618/07/201617/10/201618/01/201717/04/201717/07/201716/10/201722/01/201816/04/201816/07/201816/10/201817/01/2019

Date Date

Figure 7.15: Netflix International Results [74]. Adapted from Netflix, Financials Quarterly Earnings, retrieved from https: //www.netflixinvestor.com/financials/quarterlyearnings/default.aspx

In the analysis, we have also seen that spikes in total download traffic corresponding to the release of specific original series only available on the platform. One is in English, 13 Reasons Why, while the other is in Italia, i.e. is an original series in the local language: Suburra: Blood on Rome. We can see how these spikes in the measurements coincide with these series in Figure 7.16; similarly, traffic spikes in the same period can be observed as well in Uploads (Figure 7.3), Daily Flows (Figure 7.4) and Daily Users (Figure 7.5). In 7.16 we show when the series is released in red; then, we notice the two spikes that last for a couple of weeks. After these spikes, it is possible to know, before Netflix announces it, that these series were a success. Table 7.7 shows exactly the timings described: we see when a series is released, and when a spike in traffic is observable, which, in both cases, is shortly after the release of the series. The prediction point in this case is identified as the Traffic Spike highlighted in the table. The table lastly shows when the company releases its next quarterly report, announcing the success of each of the considered series, respectively: this confirms the series’ success. It is notable how users display a peak in traffic right after the release of the series, while the company announces its success months later in their quarterly reports. By looking at the measurements it is clearly possible to see when a certain series is successful before the company announces it.

Total Netflix Download Traffic Suburra 400 13 Reasons Why 350

300

250

200

GB/day Launch in Italy 150

100

Date

Figure 7.16: Netflix Total Download Traffic [6], [120]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https://smartdata.polito.it/fiveyearsattheedgewatchinginternetfromtheispnetwork/ Copyright by D. Giordano.

Original Show Release Date Traffic Spike/Prediction Point Success Announcement Date 13 Reasons Why 31/03/2017 1/05/2017 17/07/2017 Suburra 6/10/2017 10/11/2017 22/01/2018

Table 7.7: Netflix Original Shows & Key Dates 68 Netflix

The measurements show how subscribers in foreign markets positively respond to the original content in their own language: we have seen a substantial increase in traffic and daily users. It is therefore clear that these two series were wellreceived and appreciated by the public. Furthermore, we are able to access and observe this data as it happened. By monitoring the measurements, we clearly see a spike in traffic, and we do not need to wait for a later Netflix announcement reporting the popularity of these series [74]. The measurement can tell us in advance that these two series were highly appreciated before the company made such an announcement. Furthermore, we see that original content was highly appreciated by Italians, and in particular content in the local language. This is quite crucial information for Netflix, as it helps them plan for future expansions and produce content that appeals to its subscribers. Furthermore, knowing what users liked and disliked, is crucial information for the company’s expansion strategy and campaign to attract more subscribers. We see, in this case, that measurements can help the observer predict an increase in original content and original series in the local language in the future years. This prediction turns out to be true too, as we can see in Figure 7.13, the company largely increased its number of original content in 2018. Certain KPIs were discarded as not predicted by the measurements or not insightful. In particular, we discarded EPS for the reasons mentioned in the analysis, as there are too many factors that can possibly influence it. Similarly, profitability ratios have also been discarded, as they depend too much on other parameters such as equity, costs or assets that measurements cannot assess. For instance, all these parameters start to increase in 2018, but so does revenues, as they keep growing fast both domestically and internationally. Furthermore, KPIs such as Net Income might have started to increase as the company has finished most of its initial investments in foreign countries. In fact, the bulk of Netflix’s international expansion happened before 2018, where they entered almost every country in the world. This is why we did not select such KPIs as predicted by the measurements: revenues and subscribers have a much more direct and immediate connection.

7.5.3. Netflix Timeline In Figure 7.17, we constructed a timeline of the measurements, and the KPIs predicted by this case. We divided the timeline per year, month, and relevant quarterly report reported as QX in the Figure. Furthermore, we represent in cyan the relevant measurement period and collection. The prediction and relevant information regarding original content are displayed in cyan; the predicted KPIs increase and relevant measurements are in red. Furthermore, we highlight in cyan the point in time where there are enough measurements to predict and increase in subscribers and revenues. Similarly, we highlight when the knowledge of the series’ success is available from the measurements and not of public domain yet. With this timeline, we highlight the behaviour of the measurements and how it interacts with the KPIs, showing when a certain prediction is possible and when a certain piece of information is available before is announced. 7.6. Conclusions 69

Netflix Q1 Timeline Q2 Relevant Q3 Measurement Stable per User Download and Period Q4 Increasing per User Upload

Netflix Launches in Italy

2015 y v eb ug ep Jan F Mar Apr Ma Jun Jul A S Oct No Dec Jan

Total Download, Flows and Users Increase Stedily

2016 y v eb ug ep Jan F Mar Apr Ma Jun Jul A S Oct No Dec Jan

30 Million International Subs

Total Download Traffic Spike Total Download Traffic Spike

2017 y v eb ug ep Jan F Mar Apr Ma Jun Jul A S Oct No Dec Jan

Stable Growth of Revenues and Subscribers Release of Suburra: Blood on Rome Release of 13 Reasons Why Suburra Prediction Point Subscribers Prediction Point: Enough Measurements Available

International Subs Surpass 13 Reasons Prediction Point Domestic Subs

2018 y v eb ug ep Jan F Mar Apr Ma Jun Jul A S Oct No Dec Jan

60 Million International Subs

International Revenues Surpass 2 Billion $ Domestic Revenues International Revenues

Figure 7.17: Netflix Timeline.

7.6. Conclusions

In the conclusion section, we present the main findings of the case and discuss how they contributed to answering the main research question. We here point out the main findings from the network mea surements in this case. Firstly, throughout the case, the measurements and the company’s quarterly reports, we have a clearer picture of what the company measures and values as performance. Like Salesforce, parameters such as revenues and profits are indeed essential for Netflix. However, consid ering its business model and market, the company values other KPIs such as the number of subscribers in their different geographical market section. Furthermore, we have also seen, in their quarterly re ports, how the company always discusses their new original upcoming series and whether the public appreciated them or not. This is not exactly a KPI, but an important assessment of the company’s product. In this case, too, network measurements predict company performance: the measurements predict an increase in international subscribers and revenues. There are two main findings, one of which is strictly related to a KPI while the other belongs to a broader connotation of performance and is not mathematically quantifiable. 70 Netflix

• The measurements in Italy show that users are committed to the platform, use it regularly, appreciate its content and that, most importantly, their number is growing. Starting from its launch in October 2015 and throughout the measurement period, it is possible to predict Netflix’s success and growth in Italy and internationally. Figures 7.6 and 7.7 show user behaviours and trends. Users are committed to the platform and their shows, they spend between 1 and 2 hours on it, and the time that the spend browsing the platform and selecting their shows has increased. Next, we see that the Total Download traffic, Number of Flows and Number of Users within the city of Turin is steadily increasing Figures 7.2, 7.4 and 7.5. Netflix is steadily increasing its userbase and revenues in Italy, a quite difficult market to enter as we have seen. We selected a prediction point one and a half year after the launch in Italy, as there are enough measurements and data to predict Netflix’s success and userbase and revenue growth in Italy and abroad, as Table 7.6 and Figure 7.15 show. These measurements predict Netflix’s success abroad, by predicting an increase in international subscribers, and therefore, international revenues. This sustained growth is verified by the results obtain by the company in 2018, as Figures 7.12 and 7.8 show. The measurements anticipate with one year advance that Netflix’s international segment would become more important than its domestic one. Between 2017 and 2018, in fact, international revenues and subscribers would surpass domestic ones, the Italian growing traffic predicts. In this case, the measurements predict the steady growth of subscribers and revenues for the company, with a oneyear advance. • The second prediction that the measurement show regards the original content released by the company. We see in Figure 7.16 that 2 spikes in traffic, lasting for at least 2 weeks, corresponding to the release on the platform of new series originally produced by Netflix. In particular, a spike in May 2017 corresponds to the release of the original series 13 Reasons Why, while the spike in September 2017 corresponds to the release of the first Italian original series, Suburra: Blood on Rome. In both cases, we see from the measurements that Daily Flows, Users and Total Download Traffic (Figures 7.4, 7.5 and 7.16) display a spike after the release of these series. We consider this traffic spike as the prediction point for the success of these series, i.e. when their success is already known before the company announces it the next quarter. This clearly shows that these services were highly anticipated and appreciated by the public before it was made public in their quarterly reports [74]. In particular, Suburra: Blood on Rome, is a series in the local language, Italian, and not the mother tongue of the platform, English. The measurements show that series in the local language is highly appreciated by users. At this point in time, the measurements predict an increase in Netflix original content, both in English and in other foreign languages, to better tackle the international market. This prediction is possible to make after the two spikes in traffic, i.e.in late 2017. As it is possible to verify in Figure 7.13 and from [124], the number of original content that Netflix released increased, and many series in a foreign language were produced.

In this case study we see how the measurement predicts a sustained growth of subscribers and revenues internationally and in Italy for Netflix. The steady increase in users and traffic signal a national and international growth of subscribers and revenues. Secondly, we see that it is possible to know in advance which series the public appreciated the most. This then led to the correct prediction of an increase of original series. It is possible to notice how the measurement can predict company performance and provide insider information on its service before it is released. Even if not related to a specific or standard KPI, the release and success of original content is a meaningful performance parameter for Netflix. its market and its growth. Also, we identified when there is enough data and is possible to make such predictions or when such predictive information is available before being public. Predicting it and having access to insider information before its release provides a significant market edge for the observer. Ultimately, we have seen that network measurements can predict certain indicators for the company. 8 Snapchat

The last company considered as a third study case is Snap Inc., an American public company listed on the NYSE stock exchange as SNAP. The company is chosen for three main reasons:

• It’s the main product is a popular smartphone app with millions of users across the world. • It operates on the through the Internet. • Financial data and news reports on the company are mostly public and available online.

By quickly observing the measurements we gathered on this company, we decided to consider them as viable to build a case study around. The measurement collected refer to the company’s traffic in Italy but could prove insightful globally as well.

8.1. Snap Inc.

Snap Inc. is an American software corporation founded in 2011 by three Stanford University students, Evan Spiegel (its current CEO), Bobby Murphy, and Reggie Brown. The company started off as a small tech startup that developed a social media app called “Snapchat”. The company grew rapidly, reaching 75 million users worldwide in 2015. Already in 2017 Snapchat was recognized among the 4 most important social networks [125], along with Facebook, Instagram and Twitter. From a small startup to a large corporation, the company went IPO on the 2nd of March 2017, on the NYSE stock exchange. The company’s main product is its application, which rocked the social media market with a brand new product with unique features, rapidly creating a large and committed user base. To understand its success and popularity, it is necessary to understand its product.

8.1.1. Snapchat Snapchat does not claim to be a messaging app, but an application with a phone camera at its core. Quoting its founder Evan Spiegel: ”Snapchat isn’t about capturing the traditional Kodak moment. It’s about communicating with the full range of human emotion not just what appears to be pretty or perfect.”[75]”. Firstly only available on the iOS operating system, it is a social media app for smart phones that aims at communicating through pictures rather than text. The application is completely built around the camera and enhances its possibilities. At its core, the applications allow the user to enhance the pictures and videos he takes, through technologies like facial recognition and augmented reality. This allows users, according to Snap Inc., to communicate more thoroughly and with more emotions that normal text allows. A picture can be enhanced with text, colours, augmented reality and filters of various kinds. Users can communicate and chat with each other and send pictures that only can only be viewed once, known as Snaps. Another feature, known as Stories, allows users to share their daily activity with all their friends for a limited amount of time, as the uploaded video or

71 72 Snapchat picture would oy be visible to others for 24 hours only. Snapchat introduced the concept of volatility and ephemeral content in social media. Another feature is SnapMap, allowing users to see their friends location in realtime. Otherwise, users can use the Discover feature to follow the feeds from celebrities, news and entertainment outlets. Lastly, the app tries to keep its users engaged through Snapstreaks, allowing users access to special and unique emojis and features if they keep “snapping” with a friend for three days back and forth [126]. Back in 2011, the company offered unique features and possibilities that generated a lot of curiosity on this social media application.

8.1.2. Demographics The application was and is extremely popular among teenagers and young adults, as they represent the majority of their users. Figure 8.1 provides an overview of Snapchat’s demographic, in Italy and in the U.S as of 2018. It is possible to see the total Snapchat Italian users divided by their age brackets. Similarly, it is possible to see how much of the U.S. population in that age bracket use the app.

Total Usage in Italy Age Group Total Usage in U.S. 40 80

35 70

30 60

25 50 % 20 % 40

15 30

10 20

5 10

0 0 13-18 19-24 25-29 30-35 35+ 18-24 25-34 35-44 45-54 55-64 65-75 75+ Age Bracket Age Bracket

Figure 8.1: Snapchat Demographics In Italy and the U.S. [126][127]. Adapted from JuliusDesign and Statista , retrieved from https://juliusdesign.snap/29503/ snapchatstatistichenumeriutentiinitaliaenelmondocasestudy/ and https://www. statista.com/statistics/814300/snapchatusersintheunitedstatesbyage/

We can immediately see the popularity of this platform between teenagers and young adults. The app usage is almost disproportionate, as it is possible to notice how heavily the app is used by younger generations and how it is basically unknown to adults. [128] studied how the app is used mostly by teens and young adults and the role that it plays in interpersonal relationships. Many of the various reasons found for using Snapchat was to communicate with friends, romantic partners, parents, and close relationships. Respondents described the platform as a way to communicate with few people, an entertaining way to chat through pictures that can better convey emotions that texts: “It’s a text that you can show your emotion”[128], p. 598. It is considered a youth culture phenomenon through new media, able to enhance connections and relationships within the social sphere of the user, in a direct and unexposed way [128].

8.2. Data Sources

For this case, we used publicly available financial data and news, coupled with the network measure ments collected on the company.

• Financial Data & KPIs: we collected financial data and other useful performance indicators from the quarterly reports that the company released. Snap Inc. went IPO in March 2017, therefore the quarterly reports are available online only starting from that point in time. The reports are available online at https://investor.snap.com/companyprofile [75]. Similarly to previous cases, we also read and studied the reports for other interesting and valuable information on the company’s internal policies and practices. We selected eight quarters, from Q1 2017 to Q4 8.3. Measurements Monitoring & Interpretation 73

2018, giving us an overview of the company on its first two years as a publiclytraded corporation. Unfortunately, before that date, there is no financial information directly available from Snap Inc.; other sources provide some financial data before that date, but it did seem scarce and unreliable. Figure 8.9 through 8.13 and Table 8.1 through Table 8.14 were made from this data [75].

• Measurements: The measurements come from the same source as Snapchat’s one, from the research group SmartData@Polito used in the paper “Five Years at the Edge: Watching Intersnap from the ISP snapwork”[54]. Further details and information are already described in Section 7.3.2. Therefore, the measurements report Italian Snapchat traffic from March 2013 until late 2017.

• Financial News: Similar to other cases, financial news on the company’s operations and busi ness activities is provided. We have already seen how financial news was insightful and comple ments the case study. We searched news on the company in the considered time period on the news organization Reuters (https://www.reuters.com/). The keyword used in the search bar was “Snapchat” and we collected relevant information to the case.

Having gathered this data and information we proceed to compile and build the case on Snap Inc.

8.3. Measurements Monitoring & Interpretation

Having already discussed and analysed in detail the measurement collection method and their spec ifications in Section 7.3, we can directly proceed to present the measurements for Snapchat. The measurements show the daily traffic from a certain service, in our case, Snapchat. We have been granted access to their data, and obtained information on the traffic in download, upload, the number of observed subscribers and the total number of TCP/UDP flows. We need to point out that there is some missing data in the measurements, due to hardware failures, as [54] explain. The missing data has been replaced by a 0 value; despite these failures, the measurements already show welldefined trends that are not disrupted by this missing data and do not generally affect readability and reliability of the data.

8.3.1. Measurement Monitoring Firstly, we show the total download and upload traffic, in GB/day, in Figure 8.2 and Figure 8.3 [54]. Downloaded bytes represent the incoming traffic that users request on their devices. This includes, mostly, videos and pictures that are being watched and streamed by users, on their feeds or direct messages. Uploaded bytes, instead, is the outbound traffic, representing the uploads that users make on the app. Similarly, uploaded bytes represent the videos and photos uploaded by users to Snapchat’s servers. It is possible to see how both types of traffics, and in particular the download traffic, present a bellshaped curve. In both graphs, the curve roughly starts in spring 2015 and terminates at the end of 2017. We can clearly see how the measurements roughly peak in summer 2016. We can clearly see when and how the app started to become popular in Italy, when it peaked and when its popularity started to decrease. 74 Snapchat

Total Snapchat Download Traffic 60

30 GB/day

19/03/201318/05/201317/07/201315/09/201314/11/201313/01/201414/03/201413/05/201412/07/201410/09/201409/11/201408/01/201509/03/201508/05/201507/07/201505/09/201504/11/201503/01/201603/03/201602/05/201601/07/201630/08/201629/10/201628/12/201626/02/201727/04/201726/06/201725/08/201724/10/201723/12/2017

Date

Figure 8.2: Snapchat Total Download Traffic [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https://smartdata.polito.it/fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano.

Total Snapchat Upload Traffic 3.0

2.5

2.0

1.5 GB/day

1.0

0.5

0.0

Date

Figure 8.3: Snapchat Total upload Traffic [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano.

Next, we present the total number of TCP/UDP flows recorded and the total number of daily users in Figure 8.4 and Figure 8.5 [54]. While the number of flows follows a similar trend as the above mentioned data, the total number of users does not. We can see how the total number of daily users, however, starts to increase constantly already in 2014, and do not diminish significantly after the peak in 2016. We also notice an unexpected peak in late June and early July 2017 in Figure 8.3, which is probably due to the release of the SnapMap feature, on the 21st of June 2017, and we can guess users trying it out and then disabling the feature.

Total Snapchat Daily Flows

40000

30000

# Flows 20000

10000

Date

Figure 8.4: Snapchat Total Traffic Flows [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano. 8.3. Measurements Monitoring & Interpretation 75

Total Snapchat Daily Users

800

600

400 # Users

200

Date

Figure 8.5: Snapchat Total Users [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano.

Lastly, we present in Figure 8.6 and Figure 8.7 [6] the average daily download and uploaded traffic per user, in MB/day. From these two graphs, in particular, we can understand and observe user be haviour in detail. However, these two graphs show a similar trend to the one observed in Figures 8.2 and 8.3, unsurprisingly.

Snapchat Download per User

MB/day 40

Date

Figure 8.6: Snapchat Download MB per User [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https://smartdata.polito.it/fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano.

Snapchat Upload per User

1.0

0.8

0.6 MB/day 0.4

0.2

0.0

Date

Figure 8.7: Snapchat Upload MB per User [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https:// smartdata.polito.it/fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano.

While studying and building the case, we understood that it would be interesting to provide and observe traffic from another service: Instagram. Instagram is another social media platform, owned by Facebook Inc., that focuses on pictures as well. From the same dataset, Figure 8.8[6] show the daily downloaded bytes per user. 76 Snapchat

Instagram Download per User

140

120

100

MB/day 60

Date

Figure 8.8: Instagram Download MB per User [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https://smartdata.polito.it/fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano.

From Instagram’s traffic, we can see how it slowly increased its traffic and popularity in the Italian social media market. Then, in late summer and autumn 2016, the social media’s traffic per user started to increase exponentially. In September 2019, we can see how Snapchat’s downloaded traffic per user is below 20 MB/day, while Instagram’s peaked at around 140 MB/day.

8.3.2. Measurement Interpretation From these graphs, we can already understand much about the app’s usage and popularity in Italy. We can see, through Figures 8.2, 8.3, 8.6 and Figure 8.7 how Snapchat became increasingly popular in Italy in just one year. From the amount of downloaded and uploaded traffic, the company experienced a peak in popularity and usage in the summer of 2016, but then it started to decrease. However, the decrease in traffic did not influence greatly the number of users. In fact, in Figure 8.5 we see how daily users decrease after 2016, but not as sharply as traffic. The conclusion is rather simple: users did not abandon drastically the service but became less engaged as they did not upload or download as much content as before. We, therefore, see that users still regularly access the app, but upload and download much less content, as it lost its initial popularity. Having observed these measurements on Italian’s traffic, we proceed to observe Snapchat’s financial indicators.

8.4. Financial Performance

In this section, we proceed to present and analyse the financial data and KPIs retrieved for Snapchat [75]. As mentioned, we represented the financial data from the first two years since they went public, 2017 and 2018. Like in the previous cases, we present the main financial analysis indicators used for multinationals. However, the company also values other indicators, as they present them in their quarterly reports [75]. One of these is, for instance, Daily Active Users (DAU), that the company uses as main usage and popularity indicator for Snapchat. Firstly we introduce the global quarterly revenues of the company in Figure 8.9, where we can see them doubling in just a couple of years, from 150 million $ to almost 300 $. Except for an unexpected bump at the end of 2017, the growth behaviour is rather linear. Strictly related to revenues, the uses Average Revenues Per User (ARPU) as one of its main key performance indicators. In Figure 8.9 we can see how Snap Inc. makes more than one dollar per user each quarter. ARPU presents a similar behaviour to revenues, as it is a function of total revenues. It is noteworthy to mention how, however, the two indicators differ between Q3 2018 and Q4 2018. While total revenues decrease (even just slightly), the total revenues per user keep increasing. 8.4. Financial Performance 77

Revenue & ARPU

300 2.0 280

260 1.8

240 1.6

220

ARPU, $ 1.4 200 Revenue, million $ 1.2 180

160 1.0

31/03/2017 30/06/2017 30/09/2017 31/12/2017 31/03/2018 30/06/2018 30/09/2018 31/12/2018 31/03/2017 30/06/2017 30/09/2017 31/12/2017 31/03/2018 30/06/2018 30/09/2018 31/12/2018

Date Date

Figure 8.9: Snapchat Revenues and Average Revenue Per User [75]. Adapted from Snap Inc., retrieved from https:// investor.snap.com/companyprofile

Date Revenues ARPU million $ $ 31/03/2017 149.64 0.9 30/06/2017 181.67 1.05 30/09/2017 207.94 1.17 31/12/2017 285.69 1.53 31/03/2018 230.67 1.21 30/06/2018 262.26 1.4 30/09/2018 297.7 1.6 31/12/2018 289.82 2.09

Table 8.1: Snapchat Revenues and Average Revenue Per User [75]. Adapted from Snap Inc., retrieved from https: //investor.snap.com/companyprofile

Next, we present Earnings Per Share and Net Income. The two financial indicators are related and in fact, present similar trends. As we can see they are both negative for the whole period considered. A large negative loss is accounted for on Q1 2017, right after the company’s IPO, of around 2 billion $. After that, the net loss remains rather stable and only shows growth in the last quarter of 2018.

NI & Earnings Per Share

250 0.0 500

750 0.5

1000 1.0 1250 EPS, $ NI, million $ 1500 1.5

1750

2000 2.0

2250 31/03/2017 30/06/2017 30/09/2017 31/12/2017 31/03/2018 30/06/2018 30/09/2018 31/12/2018 31/03/2017 30/06/2017 30/09/2017 31/12/2017 31/03/2018 30/06/2018 30/09/2018 31/12/2018

Date Date

Figure 8.10: Snapchat Net Income and EPS [75]. Adapted from Snap Inc., retrieved from https://investor.snap.com/ companyprofile 78 Snapchat

Date NI EPS million $ $ 31/03/2017 2208.84 2.31 30/06/2017 443.09 0.36 30/09/2017 443.16 0.36 31/12/2017 349.98 0.28 31/03/2018 385.79 0.3 30/06/2018 353.31 0.27 30/09/2018 325.15 0.25 31/12/2018 191.67 0.14

Table 8.2: Snapchat Net Income and EPS [75]. Adapted from Snap Inc., retrieved from https://investor.snap.com/ companyprofile

We then present some key financial Ratios in Figures 8.11, 8.12, 8.13: Net Profit Margin, Return on Equity and Return on Assets. Unsurprisingly, being profitability ratios related to Net Income, the trend is quite similar to the one observed in Figure 8.10. All ratios are negative, showing constant negative profitability from the company.

NPM ROA 0

0.1 2

4 0.2

6 0.3 % 8 %

10 0.4

12 0.5 14

31/03/2017 30/06/2017 30/09/2017 31/12/2017 31/03/2018 30/06/2018 30/09/2018 31/12/2018 31/03/2017 30/06/2017 30/09/2017 31/12/2017 31/03/2018 30/06/2018 30/09/2018 31/12/2018

Date Date

Figure 8.11: Snapchat Net Profit Margin [75]. Adapted from Figure 8.12: Snapchat Return on Assets [75]. Adapted from Snap Inc., retrieved from https://investor.snap.com/ Snap Inc., retrieved from https://investor.snap.com/ companyprofile companyprofile

ROE

0.1

0.2

0.3 Date NPM ROA ROE %

0.4 31/03/2017 14.76 % 0.55 % 0.59 % 30/06/2017 2.44 % 0.11 % 0.13 % 0.5 30/09/2017 2.13 % 0.13 % 0.14 % 31/12/2017 1.23 % 0.1 % 0.12 % 0.6 31/03/2018 1.67 % 0.12 % 0.14 % 31/03/2017 30/06/2017 30/09/2017 31/12/2017 31/03/2018 30/06/2018 30/09/2018 31/12/2018 30/06/2018 1.35 % 0.14 % 0.12 % 30/09/2018 1.09 % 0.12 % 0.14 %

Date 31/12/2018 0.66 % 0.07 % 0.08 %

Figure 8.13: Snapchat Return on Equity [75]. Adapted from Table 8.3: Snapchat Financial Ratios [75]. Adapted from Snap Inc., retrieved from https://investor.snap.com/ Snap Inc., retrieved from https://investor.snap.com/ companyprofile companyprofile

We then present in Figure 8.14 the total number of Daily Active Users each quarter, starting in 2016. The company states that it considers DAUs as users who at least open the app once every a determined 24h interval [75]. We can see the total amount of users in the domestic market (U.S.), and the international, split between Europe and the rest of the world. We can clearly notice how the U.S. market is indeed the most populous one for the company. As of Q4 2018, we can see that 1/4 Americans and around 11% of Europeans use the app daily. We can see how the U.S. is already a developed market for Snapchat, as most new users, over this three year period, are coming from international markets. We can also notice how the DAUs stopped increasing after 2017. We can also 8.4. Financial Performance 79

Date Total U.S. Europe Rest millions Q1 2016 122 54 39 29 Q2 2016 143 61 46 36 Q3 2016 153 65 49 39 Q4 2016 158 58 52 39 Q1 2017 166 71 55 40 Q2 2017 173 75 57 42 Q3 2017 178 77 57 44 Q4 2017 187 80 60 47 Q1 2018 191 81 62 48 Q2 2018 188 80 61 47 Q3 2018 186 79 59 47 Q4 2018 186 79 60 47

Table 8.4: Snapchat Daily Active Users [75]. Adapted from Snap Inc., retrieved from https://investor.snap.com/ companyprofile notice a decrease in users after Q1 2018, as the company lost 5 million DAUs worldwide. It would seem that the company saturated its market and actually lost users.

Daily Active Users 200 US 175 Europe Rest 150

125

100 millions 75

0 Q1 2016 Q2 2016 Q3 2016 Q4 2016 Q1 2017 Q2 2017 Q3 2017 Q4 2017 Q1 2018 Q2 2018 Q3 2018 Q4 2018

Quarter

Figure 8.14: Snapchat Daily Active Users [75]. Adapted from Snap Inc., retrieved from https://investor.snap.com/ companyprofile

Lastly, we present Snap Inc.’s stock price; while we did not include this indicator in the two previous cases, we decided to include it in this one. We discussed the problems with this financial indicator in Section 4.2.2 and is used as a complement to the other financial performance indicators presented here. From Figure 8.15 we can see how company’s shares started to fall rapidly after the IPO [129]. The company had no stock splits in the within the two years considered. We can notice how it went from 25 $ after launch to just barely 5 $ in two years. The price fell rapidly and by a large amount in these two years. We can see the end of May 2017 as the moment when the stock exit its stable trend phase and then started steadily decreasing. 80 Snapchat

Snapchat Stock Price

20 $ 15

02/03/201713/04/201726/05/201711/07/201722/08/201704/10/201715/11/201729/12/201713/02/201828/03/201810/05/201822/06/201806/08/201818/09/201830/10/201813/12/2018

Date

Figure 8.15: Snapchat Stock Price [129]. Adapted from Yahoo! Finance, retrieved from https://finance.yahoo.com/ quote/SNAP/history/?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_ sig=AQAAAAdbcL4DvWkOZzdca2nx6a1AmTk4wbrwD7Ik7bPpFZ8HTwquWqXlB4Wm6zsujaA0aycE0RPm4d6zBwlULMnO1KccXJtPlj9joqE3WhWb2dGj1kqo5mBBR2mlaQEgPCsBl33nc8SuSpcXD3nWZ1MZNgoAZpVwxPrJGcF79g77w7K

8.5. Relevant News and Information

With the measurements and financial information at hand, we also proceed to report relevant news and information on the company in the considered time period. Through news and market information we complete the overview of the case study, to then see what implications could be drawn from this case.

8.5.1. Snap Inc. IPO While researching information on the company and news related to its operations, a good part of the retrieved information dealt with its possibility to go public and IPO. Its CEO Evan Spiegel firstly mentions the company’s need to go public in 2015, when the company was valued 15 billion $. The IPO would effectively happen at the beginning of March, for a valuation of around 22.3 billion $ [130]. From a possible starting price between 14 $ and 16 $, the IPO launched with a 17 $ offering price and secured the company 3.4 billion $, and was, as expected, the ”the richest valuation in a U.S. tech IPO since Facebook in 2012 [131]. The IPO created a lot of buzz and excitement among investors and preIPO investors:”Snap’s publicmarket debut won’t be a dreaded down round”[130], and it, in fact, went better than expected in the first days after launch, reaching the second day high of 29.44 $ [132] (as we can notice in Figure 8.15). The positive trend and performance of their stock were however shortlived; the stock value remains stable until the end of May 2017, around 22 $ per share. However, in early summer the price starts to fall steadily. Snap’s stock hit another low in August 2017 as they fail to meet the expected number of users i.e. a lower than expected user growth [133]. When in autumn 2018, Snapchat lost again two million users, their share price falls again to a minimum low of 5 $ in December [134].

8.5.2. Instagram As already anticipated, to understand Snapchat’s case we need to briefly discuss Instagram as well. Instagram is a social media where users can share and discover pictures and videos, message, and follow their friends and other pages; it is owned by Facebook. Even if it started differently, the app has filters and similar features to Snapchat and has evolved into Snapchats’s main competitor. This became even more evident on the 2nd of August 2016, when Instagram launched its own Stories, a clone to Snapchat’s [135]. Even Instagram’s CEO Kevin Systrom gives them the credit, but says it’s implementation format is more for ”adults”[135]. Instagram’s userbase is much larger and grew much more quickly than Snapchat’s, which grew even more after their own release of stories to 500 million users in 2018 [136], [137]. 8.6. Analysis & Results 81

8.6. Analysis & Results

At this point, we have concluded the overview of the company, the measurements and its performance. Snap Inc., awash with venture funding capital, still did not turn out a profit as of 2018 (Figure 8.10): in fact, in all 2017 quarters, the net losses (Figure 8.10) were even much larger than revenues. Its innovative product allowed Snapchat to grow quickly (as the measurements in Figure 8.6 show) and invest in itself. However, they failed to effectively monetize its users on a large scale, as the lagging financial KPIs in section 8.4 show. The company also discusses the 2 billion $ loss reported in Q1 2017, as required financial adjust ments for its IPO launch [75]. After that, losses stabilize around 400 million $. From the data available at hand we now discuss the possible predictions that can be made from this case. We compared the selected KPIs and news with the presented measurements. We need to consider the possibility to correctly predict, at a certain point in time, a change in company performance, with the knowledge provided by the measurements. In particular, we need to keep in mind: • Snapchat’s popularity and traffic start to decline steadily in mid2016 for one year straight. The platform lost its popularity in Italy as quickly as it obtained prominence thanks to its innovative features, in about a year. By the end of 2017, its total traffic was as low as when it entered the Italian market in 2015. • Instagram, another social media, launches the first competitor feature to Snapchat’s Stories in August 2016, exactly when Snapchat starts to lose users. Then, the measurements show a steady increase in Instagram’s traffic throughout the rest of the measurement collection.

8.6.1. Analysis Starting with Revenues, we see that the company is able to increase its revenues through 2017 and 2018, almost hitting 300 million $ in Q3 2018. Similarly, ARPU shows a growing trend. This is clearly a sign the company focused on monetizing its users and generating a profit after going IPO (Figure 8.10), after it was criticized that it was unable to do so. Since we see a decrease in popularity and traffic, we do not see a possible prediction to be made, as the measurements would signal a loss of users instead. Next, we see that Net Income and EPS display a rather negative performance. Despite a meaningful loss in the first quarter of 2017 due to the IPO, these to KPIs recover to rather stable trends. However, these indicators remain negative, showing losses for the company. These KPIs are much closely related to the interpretation of the measurements made in section 8.3. The measurements could signal and predict that these indicators would not change and remain the same, as the company is losing popularity and is now competing with another major social media platform (Instagram). A very similar argument could be made for the profitability ratios NPM, ROE and ROA, as they are a function of Net Income. Then, we see that the company reports increasing Daily Active Users throughout 2016 and 2017. However, on Q2 2018, the firsttimeever loss of 3 million DAUs is announced, and markets noticed [134]. Furthermore, the company announces another loss of users in Q3 2018 and is unable to obtain new users in Q4 2018. Now, considering the measurements at hand, this is a much more insightful indicator to work on. In particular, we need to consider the loss of popularity that Snapchat is experi encing in 2017 and the rising traffic of Instagram, as it seems strongly linked to such a loss in DAUs. For a year and a half, Snapchat lost traffic and popularity, as discussed in section 8.3. Six months after the end of the measurements, Snapchat announces a loss in daily users. This hardly seems a coincides. Supposing to place ourselves in mid2017, Snapchat’s traffic per user is already a third of its peak popularity, and we could safely predict that it would lose users at a certain point in time; this would happen exactly one year after. small skip Lastly, at the beginning of 2017, Snap. Inc. goes IPO. However, the company has been losing traffic and popularity for a year. Then, less than a year after its IPO, its stock value is just half the maximum of 30 $ dollars reached in the first few days. Two years after its IPO, the stock value is but a fourth of its initial price offering. Their IPO and first years of trading did not go well, but could have this been predicted or suggested by the measurements? 82 Snapchat

8.6.2. Results In the previous section, we discussed and analysed the proposed KPIs with the measurements at hand on the company. We see that despite increasing its revenues and monetizing its users, its profits remain negative and the company announces losses for 2 years. Considering these KPIs with the loss of users and popularity, while Instagram’s traffic was constantly growing provided one instance where the measurements predict changes in KPI and where the measurements provide insider information on the company. The first and more obvious implication we notice is that Snapchat became popular and vastly used in Italy for only a couple of years. Its popularity and usage peaked in late spring and summer 2016, as we can see in Figure 8.2 and Figure 8.6. However, popularity was already dwindling in autumn 2016, the same time as Instagram launched its competitor feature. This competition took away Snapchat’s monopoly on Stories and its primacy as a camerabased app, boosting Instagram’s userbase, proof of how impactful a competitor can be on the social media market. By mid2017, there are already enough measurements showing Snapchat’s loss in popularity, predicting the company’s future loss of users. We consider this point in mid2017 as the prediction point for the loss in users, highlighted in Figures 8.16 and 8.17. The prediction point is chosen as it is the point in time where there is more than enough measurements, on both Instagram and Snapchat, to predict such a loss, announced one year later. The point is shown in both Per User and Total Traffic, as both clearly display a longlasting downward trend at that time. By the end of 2017, Snapchat’s Daily user traffic is just a small fraction of Snapchat’s. Six months after the measurements are taken, Snapchat announces its first loss of DAUs. This happens in Q3 2018, a year later after the noticeable decrease in popularity in Italy: there is about a year between the prediction point and the actual loss of users. It is rather clear that this loss in popularity and traffic predicts a loss in total users, while also considering the rising popularity of its competitor, Instagram. The measurements show Snapchat heavily losing traffic to a competitor for at least a year, after which it becomes clear that, at a certain point in time, Snapchat will lose users. In Figure 8.16 we couple together Snapchat’s and Instagram’s daily download traffic per user, showing key timepoints for the case: Snapchat’s peak traffic, Instagram’s launch of their competitor product, and the point in time, at which, it is possible to see from the measurements a future loss in users.

User Download Traffic Traffic Peak DAU Loss Prediction Point 80

40 Snapchat, MB/day

140

120

100

80 Launch of Stories 60 Instagram, MB/day 40

Figure 8.16: Snapchat and Instagram Daily User Download Traffic [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https://smartdata.polito.it/ fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano. 8.6. Analysis & Results 83

Snapchat Total Download Traffic 60 DAU Loss Prediction Point 50

GB/day, total 20

Figure 8.17: Snapchat Total Download Traffic [6]. Adapted from SmartData@PoliTo by D. Giordano, 2019, retrieved from https://smartdata.polito.it/fiveyearsattheedgewatchingintersnapfromtheispsnapwork/ Copyright by D. Giordano.

In Figure 8.18 we clearly show how the decrease in traffic and popularity anticipates to the loss of users. By combining the previous three graphs, we can see how the measurements predict a loss in users. The measurements show that there is enough data to predict a loss in users in mid2017: the company announces such loss one year later, in Q2 2018, as Figure 8.18 shows. We compare Daily Active Users with Snapchat and Instagram traffic, showing how it predicts the loss in users. With this graph and the previous one, we highlight the relationships between the measurements and the actual announcements of DAUs by the company.

Daily Active Users 250 Snapchat Announces DAU Loss

DAU Loss Prediction Point

200 Snapchat Peak Traffic Instagram Traffic Increases

150 millions

100

0 Q1 2016 Q2 2016 Q3 2016 Q4 2016 Q1 2017 Q2 2017 Q3 2017 Q4 2017 Q1 2018 Q2 2018 Q3 2018 Q4 2018

Quarter

Figure 8.18: Snapchat Daily Active Users and Total Traffic [75], [6]. Adapted from Snap Inc., retrieved from https: //investor.snap.com/companyprofile

Next, we also considered the company’s IPO and stock price in 2017. By it’s IPO launch, the total traffic was already half its peak. After this loss in popularity, all through 2017 and 2018, we see the company’s stock value decrease steadily. Indeed, we see a simple and clear connection here. The measurements predict a loss in popularity and the stock keeps decreasing like Snapchat’s traffic. Like in the previous prediction, by mid2017, there are enough measurements suggesting that the stock price would not recover: rather, it is strong evidence from the measurements that it is expected to worsen. Figure 8.19 provides a clear image of the stock, the decreasing Total Traffic, and the prediction point around June 2017. As discussed previously, the stock would have an uncertain behaviour until mid2018, after which it continues falling steadily. This fall in anticipated by the measurements as it coincides with the loss of DAUs. We show how Total Snapchat Traffic decreases, and one year later, with the loss of users, the price falls, as the measurements suggest a year before. 84 Snapchat

30 GB/day20 GB/day Snapchat Stock Price

10 GB/day 5 GB/day Snapchat Announces DAU Loss 25

Prediction Point 20 $ 15

02/03/201713/04/201726/05/201711/07/201722/08/201704/10/201715/11/201729/12/201713/02/201828/03/201810/05/201822/06/201806/08/201818/09/201830/10/201813/12/2018

Date

Figure 8.19: Snapchat Stock Price with Total Snapchat Traffic [129]. Adapted from Ya hoo! Finance, retrieved from https://finance.yahoo.com/quote/SNAP/history/ ?guccounter=1&guce_referrer=aHR0cHM6Ly93d3cuZ29vZ2xlLmNvbS8&guce_referrer_sig= AQAAAAdbcL4DvWkOZzdca2nx6a1AmTk4wbrwD7Ik7bPpFZ8HTwquWqXlB4Wm6zsujaA0aycE0RPm4d6zBwlULMnO1KccXJtPlj9joqE3WhWb2dGj1kqo5mBBR2mlaQEgPCsBl33nc8SuSpcXD3nWZ1MZNgoAZpVwxPrJGcF79g77w7K

However, we remind that stock price is not a performance indicator, and we decided it not to consider it as such, because it can be influenced by many parameters. However, we included it in the analysis as it is a very clear example of how the measurement can provide very important information on a company. This is clearly insider information, as we see that the stock price decreases along with the total traffic in Italy. We do not claim, per se, that the measurements can predict the devaluation of Snap’s stock; however, we strongly point out that it should be a parameter to look at, and that it can be extremely insightful for the savvy investor.

8.6.3. Snapchat Timeline Like in previous cases, in Figure 6.11, we constructed a timeline of the measurements, the events and the results obtained from this case. We divided the timeline per year, month, and relevant quarterly report reported as QX in the Figure. Furthermore, we represent in cyan the relevant measurement period and collection. Through this timeline, we show how the timings of the event, the IPO, and the measurements interact with each other on a time scale. Lastly, we point out where the measurement predicts a loss in DAUs in red. Like in the previous timelines, we highlight in cyan the points in time when the measurement can make predictions on the company’s KPI. This helps us visualize the various events and information discussed in the previous sections. For instance, between 2017 and 2018, it is clearly possible to see how a traffic minimum anticipates by 6 months the loss in DAU. 8.7. Conclusions 85

Snapchat Timeline

Q2 Relevant Total and per User Measurement Download and Upload Q3 Period Trafﬁcs Increase Q4

2015

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

Snapchat's Trafﬁc Start to Decrease

Total, per User Download Instagram Trafﬁc Start and Upload Trafﬁcs, to Increase Users and Flows Peak Exponentially

2016

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

Instagram Releases Its Stories Feature

Snapchat's Total Download Trafﬁc Minimum

2017

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

Prrediction Timepoint Loss of Users and Evidence of Negative Snap Inc. Goes IPO, Stock Price With a 29 $/per Share Peak in the First Days

2018

Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan

Snapchat Loses 2 Snapchat Loses 3 Stock Price at 5 $ Million DAU Million DAU

Figure 8.20: Snapchat Timeline.

8.7. Conclusions

While only studying the measurements from a single country, we see how strongly related they are to Snap Inc.’s global performance and the usage of their application. The company’s performance has been identified with traditional financial KPIs, as for most corporations. However, these were not specifically predicted by the measurements, while Daily Active Users were. DAUs is a KPI that the company uses internally to monitor how well their social media platform is doing and is a rather specific KPI that is highly valued internally and externally. In this case, as well, a sudden shift in the measurements predicts a change in one of Snapchat’s main KPI: Daily Active Users. The difference with respect to the other two cases is that this prediction 86 Snapchat comes from Snapchat’s traffic itself and its competitor, Instagram. We group here the main findings and prediction found during the study of this case.

• The measurements show in detail user behaviour in Italy and Snapchat’s increase and decrease in popularity and usage. The measurments in Figures 8.2, 8.4, 8.4, 8.6 and 8.7 show how populare the service became in just one year, from early 2015 to early 2016. However, by summer 2016 there is a shift in this upward trend and traffic on the platform decreases steadily until the end of the measurements in late 2017. Furthermore, we have seen that this decrease in popularity coincides with Instagram’s release of a competitive feature, their own Stories. In Figure 8.16 we see how Instagram’s Daily User Download Traffic starts to increase at the exact same time as Snapchat’s start to decrease. Both measurements predict the loss in Daily Users that Snapchat announces in July 2018 during its second quarter, as we can see in Figure 8.14,[75], with a one year advance. We can see, for instance, that by the end of 2017, when the measurement end, there is a clear picture of Snapchat’s and Instagram’s popularity in Italy. While these corporations operate at a global level, the measurements clearly show a loss in popularity and the loss in users announced six months later in 2018. • While the app is experiencing a sharp decrease in popularity and usage, the company launches its IPO, which turns unprofitable after only 5 months [130]. Then, the share price keeps decreasing until it reaches onefifth of its peak value after only two years (Figure 8.15). While we decided not to focus on the stock as a means of evaluating performance, it is however interesting to mention this finding. We can see that the measurements display negative trends in Snapchat’s popularity at the end of the measurements in September 2017. One year later, the stock price is only half its maximum value (Figure 8.15,[129]). While, again, it does not predict directly one of Snapchat’s KPI, the measurements provide meaningful insider information on the company that can be used as a market edge.

In this case, that network measurements predict changes in a specific KPI that Snapchat values greatly, its Daily Active Users. We can see that this change in performance is predicted by both Snapchat’s own measurements and by one of its competitors, Instagram. In both instances, the pre diction point is in mid2017, as it is the moment when there is more than enough data from the measurements to understand Snapchat’s future success and performance. This decrease in perfor mance would then be announced in their reports one year later, wit a loss of users and the fall of its stock prices For Snapchat too, the measurements predict company performance. Furthermore, even if not included in our definition of company, the measurement provides insights on the company stock, and show that the measurements should be considered while studying and predicting the company’s stock. 9 Discussion

The literature review and three case studies analyzed to provide us with rich information and implica tions for the goal of this research. As discussed in section 2.2.4, after the study cases the research needs to group together the findings and discuss them [4]. In this section, we group together and discuss the findings from the literature and the three case studies, not individually but together. [4] refers to this section as the crosscase synthesis to be performed at the end of the case studies. The crossanalysis and study aim at generalizing and putting together the findings from the case. However, we have to remember that ”the ultimate generalizations not likely to achieve the status of “proof” in geometry but the claims must be presented soundly and resist logical challenge”[4], p. 19. The various study cases are just a way to create and provide interesting, novel and valid information that has to be studied together, with a broader perspective on the research as a whole. It is the first step toward answering the research questions proposed in Chapter 1 to bring the research to closure. Three cases are not enough data for a general theory building process and statistical analysis or proof. However, they can provide a clearer overview of a topic and insightful insights on it. This can prompt, however, further research on the subject, as to serve as a starting point for it. Here we group and study the results together, discuss their relevance and the social implications that they might have.

9.1. Summary of Results

A brief summary of the main findings for each case is here provided, to then proceed with a unified discussion of the obtained results. We present the results obtained from the literature review from Chapter 3 through 5, to then combine it with the realworld results and experiences from the three cases from Chapter 6 to 8.

9.1.1. Literature Review Results In the first chapters, we provide a general overview of network measurements, what they are, and how they are used in relations to companies. We have seen that network measurements are an effecting way of collecting analysing data collected from the networks of computers that make up the internet. Various metrics and parameters can be measured, like traffic, latency, bandwidth or packet loss rate. These metrics can be collected with active, passive or hybrid techniques. Researchers usually use these measurements to study speed and topology of the various elements in the network, in order to detect problems and improve performance. However, these measurements are not regularly used in relation to companies and their operations on the Internet. There are particular attentions to how companies operate through CDNs, which shows the prominence that CDNs are obtaining in today’s Internet. Few other measurements are however used to study companies. This gives our research the opportunity to explore this undeveloped topic and to provide the literature with new findings. Next, we studied, from a theoretical perspective, what is company performance and how it is un derstood in the academic and business world. The literature is consistent in defining and evaluating

87 88 Discussion corporate performance as a composite and heterogenous assessment of various factors. These factors are both qualitative (such as goodwill or corporate social responsibility) and quantitative, i.e. mathe matically measurable: we defined the latter as KPIs. There are various types of KPIs, that we classified as Financial, Webrelated or General, according to the category they belonged to.

9.1.2. Case Results In the first case study (Chapter 6) we relate IPv6 addresses to their hosts and monitor their number over a few months. For a cloudbased company like Salesforce, hosts reflect a good part of its routers and servers. The case shows how an increase of hosts (in two rounds of 10k each) anticipates what we can consider as positive company performance. Shortly after the deployment of hosts the company announces the acquisition of few major clients (Citigroup, PNC Bank, Telcom Argentina and others) and strategic alliances with key players in the cloud and Web service industry (Google, AWS, Cisco and others). The increase in hosts predicts an increase in market demand from which Salesforce is able to profit from. The measurements predict and anticipate beyondforecast revenues and the acquisition of major clients from Salesforce. In this case, the measurement predicts a positive improvement in company performance. The second case study on Netflix (Chapter 7) studies the company’s traffic and users in Italy, providing great insights on user behaviour. The steady and increase in traffic flows and users show how more people are using the service and that users are usually committed to the platform on a daily basis. The measurements predict the company’s success in Italy and internationally, considering the Italian market as a more difficult one to breach. In particular, the measurement predicts the increase in subscribers and revenues internationally, until the international segment becomes larger and more profitable than the domestic one. Furthermore, the measurements show that total traffic increases when original is released: there are traffic spikes right after the release of the original teen drama 13 Reasons Why and the Italian mafia drama Suburra: Blood on Rome suggests. The measurements show the success of these series before the information is made public by Netflix’s quarterly reports. In this case, the measurements predict an increase in subscribers and make available information on the company and its service before its released. Finally, we monitor Snapchat’s traffic and users in Italy in the last study case (Chapter 8) and see from the measurements how impactful competition in the social media market can be. We see how a decrease in popularity has serious implications for the parent company Snap Inc. in many ways. While the company loses popularity and traffic decreases in Italy, Snap Inc.’s IPO has a shortlived success, as its share price keeps decreasing steadily for two years. The company later also loses total users and keeps announcing losses instead of profits. The measurements also show that when Instagram launches an incredibly successful competitor feature on their platform, Snapchat’s popularity and usage starts to decrease. This decrease in traffic and popularity, predicts Snapchat’s firstever loss of Daily Active Users, for two quarters in a row. The loss in DAUs is extremely significant, as it is one of the main KPIs used by the social media platform. In this case, the measurements, despite representing only Italian traffic, show how related traffic and usage are to global performance. The measurements show user behaviours that predict a loss in stock value and DAUs by providing insider information on the company.

9.2. CrossCase Synthesis

In this section we discuss the abovementioned results as a whole, combining the literature review with the knowledge obtained from the cases on corporate performance and measurements. Furthermore, we discuss the results on company performance prediction obtained from the three cases, looking at them homogenously and not as three separate entities. [4] defines this process as a CrossCase Synthesis, necessary to bring forward the conclusions from our cases.

9.2.1. Corporate Performance & Measurements The literature review on corporate performance was used to develop the cases and select appropriate KPIs for each company. The real experience, form each of the cases, agrees and confirms the findings 9.2. CrossCase Synthesis 89 in Chapter 4. In each case, we considered a standard set of Financial KPIs (we had three large corporations in all three cases). However, through the company quarterly reports and the financial news around it, we identified other relevant KPIs and factors that affect that company. Each company had a particular factor or KPI that resulted essential to benchmark their performance. For instance, Netflix used subscribers while Snapchat used Daily Active Users. These are all measurable KPIs, but the cases showed that the companies relied as well on other factors to evaluate their own performance. For Salesforce it was the acquisition of new clients, while for Netflix it was the release and appreciation of original content from the users. Overall, however, the practical insights on company performance gained from the cases support the theoretical definition of company performance. Performance is evaluated for each company differently, using different and heterogeneous sets of metrics that can be used for that company or its market only, as that is supported throughout the three cases. The cases also showed us that the selection method for measurements sets proved effective, as all three measurements predicted changes in KPIs and performance. In particular, all the considered measurements were collected for purposes different than the one in this thesis. However, they turned out to be predictive in some way of company performance and helped us build the cases. A mea surement set should always be stored, as it might prove useful in other contexts, and this is exactly what happened. Monitoring the number of IP addresses that a company allocates or the total traffic of a certain service passing through an IXP proved to be insightful measurements set for the purposes of this thesis. We showed that network measurements can be used for other purposes other than performance evaluation and topology, that it is a topic worth investigating and that it requires further research.

9.2.2. Performance Prediction & KPIs The companies studied and the types of measurements used are different and belong to welldefined and distinct markets. This case heterogeneity contributes to the research with unique findings and implications set in different contexts and markets. The measurements reveal information on user be haviour, service usage or increase in demands. Following our analysis, it is possible to relate network measurements to prominent KPIs in each case. Salesforce increase of hosts anticipates an increase in demand, acquisition of clients and unexpected increase in revenues. The second case shows how positively original content is received and that increase in Italian traffic forecasts an increase in inter national subscribers and revenues. Lastly, the decrease in popularity and usage anticipates losses of DAUs and losses on the stock markets. Furthermore, the measurements also show how an exponential increase in traffic of a competitor service could influence Snap Inc.’s performance. In general, the measurements have also helped us better understand KPIs for each company ac cording to their business model. We have seen across the three cases how financial KPIs are somewhat related, in different ways as well, to the measurements. KPIs, however, is not the only instance of performance related to measurements: quite often, the measurements reveal information on the com pany before the general public has access to it. For instance, the number of new clients for Salesforce or the quantity of original content for Netflix are strongly connected to the company performance and predicted by measurements. In all three cases, a sudden change in the measurement or a stable trend anticipates a change in KPI or relevant marketrelated news for the company. From this perspec tive, we have seen how measurements can provide insightful information on a company, its KPIs and other factors influencing its performance. However, in this research, we do not specify when there are enough measurements to make a prediction. In the three cases, a prediction is made and discovered once there is more than enough past measurements to safely predict KPIs or acquire information on a company. Predictions are made when something is evident from the measurements when a longterm trend is observed or a small set of outliers is observed. This highlights the need for further research and larger datasets, in order the develop a framework to establish when something can be predicted. In our analysis and cases, we selected various KPIs and indicators that could be predicted by network measurements. However, the measurements clearly predicted changes in only certain cases: subscribers, revenues, beyond forecast revenues and daily active users. These KPIs can be grouped into the direct revenues and the number of users categories. Meanwhile, other KPIs did not found a direct correlation with the measurements: these were net income, operating income, profitability ratios and earnings per share. The question that now arises is why certain KPIs were 90 Discussion predicted and others were not. It could be argued that users and revenues are metrics that interact “directly” with the outside world, the company has no control over it. These KPIs are directly related to the company’s users and how users interact with the offered services. Since the measurements considered also take into account user usage and user behaviours, we can see that there could be a more direct relationship with these external KPIs, hence a prediction is possible. On the other hand, profitability and earnings KPIs are handled more internally, they pass through the company’s “filter” and therefore are less related to the outside measurements; these indicators can be considered more internal. This is only an initial analysis on the issue, as a larger set of measurements should be considered. the three cases in this research can only kickstart a discussion on the subject, but it is worth bringing further to create a deeper understanding of the subject. Despite the rather limited amount of dataset available, we have seen that network measurement can be used in a market environment and in which way. The research shows for the first time that mea surements can be used for purposes other than traffic monitoring, topology or network performance. In particular, they were able to predict unexpected results in revenues, an increase in subscribers and loss of users. The cases also show that measurements can provide important information on the com pany and that this insider information can be used as a market edge. There is a clear relationship between network measurements and company performance, and we have proved its existence. We have indeed observed a technology in a novel setting with a neverusedbefore purpose: company performance prediction. It is up to further research to investigate further the details of the relationship between measurements and performance and establish how it can be effectively used as a corporate resource. The novel results of this research aspire to be the first step toward a much larger scale study.

9.3. Social Implications

The research and its findings have clear and welldefined social implications, both in general and for specific categories. While we do not develop specific policy implications as [4] suggests (section 2.2.2), it is possible to discuss the social implication of this type of research. This is only to kickstart a discussion on the topic and to implement it with further and more detailed research. When dealing with networks, there are many stakeholders to consider: ISPs, regulatory entities, companies, clients and consumers. Our research also involves a wider category of stakeholders, less networkrelated but at the potentially the ones that could benefit the most from further studies like the one proposed here: investors, managers and market analysts. While the measurements do not substitute sound and proven traditional indicators, the research shows how analysts could use them as a valid complement. In this section, we describe what are the main implications and lessons learned from the cases for each stakeholder category.

9.3.1. Considerations for Stakeholders

Some of the issues on networks that all stakeholders share our privacy and security. Network providers and companies are indeed thoughtful on security and privacy and might indeed have concerns regarding network measurements and their usage. It is therefore natural that such type of studies and measure ments are limited to academic purposed and not market ones. Furthermore, a company might not be willing to independently share and have its traffic directly monitored. As we have seen in Chapter 1 and 4, most measurements are taken from vantage points outside of the company’s network, where it has no control of its own data. And even if measurements were to be taken outside of a company’s reach, they still might not want its traffic to be known or collected. Encrypting or blocking access to such traffic might be a viable solution to such a problem. Companies and providers want to protect and control the information regarding their own activities, as competitors might indeed take advantage of it, even though obscuring its own presence online is not always possible. For instance, [5] collected the number of IPv6 hosts that Salesforce was using. This data was collected by the researchers and is not information that the company did not release. While the researcher did not ask for Salesforce’s permission to collect it, the Internet is free and open, and these IP addresses are publicly available on DNSs lists. The measurements were collected for research purposes and were anonymized when 9.3. Social Implications 91 published, but the company still did not give consent for their collection. This is definitely a grey area that needs more study and discussion. In fact, the Internet is open and free, and a lot of information and traffic is publicly available for those able to collect it. Companyspecif traffic exists, and a company might not agree to have its own measurements collected even if they are open. The research and network measurements show how there is a need for regulations policies on this subject and that it needs to be discussed further. Similarly, individuals and users might not want their traffic to be collected, for privacy issues. This is why, as we have seen in Netflix’s and Snapchat’s case, the user traffic has been anonymized [54]. Unfortunately, it is almost impossible for users to know is someone is monitoring their traffic or where exactly their traffic is going through. VPNs and browsers like Tor can protect user’s data and privacy, but during largescale network measurements, these users are a minority. In fact, most users do not use VPNs to protect their data, as they might just be unaware of it or not care. While beyond the scope of this research, Internet privacy, industrial espionage and online privacy are topics to be considered when applying network measurements to company performance. Our research shows how network measurements are valid and reliable input when analysing and understanding the future of tech companies. We have seen how measurement can provide insightful and valid complementary data to financial and market analysis routines. It would provide investors and analysts with advantageous and insightful market intelligence, that could possibly improve their decisions and models. Network measurements are in fact not generally used in company analysis, as the most stick to financial and economic information. But Chapter 8 shows instead how network measurements can be used by investors and market analysists. The measurements [54] predict a loss in popularity followed by a loss in users: this is invaluable information for an analyst. Similarly, monitoring measurements and having this knowledge at hand could have been crucial for investors, considering Snap Inc.’s poor IPO performance. The research suggests that investors and analysts should constantly monitor network measurements and metrics providing insider information on usage, users and popularity, even if not all companies might agree to it. Predicting information and KPIs on a company, or knowing it before others, can be fundamental and outcomedefining for those categories.

9.3.2. Considerations for Managers

The cases provide implications for managers on networks and performance of various kinds. Measure ment monitoring can be a powerful tool to analyse and benchmark the status of their business and possibly predict their future performance. Measurements can provide managers with key and insider information should consider when making decisions. In particular, managers need to give particular attention to sudden shifts in the measurements during the monitoring process. Long term trends can be seen and need to be considered: Figure 8.6 shows Snapchat’s decreasing popularity across users over a year, resulting in a loss of total users. This should prompt managers to act quickly to reverse such trends and avoid larger problems in the longterm. Similarly, measurements can also provide managers with shortterm information, that can help managers assess and evaluate the correctness of their decisions. One such case is observable in Figure 7.16, where the measurements show the sharp increase in traffic after the release of original content in original content in Italy (the mafia thriller Suburra: Blood on Rome). This is a positive reinforcement cycle, showing that the company analysed and read correctly its audience, and delivered a product that was highly appreciated. Measurements can, therefore, help managers making and evaluating their operating decisions both in the long term and in the short term. Lastly, Snapchat’s case clearly shows how managers can use measurement monitoring to obtain a market edge and advantage of competitor services. This way, managers can better understand some competitor’s strengths and weaknesses, or their own. It is an opportunity to improve the company’s procedures and processes. However, other companies could obscure or encrypt their traffic not to have it monitored by its competitors and being observed, in order not to reveal anything about its internal affairs. Figure 8.16 shows how two competing services interact and how their traffic changes after a meaningful event. For chapter 8, the meaningful event is the launch of a competitor Stories feature from Instagram. The measurement immediately shows the change in popularity of the two apps and how the users reacted to this new feature. The measurements carry invaluable information 92 Discussion for managers, in order to quickly reassess their position and the size of the threat their company is facing. This case clearly shows how managers can use network measurements to monitor competitors, learn from them and react to their decisions. 10 Conclusions & Recomendations

The last chapter brings closure to the thesis and the research project. It discusses whether valid answers to the research questions were found and if the research fulfilled its objective. The research objective was to verify whether network measurements can predict company performance and provide information before it is made public. The exploratory goal of the thesis is to prove or deny, the existence of such a relationship and discussing whether it should be further investigated. Throughout the cases, we highlight various relationships and find novel and interesting information, justifying the need for further research on the subject. We are satisfied with the results obtained so far, considering the various limitations of the project, such as optingout the independent collection of measurements. The two main limitations were time and measurement availability. While scanning for usable network measurements, we found very few available and usable measurements, as discussed in Chapter 5. The other option was to perform some measurements ourselves to implement in the research: one possibility was to use the RIPE Atlas probe infrastructure for active measurements [85]. At the same time, however, there was not enough time to either spend months looking for datasets or to collect them ourselves to use in research. Performing active measurements ourselves might have taken too long for the time frame of the thesis, and it would have been impossible to have a future perspective against which validate our findings, as measurements and performance would have been contemporary. Lastly, for a largerscale study, the measurements might have taken a couple of years to become relevant as a dataset. Therefore, we used past and already available measurements. The measurements used were, in their own way, limited as well, as they some dealt with a single company or traffic only within a specific country. These were the limitations that were considered during the execution of the project.

10.1. Research Questions

To guide us during the project, we developed one main research question and three subquestions. The thesis first answered the subquestions, in order to build and provide an answer to the main problem and issue of the research. The main research question was stated as follows:

Main RQ

Do network measurements predict company performance?

Before answering the main research question, we proceed to answer the three subquestions, as they are used as guidelines towards an answer to the thesis main objective.

10.1.1. Answers to SubQuestions The first subquestion deals with the identification and formalization of the concept of company per formance. It was formulated as follows:

93 94 Conclusions

SQ1

What is company performance?

This short and straightforward question was however not trivial to answer. We found the answer to this question through literature research in Chapter 4 and through the analysis of the cases. In fact, we realized that an answer from the literature needed to be complemented with the realworld experiences developed in the cases, where we observed and analysed what companies consider as company performance. As seen in Chapter 4, providing a oneline definition of company performance is close to impossible and, in any case, not very insightful. It is a quite general definition that needs to consider various factors and elements, without a single formula or a unique multipurpose framework applicable to every company. The reasons are simple: different companies have different products, operate in different environments, have different images and are managed differently. It would be therefore difficult to provide a single definition of performance that works for every company. Hence, company performance is a composite assessment of various metrics, factors and constructs that is not the same for every business. In fact, a company’s performance needs to be evaluated with respect to its own goals and parameters, that needs to be interpreted within the context of the selected company or market section. We defined and identified these factors as Key Performance Indicators, or KPIs, and classified them as Financial, Webrelated or from other categories, identified as General KPIs. These metrics are mathematically quantifiable are of very different nature and can have different weights for different companies. These indicators are defined as a “benchmarking system to effectively assess and evaluate a certain feature or metric”, but, per se, are not enough to define what “textitcompany performance” is. Researchers and market analysts also have to consider which corporation or market it is looking at, as each has its own characteristics. This became clear to us while conducting the cases and choosing the KPIs for Salesforce, Netflix and Snapchat, as each belongs to a different market and benchmarks its performance in a different way. While conducting the cases, studying companies in a realworld setting, we understood practically, from their documents and the related news, how they value their performance and how they choose the KPIs that best suits them. Companies within the same or business model market might share and value a certain factor or indicator the same way; however, it is as likely that have two very distinct parameters for performance that they value while the other really does not. For instance, both Salesforce and Netflix operate with a subscriptionbased model, earning a fee for their services each month, earning revenues the same way. But while they similarly value a parameter such as revenues, for a company like Salesforce the acquisition of a client is much more meaningful than the acquisition of a single subscriber for Netflix: clearly, this is due to their B2B and B2C models, respectively. Similarly, a company such as Netflix cares greatly about the videocontent it produces and makes available, a performance parameter which carries no meaning for Salesforce. In conclusion, each company has more than one way to evaluate its performance and uses various and different benchmarks to see how well it is doing. The answer to this subquestion, therefore, lies in identifying company performance as a compos ite analysis and evaluation of various KPIs, and knowing that each company uses different metrics. Therefore, to understand company performance it was necessary to study the various factors used to evaluate it, as they function as proxies for performance. These indicators can be related to finance, market analysis or Web analytics, and is not uniform for each company. This composite view on com pany performance helped us in building our case studies and selecting relevant indicators to study their link with networks. Ultimately, company performance is extremely heterogeneous concept and each corporation has its own definition of it.

• SQ1 Answer: company performance is a heterogeneous composition of qualitative and quan titative metrics and factors. Each company benchmarks its performance with a different set of performance indicators, according to their market, product and business model.

The second subquestion tackles a more technical aspect of the research, network measurement. It is a necessary step to provide the research with the proper technical background on the topic to properly select and conduct the cases. We formulated the second sub question as such: 10.1. Research Questions 95

SQ2

What are suitable network measurement sets and techniques to predict companies’ performance?

To answer it, we thoroughly defined and studied network measurements, how they are performed, for which purposes and how they are used. This background on network measurements was crucial for the advancement and validity of the research Through Chapters 1, 3 and 3.6, we analysed the current literature on network measurements, their relationship to companies and their purposes. Mea surements are often used for performance and topology purposes and are related to companies to understand their technical use of network elements such as CDNs and their hierarchy [19], [24], [27]. Next, we defined network measurements as “the collection, interpretation and modelling of empirical Internet data describing finite metrics and relationships between network nodes and elements”. This thorough definition of network measurements helped us define the measurement to be scanned se lected and used for our cases. The answer to this research question lies in the type of measurement selected in our case studies as well as in some of the discarded measurement sets in Chapter 5. For this research, we identified a few types of network measurements and techniques to study and anticipate company performance. Both passive and active techniques proved effective. In Chapter 6 the Salesforce data used used was collected by active techniques [5], while in Chapters 7 and 8, Netflix’s and Snapchat’s traffic were collected by passive probes [6]. We also defined, in Chapter 2, three characteristics for selecting our cases, two of which impose requirements on the type of network measurement dataset used. In particular, we require the measurements to e collected and measured on a computer network, while being connected to a company and historical, in order to verify whether they could predict changes in company performance. According to these characteristics we selected certain measurements that were later used in the cases. According to the selected measurements and the literature review made in 3.6, we identified suit able types of measurements and techniques to use in relation to companies and possibly predict their performance. From the cases, we have seen that bot active and passive techniques are used. Fur thermore, the measurements selected for the cases collect the number of Salesforce hosts thanks to their IP addresses, while the traffic in Turin is quantified and identified via IP addresses and the DNS system [5], [6]. Other literature on the matter of studying how Netflix and YouTube use structure their CDNs systems, base their methodology on bandwidth and the DNS infrastructure [55], [19]. All these cases, the measurements are based on IP addresses and the DNS system, while various metrics are measured. In our three cases, the measurement counts the number of IP addresses, users TCP flows and the quantity of data exchanged between users and the service provider, i.e. upload and download traffic relative to a company. From the conducted research, these seem the most suitable techniques and measurement sets to predict company performance.

• SQ2 Answer: a suitable set of measurements has to be related to the company and its pres ence online. The set has to represent characteristics and metrics collected via active or passive techniques. The research identified traffic quantities and IP based metrics, such as the number of addresses, users and TCP flows as predictive of company performance.

The last subquestion combines the previous two subquestions with the main goal of the thesis. It is answered throughout the cases according to the methodology defined in Chapter 2. This question connects KPIs and network measurements, and is formulated as follows:

SQ3

Which KPIs are predicted by network measurements?

The answer to this question is in the Analysis and Results section, with each case contributing in some way. We discuss why certain measurements do not predict changes in certain KPIs while they do for others. In particular, across the three cases, we do not highlight a particular predictive relationship for KPIs such as EPS, NI or OI. Instead, we see for the Netflix and Salesforce cases that the measurement predict an increase in Revenues while and after the measurements are taken. For the former, this is strictly related to the prediction of increased subscribers, while for the latter comes as a positive breach of their own quarterly forecasts. Furthermore, the traffic measurements predict 96 Conclusions a loss in Snapchat users, that comes a year after a constant decrease in popularity. Strictly speaking, these are the only KPIs that the measurements considered predicted, but they also provide further insider information on the companies before its released. For Salesforce, the measurements anticipate an increase in demand that is confirmed, after a couple of months, by the acquisition of major clients. Similarly, the measurements show which Netflix shows were mostly appreciated by the public before it was known. This also suggests and predicts an increase in Netflix original content in English and local languages, already verified in 2018 (Figure 7.13,[124]).

• SQ3 Answer: The measurements predict changes in various KPIs, most notably subscribers, revenues and beyondforecastrevenues and daily active users.

10.1.2. Answer to the Main Research Question To answer fully answer the main research question we bring together the results from the subquestions and the crosscase synthesis made in section 9.2. By answering the first three subquestions we were able to identify corporate performance, suitable network measurements and building cases that allowed us to verify whether performance prediction was possible or not. Ultimately, the research answers the main research question for each case and provide a more comprehensive overview of measurements and companies (Section 2.2.4 and 9.1). In all three cases, we find instances of the measurements predicting KPIs. These predictions, however, are not limited to quantitative metrics such as KPIs. They predict more qualitative aspects of company performance, providing insightful information on the company before it is made public. We refer, in particular, to Salesforce acquisition of clients and Netflix’s success of series like Suburra:Blood on Rome and 13 Reasons Why before the announcement in their quarterly reports. The measurements predicted both changes in KPIs and provided crucial information on a company before it was made available. In particular, we highlighted a strong correlation between the measurements and users and revenuerelated KPIs. These predictions are part of company performance form a quantitative and qualitative perspective, adhering to our composite and heterogeneous definition of performance. While these are interesting results, we need to make some remarks on the context in which they were made. Different types of measurements for different types of companies were considered, with specific limitations: for instance, as in Chapter 7 and 8 we only consider Italian traffic. We, unfortu nately, were not able to collect adhoc measurements for the research, due to timing constraints: we use other measurements that were initially intended for other types of analysis. We limit our findings to three cases, measurements sets and companies, each with distinct results. This surely limited the possible outcomes and findings of the research but also begs the question of whether larger mea surements could have brought more prediction. As only three cases were able to show the predictive ability of network measurements and their relation to company performance, the conclusion is that this subject needs to be explored further. Our explorative approach ultimately provided a final answer to the research that should encourage further research.

• Main RQ Answer: Network measurements predict company performance in its qualitative and quantitative meaning. Furthermore, they provide performancerelated information on a company before it is made public or announced.

10.1.3. Theoretical Cotribution The research questions proposed in Chapter 1 were used as initial guidelines to build and bring forward our research. Furthermore, the explorative nature of our research brought us other findings as well. Various relevant implication and discoveries made in the cases are not directly related to the main question but contribute to a better understanding of the topic. The research contributes to the literature in various ways. Firstly and foremost, it deals and tackles a sofar overlooked subject of network research in relation to companies, trying to bring the two of them closer together. The research describes new interactions and relationships between network measurements and corporate performance, showing how one can predict the other. Furthermore, 10.2. Suggestions For Future Research 97 while monitoring network measurements it is possible to obtain insider information on a company and knowledge on its business and operations in advance, before it is made available to the public. The research also uses network measurements in a quite distinctive way from most technical research, i.e. as a tool for obtaining insider information that can provide a market edge to the company. Lastly, we introduce social implications and considerations on the use of measurements, for many stakeholders, with a particular focus on company managers. The Salesforce case in Chapter 6 shows how the mea surements could be used to understand a B2B company and anticipate the expansion of its operations and acquisition of clients. The cases in Chapters 7 and 8 show how network traffic can be used effec tively to predict not only company performance but user behaviour. For B2C services like Netflix and Snapchat, understanding user behaviour and platform usage its crucial to evaluate the state of their business, operations and predict its future. The conducted research shows that it is possible to use network measurements in a different and novel and meaningful way in other settings as well while achieving the main goal of the thesis. Since there is no relevant literature on the matter, we expect this thesis to be the first step towards larger scale research. In particular, the results from this project want to encourage researchers to collect more measurements and see to what extent their predictive ability is. We contribute to the scientific literature by proving that this idea is worth exploring and should be brought forward.

10.2. Suggestions For Future Research

The conducted research highlighted the need and relevance of a much larger and deeper analysis of the topic, as it worth investigating. It could strengthen and highlight undiscovered connections between computer networks and companies operating on it. However, a stricter and statistical analysis of network measurements and their predictive ability needs to be conducted: key is the availability of larger datasets. As more and more companies digitalize and bring their business online, the relevance of this kind of study grows, and our research points out that it is worth investigating further this subject. Other suggestions for future research were found while conducting the thesis.

10.2.1. Social Research Insights

Throughout the cases and the literature analysis, the thesis highlighted the lack of use of network measurements in social sciences. While predicting company performance through the measurements, various instances highlighted the need for a deeper understanding of the social context those net work measurements in today’s interconnected and rapidly changing markets. Firstly, a modern Socio Technical Value Map on network measurements could be developed; it can help better define the context of network measurements in contexts outside academia. In particular, it could help better define who and how could benefit using network measurements as a tool for predicting performance: after, of course, statistical validity is found. Secondly, the research shows that only certain more “external” KPIs and information were predicted by the measurements, while other, more “internal”, were not. Therefore, there is a research opportunity in developing a specific framework to establish which KPIs can be predicted by the measurements, for which company or in which markets. This could provide insights not only on the company itself but the whole industry. In fact, this still results in one of the grey areas of the research. We found out that measurements can predict certain KPIs, but they provide little insight into why others are not. Lastly, with larger measurements and statistically proven correlation, the measurements require to be included in a decisionmaking framework. In particular, stakeholders like investors and managers need a tool to base their decision while considering the requirements. In particular, managers could change their decisions or make different evaluation according to the possible predictions that certain measurements could make. The measurements should not be the focus of a decisionmaking process in companies or financial markets but should be given the proper weight and relevance, while being included in the framework. 98 Conclusions

10.2.2. Other Datasets & Measurements While scanning for measurements, building the case studies and researching the current literature on the subject, we discovered other possibly interesting datasets that might be used for future research. As noticed, one of the main highlighted issues was the lack of extensive measurement datasets related to network players. We here decided to briefly revisit the considered datasets and to suggest them as possible complement or inspiration for future research on networks and companies. First and foremost the SmartData@PoliTo [54] group has and allows access to measurements (like in Chapters 7 and 8) on other services. To mention some, the group has traffic on services like YouTube, Facebook, Whatsapp, Twitter and many others; these traffic measurements could be used for a deeper study on the Italian social media market or in general businesses operating through the Internet. Similarly, through their traffic analyser Tstat and the sponsoring project [2], Internet traces on IP TV provider Fastweb or Skype are available to be studied. Another quite interesting data repository is the OpenINTEL [86] project, where TLDs could be further studied and analysed over a period of time, and see if the measurements contain relevant trends. Unfortunately, the largest open repository we found, maintained by CAIDA [83] does not seem to have relevant measurements, but a deeper and more thorough analysis is suggested. As we have seen, the possibilities open repositories analysed are limited and have not been pur posely taken for the topic analysed in this thesis. The solution, would, therefore, be to actively produce and study measurements for a larger study on the subject. There is some open resource online, like the RIPE infrastructure of probes [85] all over the world. It could be possible to develop a study using these active measurements and probes, specifically targeting the main network players. In general, larger and worldwide studies on companies could be collected. Web players, such as social media providers and CDNs seem the most likely candidates for this kind of research. The possible targets for a study are the company’s traffic quantity, TLD and IP addresses. Specific companies could be targeted and studied for a period of time, in order to produce a meaningful amount of data for a further study.

10.2.3. Further Research The project shows how network measurements can be used to understand and study company per formance. However, we have not achieved any statistical proof but the need for further research. We have here suggested other already available datasets to use for more extensive and thorough research on the topic. It is, however, not enough. Throughout our cases and the discussion, we highlighted how network measurements can be used to analyse companies, and what more could be seen if we had more data at hand. The project proved that network monitoring for companies is an interesting procedure that can produce valuable results. The goal of the thesis was to prove the value of further and more detailed research. This requires, however, much larger datasets from many more compa nies, targeting specific KPIs to prove, if valid, a statistical correlation between the measurements and company performance. We fall short of it, but we encourage further research on the subject with spe cific and targeted measurements. In fact, we were able to demonstrate the existence of the intended relationship outsourcing measurements, i.e. using measurements meant for other purposes. And this relationship is clearly worth investigating. Bibliography

[1] A. Gerber, J. Pang, O. Spatscheck, and S. Venkataraman, Speed testing without speed tests: estimating achievable download speed from passive measurements, , 7. [2] A. Finamore, M. Mellia, M. Meo, M. M. Munafo, P. D. Torino, and D. Rossi, Experiences of internet traffic monitoring with tstat, 25, 8. [3] L. Subramanian, S. Agarwal, J. Rexford, and R. Katz, Characterizing the internet hierarchy from multiple vantage points, in Proceedings.TwentyFirst Annual Joint Conference of the IEEE Computer and Communications Societies, Vol. 2 (IEEE) pp. 618–627. [4] R. K. Yin, Case study research design and methods, , 8. [5] T. Fiebig, K. Borgolte, S. Hao, C. Kruegel, and G. Vigna, Something from nothing (there): Collecting global IPv6 datasets from DNS, in Passive and Active Measurement, Vol. 10176, edited by M. A. Kaafar, S. Uhlig, and J. Amann (Springer International Publishing) pp. 30–43. [6] D. Giordano, Five years at the edge: Watching internet from the ISP network, . [7] Wikipeda, Tumblr, . [8] S. Rodriguez, Verizon sells tumblr to wordpress owner six years after yahoo bought it for $1.1 billion, . [9] A. Cuthbertson, Tumblr says porn ban was ‘right decision’ despite 20 per cent drop in traffic, . [10] Statista, Tumblr.com: unique visitors 2019, (). [11] Y. Berman, Verizon sells tumblr to the owner of WordPress for 2% of its 2013 valuation, . [12] N. Brownlee and K. Claffy, Internet measurement, 8, 30. [13] L. Kleinrock, An early history of the internet [history of communications], 48, 26. [14] V. Cerf, Guidelines for internet measurement activities, . [15] B. Zhou, J. Li, X. Wang, Y. Gu, Y. Hu, and L. Zhu, Online internet traffic monitoring system using spark streaming, 1, 47. [16] C. Williamson, Internet traffic measurement, 5, 70. [17] A. Kozemcak and T. Kovacik, Different network traffic measurement techniques – possibilities and results, , 4. [18] M. Taylor and A. Murphy, SMEs and ebusiness, 11, 280. [19] V. K. Adhikari, Yang Guo, Fang Hao, M. Varvello, V. Hilt, M. Steiner, and Z.L. Zhang, Unreel ing netflix: Understanding and improving multiCDN movie delivery, in 2012 Proceedings IEEE INFOCOM (IEEE) pp. 1620–1628. [20] A.J. Su, D. R. Choffnes, A. Kuzmanovic, and F. E. Bustamante, Drafting behind akamai (travelocitybased detouring), in Proceedings of the 2006 Conference on Applications, Tech nologies, Architectures, and Protocols for Computer Communications, SIGCOMM ’06 (ACM) pp. 435–446, eventplace: Pisa, Italy. [21] R. Holz, L. Braun, N. Kammenhuber, and G. Carle, The SSL landscape: a thorough analysis of the x.509 PKI using active and passive measurements, in Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference IMC ’11 (ACM Press) p. 427.

99 100 Bibliography

[22] G.Q. Peng, G. Xue, and Y.C. Chen, Network measurement and performance analysis at server side, 10, 67. [23] T. Fiebig, K. Borgolte, S. Hao, C. Kruegel, G. Vigna, and A. Feldmann, In rDNS we trust: Revisiting a common datasource’s reliability, in Passive and Active Measurement, Vol. 10771, edited by R. Beverly, G. Smaragdakis, and A. Feldmann (Springer International Publishing) pp. 131–145. [24] N. Chatzis, G. Smaragdakis, J. Böttger, T. Krenc, and A. Feldmann, On the benefits of using a large IXP as an internet vantage point, in Proceedings of the 2013 conference on Internet measurement conference IMC ’13 (ACM Press) pp. 333–346. [25] J. Morley, K. Widdicks, and M. Hazas, Digitalisation, energy and data demand: The impact of internet traffic on overall and peak electricity consumption, 38, 128. [26] T. L. Mesenbourg, MEASURING ELECTRONIC BUSINESS, , 20. [27] C. Huang, A. Wang, J. Li, and K. W. Ross, Measuring and evaluating largescale CDNs, , 14. [28] S. Triukose, Z. Wen, and M. Rabinovich, Measuring a commercial content delivery network, in Proceedings of the 20th International Conference on World Wide Web, WWW ’11 (ACM) pp. 467–476, eventplace: Hyderabad, India. [29] R. E. Stake, Qualitative case studies. , 443. [30] B. Flyvbjerg, Five misunderstandings about casestudy research, 12, 219. [31] K. M. Eisenhardt, Building theories from case study research, 14, 532. [32] R. Heale and A. Twycross, What is a case study? 21, 7. [33] P. Baxter and S. Jack, Qualitative case study methodology: Study design and implementation for novice researchers, , 18. [34] A. E. Egger and A. Carpi, Data analysis and interpretation, . [35] R. Haynal, Traceroute overview, . [36] E. Tal, Measurement in science, in The Stanford Encyclopedia of Philosophy, edited by E. N. Zalta (Metaphysics Research Lab, Stanford University) fall 2017 ed. [37] A. M. Helmenstine, Measurement definition in science, . [38] V. Paxson, Strategies for sound internet measurement, in Proceedings of the 4th ACM SIGCOMM conference on Internet measurement IMC ’04 (ACM Press) p. 263. [39] G. Almes, S. Kalidindi, and M. Zekauskas, A oneway delay metric for IPPM, . [40] K. Ervasti, A survey on network measurement: Concepts, techniques, and tools, , 8. [41] V. Mohan, Y. R. J. Reddy, and K. Kalpana, Active and passive network measurements : A survey, 2, 14. [42] T. Fisher, Definition of bandwidth and how to tell how much you have, . [43] P. Eardley, A. Morton, M. Bagnulo, T. Burbridge, P. Aitken, and A. Akhter, A framework for largescale measurement of broadband performance (LMAP), . [44] N. Brownlee, C. Mills, and G. Ruth, Traffic flow measurement: Architecture, . [45] A. Morton, Active and passive metrics and methods (with hybrid types inbetween), . [46] H. Haddadi, M. Rio, G. Iannaccone, A. Moore, and R. Mortier, Network topologies: inference, modeling, and generation, 10, 48. [47] B. Eriksson, P. Barford, and R. Nowak, Network discovery from passive measurements, , 12. Bibliography 101

[48] K. Borgolte, T. Fiebig, S. Hao, C. Kruegel, and G. Vigna, Cloud strife: Mitigating the security risks of domainvalidated certificates, in Proceedings 2018 Network and Distributed System Security Symposium (Internet Society).

[49] D. Moore, C. Shannon, D. J. Brown, G. M. Voelker, and S. Savage, Inferring internet denialof service activity, 24, 25.

[50] S. Savage, STING: A TCPBASED NETWORK MEASUREMENT TOOL, , 10.

[51] K. Johnson, J. Carr, M. Day, and M. Kaashoek, The measured performance of content distribution networks, 24, 202.

[52] D. D. Clark, S. Bauer, W. Lehr, K. C. Claffy, A. D. Dhamdhere, B. Huffaker, and M. Luckie, Measurement and analysis of internet interconnection and congestion, 10.2139/ssrn.2417573.

[53] R. Lauzer, B. Lev, and J. Livnat, Internet traffic and portfolio returns, .

[54] M. Trevisan, D. Giordano, I. Drago, M. Mellia, and M. Munafo, Five years at the edge: watching internet from the ISP network, in Proceedings of the 14th International Conference on emerging Networking EXperiments and Technologies CoNEXT ’18 (ACM Press) pp. 1–12.

[55] V. K. Adhikari, S. Jain, Y. Chen, and Z.L. Zhang, Vivisecting YouTube: An active measurement study, in 2012 Proceedings IEEE INFOCOM (IEEE) pp. 2521–2525.

[56] D. Ciullo, M. Mellia, and M. Meo, Traditional IP measurements: What changes in a today multi media IP network, in 2008 4th International Telecommunication Networking Workshop on QoS in Multiservice IP Networks (IEEE) pp. 262–267.

[57] R. K. P. Mok, V. Bajpai, A. Dhamdhere, and K. C. Claffy, Revealing the loadbalancing behavior of YouTube traffic on interdomain links, in Passive and Active Measurement, Vol. 10771, edited by R. Beverly, G. Smaragdakis, and A. Feldmann (Springer International Publishing) pp. 228–240.

[58] P. Gill, M. Arlitt, Z. Li, and A. Mahanti, Youtube traffic characterization: A view from the edge, in Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement, IMC ’07 (ACM) pp. 15–28, eventplace: San Diego, California, USA.

[59] Marr Bernard, Intellectual capital – defining key performance indicators for organizational knowl edge assets, 10, 551.

[60] H. ALwaer and D. ClementsCroome, Key performance indicators (KPIs) and priority setting in using the multiattribute approach for assessing sustainable intelligent buildings, 45, 799.

[61] I. Jefferson, D. V. L. Hunt, C. A. Birchall, and C. D. F. Robers, Sustainability indicators for environmental geotechnics, , 57.

[62] F. J. Reh, How key performance indicators work, .

[63] K. Kononenko, What is a KPI? a comprehensive guide with tips and examples, .

[64] Klipfolio, What is a KPI? definition, bestpractices, and examples, .

[65] E. K. Huizingh, The antecedents of web site performance, 36, 1225.

[66] R. G. Eccles, The performance measurennent manifesto, , 8.

[67] D. Barnes and M. Hinton, Searching for ebusiness performance measurement systems, 10, 8.

[68] A. Neely, B. Marr, C. Adams, and N. Kapashi, Measuring eBusiness performance, in Business performance measurement, edited by A. Neely (Cambridge University Press) pp. 343–360.

[69] E. A. Demers and B. I. Lev, A rude awakening: Internet shakeout in 2000, 10.2139/ssrn.244231.

[70] H. Akers, What is a company’s annual report, . 102 Bibliography

[71] S. King, Financial reports vs. management reports: What’s the difference? .

[72] Alphabet, Alphabet investor relations, .

[73] salesforce, Salesforce products and business solutions: The customer success platform, .

[74] Netflix, Netflix financials quarterly earnings, .

[75] Snapchat, Snap inc. .

[76] J. B. Berk and P. M. DeMarzo, Corporate finance, third edition ed., The Pearson series in finance (Pearson).

[77] B. Marr, G. Schiuma, and A. Neely, Intellectual capital – defining key performance indicators for organizational knowledge assets, 10, 551.

[78] Z. Puspitaningtyas, Is financial performance reflected in stock prices? in Proceedings of the 2nd International Conference on Accounting, Management, and Economics 2017 (ICAME 2017) (Atlantis Press).

[79] C. Eom, G. Oh, and S. Kim, Deterministic factors of stock networks based on crosscorrelation in financial market, 383, 139.

[80] S. Quarton, 7 key metrics to track to ensure the success of your website, .

[81] H. Enochson, 27 examples of key performance indicators, .

[82] T. Jackson, 18 key performance indicator (KPI) examples defined, .

[83] CAIDA, CAIDA data overview of datasets, monitors, and reports, .

[84] MAWI, MAWI working group traffic archive, .

[85] RIPE, RIPE atlas, .

[86] O. Intel, Index of /data, .

[87] CNetS, Data repository | CNetS, .

[88] H. Electrics, Internet exchange report, .

[89] Stanford, Stanford large network dataset collection, .

[90] K. Imran, M. Mellia, and M. Meo, Measurements of multicast television over IP, in 2007 15th IEEE Workshop on Local & Metropolitan Area Networks (IEEE) pp. 170–175.

[91] Owler, Salesforce competitors, revenue and employees owler company profile, .

[92] Salesforce, What is salesforce? ().

[93] Salesforce, Salesforce financials, ().

[94] R. T. Braden, Perspective on the host requirements RFCs, ().

[95] R. Braden, Requirements for internet hosts application and support, ().

[96] M. A. Padlipsky, Perspective on the ARPANET reference model, .

[97] R. Braden, rfc1122.pdf, ().

[98] A. G. Tharakan and G. Kshitiz, Salesforce takes aim at ecommerce with $2.8 billion demandware buy, .

[99] N. Medhora and G. Tharakhan, Twitter’s shares slump on report that bids are unlikely, .

[100] N. Medhora and S. Nellis, REFILEUPDATE 2salesforce’s revenue forecast beats estimates,... . Bibliography 103

[101] Reuters, BRIEFsalesforce says won customers including citigroup, PNC bank... . [102] Salesforce, KONE selects salesforce service cloud lightning to elevate its customer service, (). [103] Salesforce, BRIEFamazon web services and salesforce extend global strategic... (). [104] Salesforce, Cisco and salesforce announce global strategic alliance, (). [105] Salesforce, AWS and salesforce extend global strategic alliance, (). [106] Salesforce, Salesforce announces strategic agreement with dell technologies, (). [107] Salesforce, IBM and salesforce announce landmark global strategic partnership, (). [108] Salesforce, Salesforce and google form new strategic partnership, (). [109] S. Rishika, Cloudbased software maker salesforce’s revenue jumps 26.8 pct, . [110] M. Iqbal, Netflix revenue and usage statistics (2018), . [111] CivicScience, CivicScience | the new netflix nation, . [112] EMarketer, Top 10 countries, ranked by netflix user penetration, 2018 (% of digital video viewers) eMarketer, . [113] L. Brennan, How netflix expanded to 190 countries in 7 years, . [114] P. Analytics, Parrot analytics the global TV demand report 2018.pdf, (). [115] P. Analytics, Italy SVOD market share trends based on audience demand for digital originals, (). [116] B. Pekic, Italy sees SVoD boom, . [117] G. Scifo, Italy: Digital lag a cause for concern? . [118] A. Flak and S. Jewkes, Italians need convincing as well as connecting to join digital drive, . [119] Statista, Videoondemand worldwide | statista market forecast, (). [120] Wikipedia, Suburra: Blood on Rome, (), page Version ID: 907414635. [121] Nakono, Netflix revenue 20062019 | NFLX | MacroTrends, (). [122] Nakono, Netflix streaming revenues italy company forecast : Nakono fusion, (). [123] Nakono, Market research, insight, analysis, trends & statistics by nakono, the digital media specialists, (). [124] Wikipedia, List of original programs distributed by netflix, (). [125] S. Alhabash and M. Ma, A tale of four platforms: Motivations and uses of facebook, twitter, instagram, and snapchat among college students? 3, 205630511769154. [126] G. Ambrosio, #snapchat: Funzionalità, numeri utenti in italia e nel mondo +6 case study, . [127] Statista, US snapchat usage by age 2018, (). [128] J. M. Vaterlaus, K. Barnett, C. Roche, and J. A. Young, “snapchat is more personal”: An ex ploratory study on snapchat behaviors and young adult interpersonal relationships, 62, 594. [129] Y. Finance, Snap inc. (SNAP) stock historical prices & data, . [130] R. Beales, Snapchat valuation requires different filters, . [131] L. Hirsch, Snapchat owner snap raises $3.4 billion in IPO source, . [132] S. Carew, L. Krauskopf, and R. Campos, Snap shares fall below $17 IPO price for first time, . 104 Bibliography

[133] M. Supantha, Snap shares hit record low as user growth miss unnerves investors, . [134] A. Venugopal, Snap shares sink as two million users move on, . [135] J. Constine, Instagram launches “stories,” a snapchatty feature for imperfect sharing | TechCrunch, . [136] Statista, Instagram stories daily active users 2019, (). [137] Statista, Instagram: active users 2018, ().