DEGREE PROJECT IN TECHNOLOGY, FIRST CYCLE, 15 CREDITS STOCKHOLM, SWEDEN 2019 Comparing Cloud Architectures in terms of Performance and Scalability Perttu Jääskeläinen KTH ROYAL INSTITUTE OF TECHNOLOGY ELECTRICAL ENGINEERING AND COMPUTER SCIENCE Authors Perttu Jääskeläinen; [email protected] Degree Programme in Computer Engineering KTH Royal Institute of Technology Place for Project Stockholm, Sweden Examiner Fadil Galjic KTH Royal Institute of Technology Supervisor Anders Sjögren KTH Royal Institute of Technology Abstract Cloud Computing is becoming increasingly popular, with large amounts of corporations revenue coming in from various cloud solutions offered to customers. When it comes to choosing a solution, multiple options exist for the same problem from many competitors. This report focuses on the ones offered by Microsoft in their Azure platform, and compares the architectures in terms of performance and scalability. In order to determine the most suitable architecture, three offered by Azure are considered: Cloud Services (CS), Service Fabric Mesh (SFM) and Virtual Machines (VM). By developing and deploying a REST Web API to each service and performing a load test, average response times in milliseconds are measured and compared. To determine scalability, the point at which each service starts timing out requests is identified. The services are tested both by scaling up, by increasing the power of a single instance of a machine, and by scaling out, if possible, by duplicating instances of machines running in parallel. The results show that VMs fall considerably behind both CS and SFM in both performance and scalability, for a regular use case. For low amounts of requests, all services perform about the same, but as soon as the requests increase, it is clear that both SFM and CS outperform VMs. In the end, CS comes ahead both in terms of scalability and performance. Further research may be done into other platforms which offer the same service solutions, such as Amazon Web Services (AWS) and Google Cloud, or other architectures within Azure. Keywords Software as a Service, SaaS, Architectures, Scaling, Performance, Microsoft Azure, Virtual Machines, Cloud Services, Mesh Network 2 Abstract Molntjänster blir alltmer populära i dagens industri, där stora mängder av företagens omsättning består av tjänster erbjudna i form av molnlösningar. När det kommer till att välja en lösning finns många för samma problem, där det är upp till kunden att välja vilken som passar bäst. Denna rapport fokuserar på tjänster erbjudna av Microsofts Azure plattform, i en jämförelse av arkitekturer som belastningstestas för att mäta prestanda och skalbarhet. För att avgöra vilken arkitektur som är optimalast mäts tre olika tjänster erbjudna i Azure: Cloud Services (CS), Service Fabric Mesh (SFM) och Virtual Machines (VM). Detta görs genom att utveckla och deploya ett REST Web API som är simulerat med användare, där prestanda mäts genom att ta medelresponstiden i millisekunder per anrop. För att avgöra skalbarhet identifieras en punkt där tjänsten inte längre klarar av antalet inkommande anrop och börjar returnera felkoder. Maskinerna för varje tjänst testas både genom att skala upp, genom att förstärka en maskin, men även genom att skala ut, där det skapas flera instanser av samma maskin. Resultatet visar att Virtual Machines hamnar betydligt efter både CS och SFM i både prestanda och skalbarhet för ett vanligt användarfall. För låga mängder anrop ligger samtliga tjänster väldigt lika, men så fort anropen börjar öka så märks det tydligt att SFM och CS presterar bättre än Virtual Machines. I slutändan ligger CS i framkant, både i form av prestanda och skalbarhet. Vidare undersökning kan göras för de olika plattformarna erbjudna av konkurrenter, så som Amazon Web Services (AWS) och Google Cloud, samt andra arkitekturer från Azure. Nyckelord Software as a Service, SaaS, Arkitekturer, Skalning, Prestanda, Microsoft Azure, Virtuella Maskiner, Cloud Services, Mesh Network 3 Acknowledgements I would like to thank Anders Sjögren for the guidance given throughout the project, in terms of constructive feedback in monthly meetings, ensuring that the project stays within its scope and requirements made by KTH. I would also like to thank Fadil Galjic for being the examinator of the project, providing valuable constructive feedback on how the project is conducted and presented in a clear, concise way. Lastly, I would like to thank Niko Rosenquist and Tim Liberg from Triona for giving me the opportunity to perform this project for them, with special thanks to Tim for providing guidance and support throughout the project. 4 Contents 1 Introduction 1 1.1 Background ................................ 1 1.2 Problem .................................. 2 1.3 Purpose .................................. 3 1.4 Goal .................................... 3 1.5 Methodology ............................... 4 1.6 Delimitations ............................... 4 1.7 Outline .................................. 5 2 Theoretical Background 7 2.1 Cloud Computing and Architectures .................. 7 2.2 Microsoft Azure ............................. 12 2.3 Measuring Performance and Scalability . 14 2.4 Related Works .............................. 15 3 Methodology 17 3.1 Research Strategies ........................... 17 3.2 Research Methods ............................ 19 3.3 Managing the degree project ...................... 23 3.4 Tools .................................... 25 4 Developing and Deploying The API, Gathering Data 27 4.1 Developing the API ............................ 27 4.2 Deploying the API ............................ 30 4.3 Gathering Performance and Scalability Data . 36 5 Measured Data and Service Comparison 43 5.1 Gathered Data .............................. 43 5.2 Service Comparison ........................... 64 5 6 Discussion and Conclusions 69 6.1 Research Methods ............................ 69 6.2 Validity and Reliability ......................... 71 6.3 Conclusion ................................ 73 6.4 Future Work ............................... 74 References 75 7 Appendices 78 A API Specification 78 B API Sequence Diagram 79 6 1 Introduction Cloud computing consists of delivering on-demand computing resources from a cloud provider, typically on a ”pay for what you use” basis. This can be in the form of different types of services, most commonly as: IaaS (Infrastructure as a Service), PaaS (Platform as a Service) and SaaS (Software as a Service) [3, 20, 25]. By hosting parts of an application on an external party, businesses can avoid large upfront investments in hardware, infrastructure, maintenance and management, only paying for what they use and being able to focus on developing and deploying applications. Cloud computing is becoming increasingly popular in the industry, accumulating $266B in revenue in 2017 and predicted to grow up to $411B by 2020 [13]. 1.1 Background When it comes to choosing a service, multiple platforms and solutions exist for the same problem. Different portions of an application can be hosted and managed by the service provider, while the remainder is managed by the individual utilizing the service. In addition to choosing a suitable service provider and solution, it is unclear how they perform in different categories, such as performance and scalability. Microsoft has a platform for cloud-based solutions, referred to as Microsoft Azure [11]. Here, they offer various cloud solutions with different prize plans and capabilities, such as Virtual Machines [9], Cloud Services [5] and Service Fabric Mesh [7]. 1 1.2 Problem Triona [32] is a software solutions company based in the Nordics, with headquarters located in Dalarna, Sweden, and offices located in Norway, Sweden and Finland. They employed 120 people in 2017 and had an annual revenue of about 150 MSEK [1]. Their software is used by various customers, including but not limited to, the forest industry, transportation, live-streaming and oil/gas. Triona utilizes Microsoft Azure on a daily basis, hosting most of their software, databases and development environments, among others, using services provided by Azure. Even though many of the services are used daily, there is not a clear picture of exactly how much they differ in performance and scalability. An over- all difference is detectable and known, but they are looking to analyze each service more specifically in order to better differentiate between them in order to save money and improve performance for existing and future systems. A sudden increase of users (surge) utilizing their services is common, in cases such as a popular hockey game being live-streamed at a certain time. This surge may cause spikes in latencies and response times, which is a specific problem they are looking to solve. Since Cloud Computing is still relatively new with additional services coming every year, choosing the correct service may be difficult, and information about which one performs the best and handles the most users well is of interest. In order to gather information to answer their questions, the following research question can be defined: RQ: How do various Cloud Architectures differ in terms of performance and scalability? 2 1.3 Purpose Due to the increasing popularity of Cloud Computing, analyzing services in terms of performance and scalability is of interest for both small and large organizations. By doing a study into services within Azure, organizations will have an idea of the benefits from moving to the cloud and how the different