DEGREE PROJECT IN TECHNOLOGY, FIRST CYCLE, 15 CREDITS STOCKHOLM, SWEDEN 2020 Scalability of push and pull based event notification MARCUS NILSSON DANIEL DUNÉR KTH ROYAL INSTITUTE OF TECHNOLOGY SCHOOL OF ELECTRICAL ENGINEERING AND COMPUTER SCIENCE Scalability of push and pull based event notification A comparison between webhooks and polling MARCUS NILSSON DANIEL DUNÉR Högskoleingenjör Datateknik Date: June 9, 2020 Supervisor: Peter Sjödin Examiner: Markus Hidell School of Electrical Engineering and Computer Science Swedish title: Skalbarhet hos push- och pullbaserad eventnotifikation Swedish subtitle: En jämförelse mellan webhooks och polling Scalability of push and pull based event notification / Skalbarhet hos push- och pullbaserad eventnotifikation © 2020 Marcus Nilsson and Daniel Dunér Abstract | i Abstract Today’s web applications make extensive use of APIs between server and client, or server to server in order to provide new information in the form of events. The question was whether the different methods of procuring events are different in how they scale. This study aims to compare performance between webhooks and polling, the two most commonly used pull and push based methods for event notification when scaling up traffic. The purpose is to create a basis for developers when choosing the method for event notification. The comparison has been developed through measurements of typical indicators of good performance for web applications: CPU usage, memory usage and response time. The tests gave indications that webhooks perform better in most circumstances, but further testing is needed in a more well-defined environment to draw a confident conclusion. Keywords Event notification, Webhooks, Polling, Scalability, Performance test, Web application ii | Abstract Sammanfattning | iii Sammanfattning Dagens webbapplikationer använder sig i stor utsträckning av API:er mellan server och klient, eller server till server för att inhämta ny information i form av events (händelser). Frågan är om de olika metoder som finns för att inhämta events skalar olika bra. Förevarande studie ämnar att jämföra prestanda mellan ”webhooks” och ”polling”, de två mest använda pull- och pushbaserade metoderna för eventnotifikation vid uppskalning av trafik. Syftet är att skapa ett underlag för utvecklare vid valet av metod för eventnotifikation. Jämförelsen har tagits fram genom mätningar av typiska indikatorer för god prestanda hos en webbapplikation: CPU-användning, minnesanvändning och svarstid. Testerna gav indikationer om att webhooks är bättre men det krävs vidare testning i en mer väldefinierad miljö för att dra en säkrare slutsats. Nyckelord Eventnotifikation, Webhooks, Polling, Skalbarhet, Prestandatest, Webbapplikation iv | Sammanfattning Acknowledgments | v Acknowledgments We want to thank our supervisor Peter Sjödin and our examiner Markus Hidell for their advice and contributions during this degree project. Stockholm, June 2020 Marcus Nilsson and Daniel Dunér vi | Acknowledgments CONTENTS | vii Contents 1 Introduction 1 1.1 Background ........................... 2 1.2 Problem ............................. 4 1.3 Goal ............................... 4 1.4 Benefits, Ethics and Sustainability ............... 5 1.5 Delimitations .......................... 6 1.6 Outline ............................. 6 2 Theoretical Background 7 2.1 Polling .............................. 8 2.2 Webhook ............................ 9 2.3 Performance criteria for web applications ........... 10 2.4 Testing scalability ........................ 10 2.4.1 Performance testing ................... 10 2.4.2 Core Performance Testing Activities .......... 11 2.4.3 Test Environment .................... 12 2.4.4 Identifying Acceptance Criteria ............ 12 2.4.5 Planning and Designing Tests ............. 13 2.4.6 Configuring the Test Environment ........... 13 2.4.7 Implementing the Designed Test ............ 13 2.4.8 Performing the Tests .................. 14 2.4.9 Analyze Results, Report, and Retest .......... 14 2.5 Related Work .......................... 15 3 Methodology and Methods 17 3.1 Research process ........................ 17 3.2 The model ............................ 17 3.3 Performance Testing ...................... 18 3.3.1 Identifying test environment .............. 18 viii | CONTENTS 3.3.2 Identifying performance Acceptance Criteria ..... 18 3.3.3 Planning and designing tests .............. 19 4 Implementation 21 4.1 The server ............................ 21 4.2 The client ............................ 21 4.3 Webhook ............................ 22 4.4 Polling .............................. 23 4.5 Implemented test design .................... 23 4.5.1 Load test ........................ 24 4.5.2 Stress test ........................ 24 4.5.3 Endurance test ..................... 24 4.6 Measurement Criteria ...................... 25 4.6.1 CPU and memory measurements ............ 25 4.6.2 Response time measurements ............. 25 4.7 Testbed ............................. 25 5 Results 27 5.1 Load test ............................. 27 5.1.1 CPU ........................... 28 5.1.2 Memory ......................... 30 5.2 Stress test ............................ 31 5.2.1 CPU ........................... 32 5.2.2 Memory ......................... 35 5.3 Endurance test .......................... 37 5.3.1 CPU ........................... 38 5.3.2 Memory ......................... 41 6 Discussion 45 6.1 Load test ............................. 45 6.2 Stress test ............................ 46 6.3 Endurance test .......................... 47 6.4 Evaluation against acceptance criteria ............. 48 6.5 Conclusion ........................... 49 6.6 Further Research ........................ 49 6.7 Reflections ........................... 50 References 51 Contents | ix A 55 A.1 Load Test Plots ......................... 56 A.1.1 CPU ........................... 56 A.1.2 Memory ......................... 57 x | Contents LIST OF FIGURES | xi List of Figures 4.1 Webhook model. ......................... 22 4.2 Polling model .......................... 23 5.1 Central Processing Unit (CPU) usage of one burst of 100 clients. 29 5.2 CPU usage of one burst of 1000 clients. ............ 29 5.3 Memory usage of one burst of 1000 clients. .......... 30 5.4 CPU usage of one burst of 10 000 clients. ........... 32 5.5 CPU usage of one burst of 11 000 clients. ........... 33 5.6 CPU usage of one burst of 12 000 clients. ........... 33 5.7 CPU usage of one burst of 13 000 clients. ........... 34 5.8 CPU usage of one burst of 14 000 clients. ........... 34 5.9 Memory usage of one burst of 11 000 clients. ......... 35 5.10 Memory usage of one burst of 12 000 clients. ......... 35 5.11 Memory usage of one burst of 13 000 clients. ......... 36 5.12 Memory usage of one burst of 14 000 clients. ......... 36 5.13 CPU usage of 300 bursts of 500 clients each .......... 38 5.14 CPU usage of 900 bursts of 500 clients each .......... 39 5.15 CPU usage of 1800 bursts of 500 clients each ......... 39 5.16 CPU usage of 300 burst of 100 clients each. .......... 40 5.17 CPU usage of 300 burst of 1000 clients each. ......... 40 5.18 Memory usage of 300 burst of 100 clients each. ........ 41 5.19 Memory usage of 300 burst of 1000 clients each. ....... 42 5.20 Memory usage of 300 burst of 500 clients each, using polling. 42 5.21 Memory usage of 900 burst of 500 clients each. ........ 43 5.22 Memory usage of 1800 burst of 500 clients each. ....... 43 A.1 CPU usage of one burst of 1 client using polling. ....... 56 A.2 CPU usage of one burst of 10 client using polling. ....... 56 A.3 Memory usage of one burst of 1 client using polling. ..... 57 A.4 Memory usage of one burst of 10 client using polling. ..... 57 xii | LIST OF FIGURES A.5 Memory usage of one burst of 100 client using polling. .... 58 A.6 Memory usage of one burst of 1000 client using polling. ... 58 LIST OF TABLES | xiii List of Tables 4.1 Testbed computer ........................ 25 5.1 Average response time measured during load testing. ..... 28 5.2 Total number of requests .................... 28 5.3 Average response time for stress test .............. 31 5.4 Total number of requests sent for stress test .......... 31 5.5 Average response time ..................... 37 5.6 Total number of requests sent .................. 37 xiv | LIST OF TABLES List of acronyms and abbreviations | xv List of acronyms and abbreviations API Application Programming Interface CPU Central Processing Unit DDoS Denial of Service Attack HTTP Hypertext Transfer Protocol HTTPS Hypertext Transfer Protocol Secure NAT Network Address Translation RAM Random Access Memory REST Representational State Transfer SSL Secure Sockets Layer TCP Transmission Control Protocol URL Uniform Resource Locator xvi | List of acronyms and abbreviations Introduction | 1 Chapter 1 Introduction The development of connected devices, mobile phones, and personal computers has been rapid. This has been accompanied by an increase in the amount of people with access to the Internet. In 2019, more than half of the global population used the Internet, a rapid increase from 16% in 2002 [1]. As access to the Internet increases and more and more of our daily actions are digital, expectations on the speed at which new data can be delivered to applications (hereafter, apps) and devices are also increasing. Events, such as the arrival of email, news, and messages, are expected to be delivered instantly. As most apps and devices get their data from servers it becomes a problem for the server of how to best notify