Mocking Saas Cloud for Testing
Total Page:16
File Type:pdf, Size:1020Kb
DEGREE PROJECT FOR MASTER OF SCIENCE IN ENGINEERING GAME AND SOFTWARE ENGINEERING Mocking SaaS Cloud for Testing Johannes Henriksson | Simon Svensgård Blekinge Institute of Technology, Karlskrona, Sweden, 2017 Supervisor: Mikael Svahnberg, Department of Software Engineering, BTH Abstract In this paper we evaluate how software testing is affected by the usage of a mock-object, a dummy implementation of a real object, in place of having data in a cloud that is accessed through an API. We define the problems for testing that having data in the cloud brings, which of these problems a mock-object can remedy and what problems there are with testing using the mock-object. We also evaluate if testing using the mock-object can find the same faults as testing against the cloud and if the same code can be covered by the tests. This is done at Blekinge Institute of Technology(BTH) by creating an integration system for the company Cybercom Sweden and Karlskrona Municipality. This integration system is made in C] and works by syncing schedules from Novaschem to a cloud service, Google Calendar. With this paper we show that a mock-object in place of a cloud is very useful for testing when it comes to clean-up, triggering certain states and to avoid query limitations. Keywords: Mock-Object, Cloud Computing, Testing, Test-Evaluation i Sammanfattning I detta arbete utvärderar vi hur programvarutestning påverkas av användandet av ett mock-objekt, en dummy-implementation av ett riktigt objekt, istället för att ha data i ett moln som man kommer åt via ett API. Vi definierar de problem som uppkommer av att ha data i molnet, vilka problem som kan avhjälpas av mock-objektet och vilka problem mock-objektet medför. Vi utvärderar även om testning med ett mock-objekt kan finna samma fel som testning mot molnet och om samma kod kan täckas av testerna. Detta görs på Blekinge Tekniska Högskola(BTH) genom att skapa ett integrationssystem för företaget Cybercom Sweden och Karlskrona Kommun. Integrationssystemet görs i C] och fungerar som så att det synkar scheman från Novaschem till en molntjänst, Google Calendar. Med detta arbete visar vi att ett mock-objekt istället för molnet är väldigt användbart när det kommer till städning efter tester, att utlösa vissa tillstånd och för att undvika begränsningar. Nyckelord: Mock-Objekt, Molntjänster, Testning, Testutvärdering iii Preface This thesis is the final step for our master in game and software engineering at BTH and represents 20 weeks of full time study. At BTH, we would like to thank our supervisor Mikael Svahnberg and our examiner Emil Alégroth for valuable feedback during the project. As well as Bogdan Marculescu for help with insight into the world of academia. We want to thank Alexander Andersson and Johan Persbeck at Cybercom Sweden for providing us with the idea and assistance throughout the project. We also want to thank those who have read and given feedback on this thesis. v Nomenclature Acronyms API Application Programming Interface AWS Amazon Web Services FITTEST Future Internet Testing IaaS Infrastructure as a Service NIST National Institute of Standards and Technology PaaS Platform as a Service SaaS Software as a Service SUT System Under Test UI User Interface vii Table of Contents Abstract i Sammanfattning (Swedish) iii Preface v Nomenclature vii Acronyms ........................................ vii Table of Contents ix 1 Introduction 1 1.1 Introduction .................................... 1 1.2 Background .................................... 1 1.3 Objectives ..................................... 2 1.4 Delimitations .................................... 2 1.5 Thesis question and technical problem ...................... 2 2 Theoretical Framework 3 2.1 Cloud Computing ................................. 3 2.2 Testing ....................................... 5 2.3 Test Evaluation .................................. 8 3 Method - Design 11 3.1 Literature Study .................................. 11 3.2 Interview and Observations ............................ 12 3.3 Testing ....................................... 12 3.4 Test Evaluation .................................. 12 4 Method - Execution 15 4.1 Literature Study .................................. 15 4.2 Interview and Observations ............................ 15 4.3 System Under Test ................................ 16 4.4 Testing ....................................... 20 4.5 Test Evaluation .................................. 21 4.6 Validity Threats .................................. 22 5 Results 23 5.1 Research Question 1 ............................... 23 5.2 Research Question 2 ............................... 26 5.3 Research Question 3 ............................... 28 6 Discussion 29 6.1 Coverage Measurements ............................. 29 6.2 Mutation Score .................................. 29 6.3 Advantages .................................... 29 6.4 Disadvantages ................................... 31 6.5 When to use a mock-object ............................ 31 6.6 Sustainability ................................... 31 7 Conclusions 33 8 Recommendations and Future Work 35 References 37 ix A Interview Developer Cybercom A-1 B Example unit tests from Test Suite A-5 B.1 EventExists .................................... A-5 B.2 NewActivityAdded ................................. A-6 C Example Mock-Object Unit Test A-7 x 1 INTRODUCTION 1.1 Introduction Cloud computing [3] has been on the rise over the past decade[18] with a heavy increase of its use in the software industry. A cloud is to put it simply, computer functionality that can be reached over network. Cloud computing can, depending on the cloud service, be used to host executable code and/or data through different ways of access; everything from simple Application Programming Interfaces (APIs) or User Interfaces (UIs) that give the user very little control to full control over the system. The focus of this paper is the highest abstraction level of cloud services, Software as a Service (SaaS), where the user have the least control over the cloud system. When it comes to testing of a system that uses data that is stored on a SaaS cloud, there are some extra challenges compared to having the data in a database. Since a SaaS cloud can only be accessed at a high abstraction level, such as through an API, the available requests can be limited both in how many requests can be sent and the amount of different requests; which limits the testing that can be done. This area of testing is very relevant to modern internet and not that much have been written about it before which makes it an area that have potential to evolve. This study is focused on evaluating how testing such a system will differ when using a mock of the SaaS in the form of a database compared to regular testing of the system. This is done in two main parts, with the first part being a classification of differences when testing an application that make use of a SaaS cloud and a database respectively to store data. This classification is based on a literature study as well as observations and an interview at Cybercom. The second part is done by implementing an integration system between a scheduling service by Nova Software, Novaschem[26], and a SaaS, Google Calendar[12]. This is done to see in practice what difficulties there are when it comes to testing this kind of system and if a mock of the SaaS cloud is useful in a concrete scenario. In addition to that the testing is evaluated using three test measurements, Block coverage, Branch coverage and Condition coverage. With respect to cloud testing, there is plenty of research discussing how to use the cloud for testing[34][16]. However, very little is written about testing of applications that use cloud services. This study is made to see if there are any specific problems and if a mock pattern can be used to make testing easier and in what way. 1.2 Background Cybercom Group Karlskrona have been tasked by the municipality of Karlskrona with creating an integration system between Novaschem, a scheduling system by Nova Software, and Google Calendar, a SaaS cloud. To make sure the system is robust the system needs to be well tested. Since Cybercom does not have any special routines for testing applications that are using data in a cloud and there are some challenges within this area that we thought could be explored more in depth with the possibility of finding a better testing routine. 1 1.3 Objectives The goal of this thesis project is to create a stable integration system between a scheduling service called Novaschem and Google Calendar. This integration system needs to be tested thoroughly to ensure that it is stable and functional. When testing two different approaches will be compared, regular unit testing against Google Calendar, the SaaS cloud, and unit testing using a database as a mock object. These two approaches will be evaluated and compared both in terms of what challenges and experiences from applying the techniques, but also by measuring different unit test coverages for the two test suites. 1.4 Delimitations The implementation of the integration system which will be used to compare and evaluate the testing will make use of a single SaaS, Google Calendar, and results could vary depending on what SaaS is used. However some general conclusions can be drawn from the results when taking into account the other data sources for the case study, such as the theoretical framework and looking at documentation of other SaaSs services. When looking at these SaaS we limit ourselves to 3rd party