Master Thesis Test-Suite for Automated Invariant-Based Testing of Ajax Web-Applications
Total Page:16
File Type:pdf, Size:1020Kb
Master Thesis Test-suite for automated invariant-based testing of Ajax web-applications Sven Rohde, 10378782 [email protected] March 14, 2015, 69 pages This thesis is submitted to the University of Amsterdam in partial fulfilment of the requirements for the degree of Master of Science in Software Engineering Supervisor: Vadim Zaytsev, [email protected] Host organisation: Guidion ICT Nederland B.V., http://guidion.nl Host organisation contact: Mike Out (Manager IT department), [email protected] Universiteit van Amsterdam Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering http://www.software-engineering-amsterdam.nl Contents Abstract iii Preface iv 1 Introduction 1 1.1 Host organization....................................... 2 1.2 Thesis document structure.................................. 3 2 Problem statement and motivation4 2.1 Evaluation of the proposed ATUSA approach....................... 4 2.2 Manual testing at Guidion.................................. 5 2.3 Thesis Motivation ...................................... 6 3 Research method 8 3.1 Replication of case studies in confirmatory form...................... 8 3.2 Web developer survey research ............................... 9 3.3 Case Study: Effectiveness of the integration of ATUSA.................. 10 3.4 Summary ........................................... 10 4 Background and Context 11 4.1 Why is AJAX so interesting? ................................ 11 4.2 Challenges in testing AJAX based web application.................... 11 4.3 FSM based web application testing............................. 13 4.4 Invariants........................................... 14 4.5 Crawljax............................................ 15 4.6 Automatically Testing UI States of Ajax (ATUSA).................... 16 4.6.1 Core functionality .................................. 16 4.6.2 Plug-ins........................................ 18 4.7 Continuous Integration (CI)................................. 19 5 Research 21 5.1 Cases study setup and default configuration........................ 21 5.1.1 Crawljax - ATUSA default configuration...................... 21 5.1.2 Required plug-ins for the replication........................ 22 5.2 Case study 1 Introduction.................................. 23 5.3 Case study 1.A \The Organizer" .............................. 23 5.4 Case study 1.B \TaskFreak"................................. 26 5.5 Case study 1.C \HitList" .................................. 28 5.6 Case study 1.D \The Tunnel"................................ 30 5.7 Case Study 2 \Tudu" .................................... 30 5.8 Case Study 3 \Finding real bugs".............................. 34 5.9 Invariant research....................................... 38 5.10 Tool Development ...................................... 42 5.10.1 ATUSA stack reimplementation........................... 43 5.10.2 Plug-in implementation required by the replications ............... 46 i 5.10.3 Invariant provider, collector and violation reporting ............... 47 5.10.4 CI integration..................................... 53 5.11 Case study “Effectiveness of the integration" ....................... 54 6 Analysis and Conclusions 59 6.1 Conclusion .......................................... 59 6.2 Threats to validity...................................... 60 6.2.1 Internal threats to validity.............................. 60 6.2.2 External threats to validity............................. 61 6.3 Contributions......................................... 61 6.4 Future Work ......................................... 62 Bibliography 63 7 Appendix 65 7.1 Questionnaire......................................... 65 ii Abstract Manual testing of complex AJAX based web applications can require a significant amount of time and effort. At Guidion the time limitations for manual testing leads to the identification of errors after live deployment of a new release. The web applications of Guidion were used by around 700 users for their daily work, errors in the web applications can impact the work of these users. Automated testing of dynamic AJAX based web applications was researched by the work of Mesbah et.al in [MvDR12] using the open-source crawler Crawljax. This thesis replicates the case studies described in [MvDR12] and evaluates their findings on performance, invariant creation, required manual effort and fault revealing capabilities using Crawljax and the proposed plug-in suite called ATUSA. Furthermore is included a survey research which investigates how web developer can be supported to provide invariants for ATUSA. The results of the research forms the basis for the proposed invariant collector Chrome plug-in and web application. The effectiveness of the ATUSA approach and the invariants delivered from web developers is investigated in the last case study with the ATUSA stack integrated in a Jenkins deployment pipeline at Guidion. Based on the observations the re-implemented and proposed ATUSA plug-in suite is capable to be used for automatic invariant based testing of AJAX web applications. The proposed invariant support integrated in the web developer development process, stimulates and separates the invariant creation process apart from ATUSA. Web developers are able to create invariants without Java, Crawljax or ATUSA knowledge. The integration in a sample Jenkins deployment job enables to trigger the test execution after actual test deployment. The case study reveals that the ATUSA stack integration at Guidion augments the testing phase of the web applications. iii Preface This thesis was written as part of the requirements for the Master Software Engineering of the Uni- versity of Amsterdam. The study was conducted at Guidion B.V., in Amsterdam. This thesis would never be possible without the help and support of my supervisors Vadim Zaytsev and Mike Out. Furthermore I would like to thank the whole Guidion developer team for their support and feedback during the research. Furthermore I would like to thank my family, the love and support I received from my mother Marion Rohde and my twin brother Philip Rohde. Another special thanks goes out to my father Reinhard Rohde, a lost resource of support and geniality R.I.P 2007. "One of God's own prototypes. A high-powered mutant of some kind never even considered for mass production. Too weird to live, and too rare to die." Hunter S. Thompson iv Chapter 1 Introduction Interactive \Web-2.0" [Mes09] applications are a well-known part of the world-wide-web. Still more and more known services as well as desktop applications move to the web. This reflects the trend to move applications to the web to provide more flexibility and require less effort from the end user. Reasons for this move to the web are [MvDR12]: • No installation effort. • Up to date software, reduction of maintenance and support costs. • Unbounded access to user data with any device. (Assume Internet Connectivity) • New ways of communication and collaboration. • Better chances for an international user audience. These Web-2.0 applications are wide spread and are an inherent part in the personal and business domain. Popular examples for that are Microsoft Office 3651, the Google Service Stack (with GMail, Google+)2, Atlassian Stack (Jira, Bitbucket, Confluence)3 and Social Media platforms4. The host organization realized this trend and is using different web applications to operate. Besides using existing web applications, develops and maintains the host organization own web applications for supporting their business processes. To provide more flexibility, information sharing and collaboration possibilities, the host organization started to change existing traditional web applications to interactive, responsive and complex web applications. This includes new features and constant refactoring to move existing functionality into the same direction. One of the technologies that supported and enabled this enormous change in the domain of the web is AJAX [MvD07]. AJAX is an abbreviation and stands for \Asynchronous JavaScript and XML" [Gar05]. It enables browsers to provide more functionality than only loading, rendering and navigating through static pages. It enables the browser to provide a more dynamic and interactive user experience through respon- siveness and the use of user interface (UI) components [MvDR12]. These components are also known as partial site updates through asynchronous client-server communication and client-side JavaScript execution. Despite the fact that AJAX positively impacts the whole user experience when brows- ing the web, or using web based applications, it comes with new challenges for the developers. The main challenge is testability of these modern web applications, which stands in the main focus of this thesis. Testability of the AJAX UI components is a challenging task because of the asynchronous, event-driven and stateful nature of AJAX, and of client-side JavaScript based manipulation of the Document Object Model (DOM) [MvDR12][vDM10]. In the host organization testing is a growing part of the development process, and besides functional testing of server-side components, client-side testing and UI testing gain more interest. Through the 1Microsoft Office 365 http://office.microsoft.com/ 2Google website https://google.com 3Atlassian https://atlassian.com/ 4Facebook and Twitter https://facebook.com and https://twitter.com 1 fact that the host organization moves more and more new and existing features to a new and modern user interface with an implied move of server-side functionality