Master Thesis Test-suite for automated invariant-based testing of Ajax web-applications
Sven Rohde, 10378782 [email protected]
March 14, 2015, 69 pages
This thesis is submitted to the University of Amsterdam in partial fulfilment of the requirements for the degree of Master of Science in Software Engineering
Supervisor: Vadim Zaytsev, [email protected] Host organisation: Guidion ICT Nederland B.V., http://guidion.nl Host organisation contact: Mike Out (Manager IT department), [email protected]
Universiteit van Amsterdam Faculteit der Natuurwetenschappen, Wiskunde en Informatica Master Software Engineering http://www.software-engineering-amsterdam.nl Contents
Abstract iii
Preface iv
1 Introduction 1 1.1 Host organization...... 2 1.2 Thesis document structure...... 3
2 Problem statement and motivation4 2.1 Evaluation of the proposed ATUSA approach...... 4 2.2 Manual testing at Guidion...... 5 2.3 Thesis Motivation ...... 6
3 Research method 8 3.1 Replication of case studies in confirmatory form...... 8 3.2 Web developer survey research ...... 9 3.3 Case Study: Effectiveness of the integration of ATUSA...... 10 3.4 Summary ...... 10
4 Background and Context 11 4.1 Why is AJAX so interesting? ...... 11 4.2 Challenges in testing AJAX based web application...... 11 4.3 FSM based web application testing...... 13 4.4 Invariants...... 14 4.5 Crawljax...... 15 4.6 Automatically Testing UI States of Ajax (ATUSA)...... 16 4.6.1 Core functionality ...... 16 4.6.2 Plug-ins...... 18 4.7 Continuous Integration (CI)...... 19
5 Research 21 5.1 Cases study setup and default configuration...... 21 5.1.1 Crawljax - ATUSA default configuration...... 21 5.1.2 Required plug-ins for the replication...... 22 5.2 Case study 1 Introduction...... 23 5.3 Case study 1.A “The Organizer” ...... 23 5.4 Case study 1.B “TaskFreak”...... 26 5.5 Case study 1.C “HitList” ...... 28 5.6 Case study 1.D “The Tunnel”...... 30 5.7 Case Study 2 “Tudu” ...... 30 5.8 Case Study 3 “Finding real bugs”...... 34 5.9 Invariant research...... 38 5.10 Tool Development ...... 42 5.10.1 ATUSA stack reimplementation...... 43 5.10.2 Plug-in implementation required by the replications ...... 46
i 5.10.3 Invariant provider, collector and violation reporting ...... 47 5.10.4 CI integration...... 53 5.11 Case study “Effectiveness of the integration” ...... 54
6 Analysis and Conclusions 59 6.1 Conclusion ...... 59 6.2 Threats to validity...... 60 6.2.1 Internal threats to validity...... 60 6.2.2 External threats to validity...... 61 6.3 Contributions...... 61 6.4 Future Work ...... 62
Bibliography 63
7 Appendix 65 7.1 Questionnaire...... 65
ii Abstract
Manual testing of complex AJAX based web applications can require a significant amount of time and effort. At Guidion the time limitations for manual testing leads to the identification of errors after live deployment of a new release. The web applications of Guidion were used by around 700 users for their daily work, errors in the web applications can impact the work of these users. Automated testing of dynamic AJAX based web applications was researched by the work of Mesbah et.al in [MvDR12] using the open-source crawler Crawljax. This thesis replicates the case studies described in [MvDR12] and evaluates their findings on performance, invariant creation, required manual effort and fault revealing capabilities using Crawljax and the proposed plug-in suite called ATUSA. Furthermore is included a survey research which investigates how web developer can be supported to provide invariants for ATUSA. The results of the research forms the basis for the proposed invariant collector Chrome plug-in and web application. The effectiveness of the ATUSA approach and the invariants delivered from web developers is investigated in the last case study with the ATUSA stack integrated in a Jenkins deployment pipeline at Guidion. Based on the observations the re-implemented and proposed ATUSA plug-in suite is capable to be used for automatic invariant based testing of AJAX web applications. The proposed invariant support integrated in the web developer development process, stimulates and separates the invariant creation process apart from ATUSA. Web developers are able to create invariants without Java, Crawljax or ATUSA knowledge. The integration in a sample Jenkins deployment job enables to trigger the test execution after actual test deployment. The case study reveals that the ATUSA stack integration at Guidion augments the testing phase of the web applications.
iii Preface
This thesis was written as part of the requirements for the Master Software Engineering of the Uni- versity of Amsterdam. The study was conducted at Guidion B.V., in Amsterdam. This thesis would never be possible without the help and support of my supervisors Vadim Zaytsev and Mike Out. Furthermore I would like to thank the whole Guidion developer team for their support and feedback during the research.
Furthermore I would like to thank my family, the love and support I received from my mother Marion Rohde and my twin brother Philip Rohde. Another special thanks goes out to my father Reinhard Rohde, a lost resource of support and geniality R.I.P 2007.
”One of God’s own prototypes. A high-powered mutant of some kind never even considered for mass production. Too weird to live, and too rare to die.” Hunter S. Thompson
iv Chapter 1
Introduction
Interactive “Web-2.0” [Mes09] applications are a well-known part of the world-wide-web. Still more and more known services as well as desktop applications move to the web. This reflects the trend to move applications to the web to provide more flexibility and require less effort from the end user. Reasons for this move to the web are [MvDR12]:
• No installation effort.
• Up to date software, reduction of maintenance and support costs.
• Unbounded access to user data with any device. (Assume Internet Connectivity)
• New ways of communication and collaboration.
• Better chances for an international user audience.
These Web-2.0 applications are wide spread and are an inherent part in the personal and business domain. Popular examples for that are Microsoft Office 3651, the Google Service Stack (with GMail, Google+)2, Atlassian Stack (Jira, Bitbucket, Confluence)3 and Social Media platforms4. The host organization realized this trend and is using different web applications to operate. Besides using existing web applications, develops and maintains the host organization own web applications for supporting their business processes. To provide more flexibility, information sharing and collaboration possibilities, the host organization started to change existing traditional web applications to interactive, responsive and complex web applications. This includes new features and constant refactoring to move existing functionality into the same direction. One of the technologies that supported and enabled this enormous change in the domain of the web is AJAX [MvD07]. AJAX is an abbreviation and stands for “Asynchronous JavaScript and XML” [Gar05]. It enables browsers to provide more functionality than only loading, rendering and navigating through static pages. It enables the browser to provide a more dynamic and interactive user experience through respon- siveness and the use of user interface (UI) components [MvDR12]. These components are also known as partial site updates through asynchronous client-server communication and client-side JavaScript execution. Despite the fact that AJAX positively impacts the whole user experience when brows- ing the web, or using web based applications, it comes with new challenges for the developers. The main challenge is testability of these modern web applications, which stands in the main focus of this thesis. Testability of the AJAX UI components is a challenging task because of the asynchronous, event-driven and stateful nature of AJAX, and of client-side JavaScript based manipulation of the Document Object Model (DOM) [MvDR12][vDM10]. In the host organization testing is a growing part of the development process, and besides functional testing of server-side components, client-side testing and UI testing gain more interest. Through the
1Microsoft Office 365 http://office.microsoft.com/ 2Google website https://google.com 3Atlassian https://atlassian.com/ 4Facebook and Twitter https://facebook.com and https://twitter.com
1 fact that the host organization moves more and more new and existing features to a new and modern user interface with an implied move of server-side functionality to client-side functionality. These web applications support the core business processes and are under a constant and rapid change in the last year from traditional web applications to AJAX based web applications. Testing gets more important for the host organization but a solution is not yet implemented to test the new features, especially on UI side. The host organization needs an automated testing approach after every release, which is not possible at the moment, only new features are manually tested which is a hard and time consuming task. This problem is addressed in the research “Invariant based Automatic Testing of Modern Web Applications” [MvDR12], published by A. Mesbah, A. van Deursen and Danny Roest. It proposes a Java based crawler named Crawljax to support automated AJAX based application testing by crawling the web application’s DOM, and inferring an (finite) event-state-machine by clicking all specified DOM events. The event-state-machine or state flow graph serves as a basis for testing approaches. Additionally to the use of Crawljax, the paper proposes a new technique and tools to use Crawljax for Regression testing of AJAX web applications including invariant based testing. This tool composition is called ATUSA, an abbreviation for “Automatically Testing UI States of Ajax” [MvDR12]. ATUSA provides new core features, for example generic comparators (for detecting relevant UI elements) for generic error detection (mostly DOM related errors), a plug-in API, generic and application specific invariant checking. Additionally it provides plug-ins for visualization and test case generation, which enables automated testing as well as a control mechanism.
Thesis focus One goal of this thesis is to replicate the case studies with ATUSA as described in the initial research. For validation purposes the case studies were replicated in confirmatory form. The findings are used for comparison with the findings of the initial research, for providing more empirical evidence to support the use of Crawljax with the ATUSA extensions. Furthermore, the possibility to create application specific state and state-machine invariants to use with ATUSA is investigated. Creating these invariants is a pure manual approach and requires a understanding of the concept of invariants and Java programming skills, which is also identified as an thread of acceptance in the initial research [MvDR12] as well as in the research of van Deursen et al. [vDM10]. Addressing the problem of the invariant creation provides this thesis a investigation about an easier way to provide application specific invariants for non-academics for ATUSA. The main focus here lies especially on the manual effort to provide those invariants. Additionally a proof of concept is included to deliver these invariants for the automatic web application testing approach. The last goal of this thesis is the integration of ATUSA into an existing development pipeline and a investigation of its fault revealing capabilities and its impact on the initial research questions [MvDR12].
Summarized goals of this thesis
• Replicating the case studies in confirmatory form.
• Conducting an interview research about providing invariants and fault information delivery.
• Creating a proof of concept about invariant creation support and error insights.
• Integrating the tools in deployment pipeline.
• Conducting a research about its effectiveness at the host organization.
1.1 Host organization
This thesis was conducted at Guidion ICT Nederland B.V.5 Guidion is a Dutch provider of a na- tionwide network of technical specialists. These technicians are the field force of Guidion and deliver technical support and services in every region of the Netherlands. The main sectors in which Guidion operates are:6
5Guidion ICT Nederland B.V. will be further referred as Guidion throughout this document 6Information stated in brackets are business use cases examples.
2 • Energy support sector (Smart Energy Devices)
• Telecom support sector (Installations, Technical all-round service)
• Call Center support sector (Appointments, Surveys)
• IT Sector (Computer trouble-shooting)
• Retail support sector (Printer support)
The company delivers service and works for well-known Dutch telecom and energy providers, inter alia for companies such as: UPC7, Eneco8, XS4ALL9 and Telfort10. Guidion uses mainly web based applications to operate and their development and maintenance is for 80% in-house development and the rest is commercial software as well as outsourced development. The field force and the office busi- ness processes rely on these web applications. Uptime and correct functionality of the web application is essential for Guidion and their technicians in the field. For example, the technicians have to log every step they do at the end customers side, including logistics, task and time logging. If one of the tasks is not executable within the web application, the technician needs support from the office, which implies a longer stay at the end customers side, as well as a possible impact on the following appoint- ments of the technicians. The in-house developed web applications are under constant development. The IT department uses an agile development process with an existing deployment pipeline, which enables the developers to deliver new functionality in short development cycles. The IT department consists of inter alia five developers including UI front-end developers who are the main information sources for this thesis.
1.2 Thesis document structure
This thesis document is structured as follows. After this chapter follows the problem explanation, which explores the existing scientific problem and the existing real world problem of the host orga- nization Guidion. The problem section forms the basis to understand the motivation, which is the following section. Right after the motivation section follows the chapter research methods, which introduces the research method used for this thesis. After that follows background section, to provide the required information about Crawljax and ATUSA. The next chapter includes the research with detailed information about the conducted case study replications, the invariant research, the devel- oped proof of concept and the final case study about the effectiveness of the approach. The results and answers to the research questions are presented in the following final chapter of this thesis document, the analysis and conclusion.
7UPC https://upc.nl 8Eneco https://eneco.nl 9XS4ALL https://xs4all.nl 10Telfort https://telfort.nl
3 Chapter 2
Problem statement and motivation
This thesis addresses the challenge to execute automated invariant-based testing of modern web- applications. At the moment client-side testing is executed by a small team of User Interface (UI) designers / developers. Due to a small time budget for testing, the developers execute the tests in a fast manner and without sufficient test reports. This leads to the problem that several bugs are encountered during the use of the application when certain functionality is needed but does not work correctly. This leads to delayed or blocked business processes at the host organizations head-quarter or at the field force side. In this case the development team has to react immediately. These critical bugs can cause delays in the business operation and create extra costs. Those bugs are mostly fixed in a quick manner without further validation of correctness. Guidion has no insight in the current state of the web applications, and has no insight if bugs are really resolved within the next release. Also reproducing bugs is not always possible, through a lack of adequate information delivery to reproduce these bugs. Guidion has no existing basis as test framework, which focuses on user interface related testing. These problems are addressed throughout several research publications from A. Mesbah and A. van Deursen using the Java based open-source crawler Crawljax1.
2.1 Evaluation of the proposed ATUSA approach
Since AJAX was coined in 2005 by J. J. Garret [Gar05], testing of AJAX web applications gave rise to new challenges. These challenges in testing AJAX applications are further described in section 4.2. Mesbah et al. analyzed the testing issues with AJAX web applications and proposed a Java based open-source crawler called Crawljax [MBvD08][MvDL12]. Crawljax uses a web browser to simulate user behaviour through clicking predefined HTML candidate elements. The basic functionality of Crawljax combined with the customizations and extensions provided with ATUSA address these challenges. AJAX is commonly used to create interactive UI elements within a few or even a single web page. The advantage is that AJAX’s asynchronous nature enables the browser to update small DOM elements without the need of reloading the whole web page. Precisely this advantage is used to build UI widgets within a web application, which uses the client-side AJAX functionality to retrieve data from a server and update the document object model (DOM). These client-side DOM manipulations are hard to test in complex web applications, because the state of the web application depends on the user input. Traditional web application testing approaches reaches their limits by testing AJAX in combination with client-side JavaScript based DOM manipulation. The field of interest is here, that all AJAX functions are working correctly and manipulating the DOM in a valid manner. Additionally to testing AJAX, web applications are a fast changing branch in the domain of soft- ware development. The existing testing tools to test AJAX based web applications require manual programming effort. Testing a web application after a new UI design also requires updating the exist-
1Crawljax http://crawljax.com
4 ing test suite. The proposed approach with ATUSA is already two years old but still unique and valid [GMBCM13]. The open source crawler Crawljax evolved since 2012, and was used in later research with resulting improvements [NMvD14]. Mesbah et al. states that more empirical data is required to validate the ATUSA approach as a valid approach for a automated way to test AJAX based web applications. The main focus of the confirmatory case study replication lies on the evaluation of the following factors:
• Obtaining of invariants, and expressing them in ATUSA.
• Measuring fault revealing capabilities and effectiveness.
• Measuring performance and scalability.
• Measuring the level of automation and required manual effort.
An additional problem mentioned by Mesbah et al. is the acceptance of the invariant based testing approach. Obtaining and creating those invariants for web developers without a background in testing can lead to this problem. The invariant creation problem is the focus of the second part of this research and is described in detail in section 5.9.
2.2 Manual testing at Guidion
Testing approaches at Guidion are only implemented in the field of server-side unit testing, only for new and complex features. This covers a few parts of server-side functionality, including server-side implementations for AJAX related calls. These functions are only used within AJAX requests to manipulate the DOM. More than a thousand users use these web applications of Guidion across the Netherlands. The users depend daily on these web applications to do their daily work. Without them, business processes are blocked and the field force technicians are unable to retrieve work relevant information, which can lead in worst-case scenarios to blocked work at the end customers side. Those problems lead besides additional costs to delays in the technicians schedule, frustrated employees, frustrated end users and a bad reputation for Guidion. The company relies on responsive and correctly working web applications. The list below mentions only a few problems occurring through the lack of testing.
• Blocking user support, appointment creation.
• Blocking work logging for field-force technicians.
• Blocking end-user logistics.
• Blocking depot logistics.
• Blocking recruitment or certification processes.
These problems have direct impact on a correct and efficient flow of the core business processes, and every delay has direct impact on unnecessary cost creation. Waiting times and even extra working hours is a recurring outcome of those problems. Guidion is interested in a process to validate the web applications shallow working test, including a proof of concept with the possibility for regression testing of the web application. Furthermore is it interesting for Guidion to get insights in the current state of deployed web applications. Guidion employs besides a group of software developers also different interns. These interns are working on several feature branches as well as creating whole new applications in the scope of their intern-ships. Through the lack of technical support and control have those interns a high level of self-responsibility to deliver maintainable applications. So this implies that the complex web applications are partly unknown for the current UI development team. Most of the interns within the IT department of Guidion are front-end developers. They actively help delivering new features or modernize old ones. The challenge here is to support the UI web developers, which are partly non-academics or academics with a different background, in creating invariants for ATUSA.
5 At the moment Guidion does not have basic testing approach implemented in their development process. A selected group of power users test the new functionalities in staging environment. After approval of the new features they are released. One huge problem is the lack of time reserved for testing within Guidion. The selected group concentrates mostly on the new features and accepts those. An additional validation of business crucial application functionalities are not included or get skipped due to time limitations. In the past, server-side code could be tested within a existing unit testing suite. The development team uses the unit testing framework PHPUnit [Ber05]. The technical change and the shift of server- side to client-side functionality is not the only problem identified within the development department. Several factors have an influence on the delivered code, functionality and its maintainability. Due to a small time budget for testing, the testers execute the tests in a fast manner and without satisfying test reports. This leads to the problem that several bugs are encountered during the work of the user and some can cause blocked business processes. In this case the development team has to react immediately. These critical bugs can cause delays in processing business processes and also extra costs in extra effort which is needed to create a workaround or process data manually. This causes problems within the development team such as:
• UI related quick fixes.
• Current development sprint freeze and stop.
• Need for regression and smoke testing.
• DOM based insights.
To avoid this Guidion currently is establishing its own testing strategy. This is supported through the facilitation of a deployment pipeline in form of a Continuous Integration (CI) system. The system of choice is a Jenkins CI 2 Server [Sma11]. Jenkins is a widely used open-source system with a huge palette of extensions for nearly every testing framework as well as building systems. It is in Guidion’s interest that the approach with ATUSA can be integrated in the deployment pipeline.
Guidion web developers At Guidion a team of UI web developers create the AJAX client-side functionality as well as UI related functionalities. The team consists of two full time web developers and one intern. These UI web developers have an academic background, but with a different focus than software engineers. This means that the use of invariants as well as a basic understanding of tool supported testing is missing. Additionally to the lack of automated testing, these UI web developers have every half year one or even more new interns. Those interns have a high level of responsibility and the features they deliver are integrated into the main web applications after a quick validation of the required functionality. Due to time limitations, the UI web developers are forced to accept those features without an overview about all their UI elements, and DOM related manipulations. These web developers form the target group of this research regarding the invariants research in section 5.9.
2.3 Thesis Motivation
This section describes the motivation behind this thesis and is divided into three parts, every part covers one aspect of the complete motivation.
Replication in confirmatory form to provide empirical evidence The first aspect is the replication of the initial research [MvDR12]. The motivation itself is to validate if the ATUSA approach as described is still applicable and delivers the comparable results.
Basis for invariant based web application testing Guidion currently has no existing testing basis for automated testing of the client-side behaviour of their web applications. Within the scope of the replication of the case studies, the ATUSA approach is theoretically suited to build a basis to
2Jenkins CI, http://jenkins-ci.org/
6 provide automated testing. The motivation here lies on integrating and supporting the test mentality at Guidion with providing a basis for further testing possibilities.
Basis for reporting on encountered errors The ATUSA approach include different plug-ins for example an overview about the crawled web application and a error report plug-in to inform about failed invariants. Especially at Guidion where the applications are maintained by different groups of developers and interns an overview and reporting mechanism deliver insight about the web applications state. The reporting mechanism also establishes a basic reporting manner for Guidion.
7 Chapter 3
Research method
This section describes three different research methods used within this thesis. Every research method introduces their target research questions. In order to replicate and validate the findings of [MvDR12] we have conducted three confirmatory case study replications. In addition to the replication a survey research was conducted to gather information of a group of UI web developers, about their current testing approaches, invariants support and expectations of the integration of ATUSA. Its result formed the basis of the proof of concept to support web developers to provide invariants for ATUSA. Finally the research includes a case study to evaluate the effectiveness of proof of concept integration.
3.1 Replication of case studies in confirmatory form
The replication is divided into three confirmatory case studies. The first case study focuses on the use of invariants and their expression in ATUSA. The second case study evaluates the fault revealing capabilities and scalability of ATUSA. The third case study is applied to a Guidion web application, to evaluate the ATUSA approach in an active development process. Every single case study includes a measurement about the manual effort required to configure and setup ATUSA. The replication covers the following research questions of this thesis.
• RQ1 - What kind of invariants can be obtained for AJAX applications and how can they be expressed in ATUSA?
• RQ2 - What is the fault revealing capability and effectiveness of the approach?
• RQ3 - What is the performance of the proposed approach, and how well does it scale?
• RQ4 - What is the automation level and how much manual effort is involved in the testing process?
The conducted case studies were set up following the guidelines of Yin [Yin02]. because Mesbah et al. also followed those guidelines.
Confirmatory case studies 1.A - 1.D : RQ1 In order to validate the results on RQ1, the same setup, open-source web applications and techniques are used. According to [MvDR12] four different web applications were investigated, to create application specific invariants. The experiments per web application use the same invariants, and the same configuration for ATUSA as specified in [MvDR12] as well as described in earlier work [RMvD10]. Every case study has its own scenario, which is also included to provide a complete confirmatory case study replication. Besides identification and the possible expression of invariants in ATUSA, a measurement of the required manual effort per experiment is included, to provide complete and comparable findings.
Confirmatory case study 2 : RQ2 - RQ4 The second part of the confirmatory case study focused on the research questions RQ2 - RQ4. It follows the description of the initial research to replicate the case study with the open-source web application Tudu 5.7.
8 Confirmatory case study 3 : RQ2 and RQ4 The third part focused on all four research questions RQ1 - RQ4. In this experiment one real world web application is used to find real bugs in a active development cycle. This experiment uses a web application of Guidion as test system subject.
Hypothesis: RQ1 The validation and creation of the proposed invariants from the original research [MvDR12] identify the same errors by testing the web applications with ATUSA.
Hypothesis: RQ2 The implementation of the proposed test scenarios with their ATUSA configu- ration, identify the same or different errors as in the initial research. The term effectiveness is in the case of this research an objective and commonly used indicator of the quality of a test suite [MvDR12].
Hypothesis: RQ3 The improved version of Crawljax and ATUSA provides a better performance with respect to the case study replications as in the original research [MvDR12].
Hypothesis: RQ4 The detailed study of Crawljax and ATUSA requires a manual effort for the implementation of the proposed invariants and plug-ins which is comparable with the reported manual effort from the initial research [MvDR12].
Reporting on Replication The replication follows the proposal of Carver [Car10] on reporting on replications. This ensures that the confirmatory case studies report in the manner, level of detail and in a comparable form.
Validation of results The validation of the results from the confirmatory case studies, are com- pared with the results from the initial research [MvDR12]. The findings from the first part targeting RQ1, are compared with the findings from the initial research. This part focused on the invariant configuration of ATUSA. Additionally the findings from the initial research [MvDR12] on page 47 in section 8.2.6 are compared. For validating the findings of the experiment targeting RQ2 - RQ4 the findings are compared with the findings from the initial research [MvDR12] stated on page 48 in section 8.3.3, and for the second part the case study results from the real bugs case study are compared on their similarities in the findings.
3.2 Web developer survey research
Information gathering about the developed web application is the second part of this research. To collect evidence about the current testing habits within Guidion as well as development tools, a survey study in form of a questionnaire is conducted. The questionnaire is used to collect different information about the web developers development tools, their testing tools, knowledge about invariants, about mapping events as well as used JavaScript Frameworks for the current web application development and also their expectations of ATUSA’s integration. The main focus lies here to determine where web developers can be supported to create application specific invariants. The web developer research survey and the invariants research cover the research question:
• RQ5 - How can web developers be supported to create application specific invariants for ATUSA?
The results of the survey research are used to propose a application to support invariant creation for ATUSA.
Hypothesis: RQ5 The separation of the invariant expression for ATUSA from the Java based configuration via a mechanism integrated in the web developers test process, enables the web developer to create application specific invariants for ATUSA.
9 3.3 Case Study: Effectiveness of the integration of ATUSA
The final conducted experiment of this thesis is a case study about the effectiveness of the integration of ATUSA at Guidion. For this cases study, ATUSA will be integrated in a existing deployment cycle, and tests one web application of Guidion. In addition to show the crawl results and a possible historical evolution of the web application, a useful error report is delivered and information to support the invariant creation. For this case study, the integration fault revealing capabilities of ATUSA are evaluated. The web developer research survey and the invariants research cover the research question:
• RQ6 - How effective is the integration of ATUSA at Guidion?
Hypothesis: RQ6 The usage of ATUSA in a Jenkins job on a Guidion web application delivers otherwise unavailable insights in the web application and can be used as additional testing phase, which tests the web application before user acceptance testing phase.
3.4 Summary
This section provides a complete list of all research questions, which are covered in this thesis.
• RQ1 - What kind of invariants can be obtained for AJAX applications and how can they be expressed in ATUSA?
• RQ2 - What is the fault revealing capability and effectiveness of the approach?
• RQ3 - What is the performance of the proposed approach, and how well does it scale?
• RQ4 - What is the automation level and how much manual effort is involved in the testing process?
• RQ5 - How can web developers be supported to create application specific invariants for ATUSA?
• RQ6 - How effective is the integration of ATUSA at Guidion?
10 Chapter 4
Background and Context
This thesis project is based on the research by Mesbah et al. [MvDR12]. This research proposes a technique and a tool implementation for automatically testing JavaScript-based AJAX web applica- tions. The proposed testing approach focusses on testing dynamic User Interface (UI) states within a JavaScript-based AJAX web application. The goal of this section is to offer a basic background which is required to understand the replication and the contributions of this thesis.
4.1 Why is AJAX so interesting?
AJAX is interesting through the fact that it introduces so many new dynamic client-side possibilities in domain where dynamism was limited on the server-side implementation of web applications. It introduces the creation of the so called Web-2.0 [MvD07], which is basically defined as interactive web applications with user generated content. In addition AJAX also introduced Web User Interface (UI) widgets. This can be seen as widgets from the GUI application development. For example, a table component can be referenced and updated without updating or forcing a reload of the whole application. Also the browser takes care that the whole web application stays responsive until the table gets updated within the DOM1 and the result is rendered. AJAX enables new user interface features and improves the user experience, but it also introduces new challenges. For example, use of the browsers back button, to navigate to the last state does result in another state than expected. Getting indexed by a search engine crawler. In addition to those examples, the developer needs to take care of the DOM manipulation, which includes also a DOM clean-up after disabling visibility. A common error here is that the DOM grows unexpected as result of a missing DOM clean-ups.
4.2 Challenges in testing AJAX based web application
AJAX introduces also new challenges for testing. Especially automated testing is challenging through the following aspects. The research of A. Mesbah and A. van Deursen identifies the Reach, Trigger and Propagate challenges which complicate testing of JavaScript-based AJAX web application [MvD09]. A. Mesbah and A. v. Deursen identified in [vDM10] additional issues in testing of AJAX web appli- cations. Further research in this field conducted by A. Mesbah and A. van Deursen with help of S. Lenselink and D. Roest identified different challenges [RMvD10][MvDL12][Roe10][MvDR12]. This section introduces those challenges, issues and problems identified, and lists a summary of them with a detailed description per challenge.
Reaching states In traditional web applications different states of the web application are reachable through a unique URL, including possible HTTP GET parameters. For reaching AJAX states a unique URL is not enough. Those dynamic AJAX states are hidden behind JavaScript events included in the DOM. This means to reach AJAX states those events need to be identified during the crawling
1W3C Document Object Model http://www.w3.org/DOM/
11 process. Testing these states, the crawler needs besides identifying those states a mechanism to execute the AJAX events, which can lead to another state. In addition to the reaching problem, another problem connected to the reaching state problem is the problem of bookmarking and sharing the web application page. Through the fact that the web application has only a few different URLs, creating and sharing bookmarks is a additional problem here. Another problem is back-tracking, which is further described in paragraph 4.2.
Triggering events The identified events of a web application state need to be triggered/executed within a test environment to validate, if the event results in a new state. Not only triggering those events is important in this phase. AJAX states are not only bounded to an event, also user input needs to be considered here. User input can result in a different state, even if the same event was triggered. Before triggering an identified event, data input entry points need to be identified and data needs to be inserted before triggering.
Propagate result After executing the possible data inputs and events the result needs to be prop- agated. This results by AJAX in a client-side DOM manipulation. The manipulated DOM needs to investigated and compared with given states to determine if a new state was reached. Additionally to the new state identification the JavaScript errors need to be identified, to see if the new state does not only results of missing or incorrect implemented JavaScript event. The propagated result is the basis for invariant or condition based validation.
Asynchronous nature The browser uses a AJAX engine, which executes asynchronous Xml- HTTPRequest to request new information from the server. This gets executed in a asynchronous way, which means that the user can use the browser and can click on other elements on the same page, and the browser will work further without freezing an waiting for the AJAX response. Test- ing these asynchronous calls the testing approach needs to wait on the response of the server and the AJAX resulting DOM manipulation, to test the propagated state against the old state for state identification described in paragraph 4.2. In addition to waiting for the response, the wait time is important to determine if another candidate element event can be triggered without receiving other asynchronous AJAX responses. This makes event identification and result comparison of the DOM more complex.
Stateful behaviour Another challenge in deriving AJAX states is that the client-side gets more stateful. The switch from server-side to client-side functionality includes also a switch to investigate and store the state-full behaviour from the DOM states. This is especially interesting for deriving test cases from the crawled web application. The stateful behaviour needs to be reproduced within a stored test case to reproduce the same state results in the test case execution phase. The newly crawled states need to base on the same user behaviour, which is the navigation path as well as the used input values.
Backtracking The identified fact that the client-side gets more stateful, consistent back-tracking is a known problem in current AJAX web applications.[MvDR12] To test back-tracking collected states need to be bidirectional navigable and the states need to be stored and compared. This means that the target state needs to have an edge back to the source state. As briefly described before in section 4.1 the back-tracking problem is a known downside of AJAX web applications, and the test framework needs to support comparing the states if they support correct back-tracking for the crawled web application.
Correct DOM-tree management A additional problem through the use of AJAX in web appli- cations, is the correct DOM-tree management. This means that AJAX is used to make certain DOM elements visible, or add and remove elements. To avoid a growing DOM tree and a slower browsing experience, the DOM elements added through AJAX need to be removed correctly.
12 The oracle problem The oracle problem was identified in the research “Regression Testing Ajax Applications: Coping with Dynamism [RMvD10]”. The oracle problem states that fault positives occur in state comparison and its resulting creation. This means that states identified as two sep- arate states are the same. A example here is a date component on a web page, which shows the date and current time with a running clock. Comparing those states without stripping the textual information would result in additional states. To avoid additional and unnecessary state creation, oracle comparators are proposed by the research [RMvD10] to strip all unnecessary state information before actual state comparison. The oracle comparators proposed in [RMvD10] avoids also the state explosion problem. This problem is further described in “Avoiding state explosion problem of gen- erated AJAX web application state-machine using BDD [AS13]” which also proposes an additional approach to avoid state explosion in generating finite state-machines through crawling AJAX based web applications. The problem of state explosion does not receive further attention in this thesis, but could be interesting for further research.
State correctness: DOM validation The dynamic states are hard to test for validity. This means the check if the state DOM is a valid Document Object Model. The identified and collected states need to be validated on correctness. The Document Object Model relies on its corresponding HTML standard and can be validated against its specification. To check for state and DOM correctness the state DOM needs to be validated as valid Document and needs to be persisted for further validation.
State correctness: Invariants To test state correctness during the crawling phase the initial research [MvDR12] proposes application specific invariants as a additional mechanism to test DOM validity. Those invariants can be used to check if a DOM of a state is also valid to its application specific DOM specifications. Invariants and application specific invariants are further described in detail in section 4.4.
4.3 FSM based web application testing
Finite state-machine (FSM) based testing The state based testing approach was initially pub- lished by Marchetto et al. in “A case study-based comparison of web testing techniques applied to AJAX web applications [MRT08]”. This approach uses traces of the web application to infer a fi- nite state-machine (FSM). The FSM is redefined by the tester and further used as basis to generate test cases with help of sequences of “semantically interacting events [MvDR12]”. This approach also involves additional manual effort for redefining the FSM by the tester.
FSM - Invariant based web application testing The testing approach of the initial research is the first automated attempt to test AJAX based web applications. It uses a crawler to infer a FSM based on the approach explained above in paragraph 4.3. In the FSM are the different states the nodes and the events fired the edges, and it is the basis for further test generation.
Collecting AJAX states through crawling Crawling the web application using Crawljax and deriving a finite state-machine (FSM, state flow graph) from a web application was initially proposed in [MBvD08]. Based on the research [MBvD08] the authors released their crawler tool implementation “Crawljax”. Crawljax’s implementation and its crawling algorithm was improved through further research conducted by A. Mesbah and A. v. Deursen [MvD09], [RMvD10], [MvDL12], the initial research [MvDR12] and one research after publication of the initial research [NMvD14]. Crawljax uses the WebDriver API, which enables the use and control of the WebDriver browser interface. The WebDriver browser interface enables Crawljax to crawl the application with a full browser support. The browser support enables Crawljax to execute JavaScript related events.
Conclusion - Web application testing Web application testing has evolved to support mod- ern web applications. The host organization realizes the need of a automated testing approach to
13 reduce manual testing effort and to introduce a static test basis. The initial research and the Crawl- jax enhancement with ATUSA [MvDR12], provides a promising approach in testing modern web applications [AS12].
4.4 Invariants
Using invariants for validating programming functionality is well-known in the domain of software testing. In the domain of web application testing, invariant based testing can be understood as condition on validating the DOM of certain or all states. Traditional testing approaches also uses invariants for example checking on a functional level if pre or/and post conditions of a function are still valid. A simple example is in a unit testing framework such as PHPUnit the Assert statement to check with invariants. Assert checks in this case if the function still produces correct results or handles correct errors, and no side effects of the function can occur. Invariants for ATUSA are also named conditions, these conditions must be satisfied after a DOM manipulation otherwise the invariant is violated. The initial research [MvDR12] divides invariants for ATUSA into four types. These four types are generic state invariants, application specific state invariants, generic state-machine invariants and application specific state-machine invariants. Generic state invariants can be applied on every state in the state flow graph. These generic state invariants are validated with every new state and need to be valid for each inferred state of the crawled web application. Application specific invariants need to be specified by the developer or the tester. Those invariants are bounded to one specific web application, and need to be manually adjusted when the web application and its DOM structure changes. Generic state-machine invariants are invariants on the inferred state-machine, where every state in the inferred state flow graph includes a certain transition. The back-tracking invariant is a generic state-machine invariant. The last type of invariants are the application specific state-machine invariants. Those invariants need also be implemented by the tester as a plug-in which checks the inferred state flow graph includes certain transitions, e.g every state with a logout button has a transition to the log-out state. Every type is explained in detail in the sections below. In addition to the types of invariant proposed in [MvDR12] and implemented for ATUSA, this section further describes the current research status of collecting or automatically inferring invariants.
State-machine and state invariants Mesbah et al. describes two main sorts of invariants which can be used for testing. These sorts are state invariants and state-machine invariants. Both sorts consist of generic and application specific invariants. The difference is that state invariants are used to test every new state and the state-machine invariants, are used to validate the inferred state-machine against certain constraints. Both are described in the sections below. At first the state invariants for DOM testing and then the state-machine invariants for validating against the inferred state flow graph (or FSM).
Generic state invariants The generic state invariants are invariants, which should be valid for every inferred state from the web application. The initial research states here two invariants, which are applied on every state in ATUSA. Dead Clickables and Back Button Functionality (back- tracking). In addition to those generic invariants used by ATUSA, generic invariants can also be used e.g. for error message detection. Every new state can be checked if it includes server side generated error messages. Furthermore for W3C HTML validation, detecting JavaScript errors and Image reference errors. More information about generic invariants and their implementation as well as use in ATUSA can be found in section 5.9.
Application specific state invariants Application specific invariants are invariants specific to test the states of a special web application. Those invariants need to be specified manually by the developer or tester. The initial research mainly uses application specific invariants to add constraints on the DOM of the case study web applications [MvDR12]. Those invariants need to be implemented in Java and requires a general knowledge of Java and a Java friendly development environment. Mesbah et al. proposes different ways to express application specific invariants. More information
14 about the expression, implementation and the use of application specific invariants in ATUSA can be found in section 4.5 and section 4.6.
Invariants, conditions and pre-conditions The invariants are based on condition, which need to be satisfied for every state or just a certain set of states. The set of states on which the invariants apply can be customized within Crawljax with the ability to specify pre-conditions. These pre-conditions need to be satisfied before a application specific state invariant is used as a DOM constraint.
Generic state-machine invariants Generic state-machine invariants can be used to test against the inferred state flow graph (FSM) and its transitions. This offers the ability to test the state flow graph in its whole, and for certain transition constraints which need to be satisfied for the whole inferred state flow graph. Generic state-machine invariants are for example invariants to check the back-tracking functionality, which means that every state in the state flow graph should have a transition back to its source states. This generic state machine invariant can be checked against the whole state flow graph. Another example proposed by the initial research is to check the state flow graph for dead clickables, which means that for every state identified events are checked if they produce a error on execution.
Application specific state-machine invariants Application specific state-machine invariants can be created and applied by the developer or tester to check the state-machine of a web application on application specific constraints. Pressing a logout button in the web application must end in the login state, and this constraints need to be satisfied for every state where a logout button is visible. Those constraints are expressed in a “Source” → “Action” → “Target” manner [MvDR12], and validates transitions in the state flow graph.
Related work in deriving state invariants Inferring invariants automatically is still an open research topic. In the domain of invariant derivation, K. Pattabiraman and B. Zorn proposed a tool called DoDOM [PZ10] which is able to derive invariants automatically from a test system. DoDOM records user interactions and events and repeats these executions in different sequences and captures the DOM. DoDOM repeats these events six times to gain a good invariant coverage. DoDOM consist of three components a proxy, a JavaScript executor including logging and a invariant extractor. The proxy captures the different DOMs and events and the JavaScript logger executes them in the browser. The Invariant extractor infers the different DOM invariants. DoDOM comes with several limitations and trade off’s, one limitation to mention is that DoDOM is in the proposed tool only applicable for single page web applications, and one trade off is the execution of six times. DoDOM provides a promising approach to infer invariants, but is not directly applicable for this thesis, because DoDOM is mostly implemented in JavaScript and its source is as a open-source project public available. The implementation in ATUSA was out of scope for this thesis. Furthermore the approach requires in average six crawling executions to reach 100% coverage of state invariants. To execute ATUSA six times can require a long time when the test subject system consist of a large set of different states. Another approach to automatically infer invariants was proposed by Groeneveld, Mesbah and van Deursen in [GMvD10]. They proposed a plug-in for Crawljax which is able to infer invariants within several crawling runs. It uses the proposed tool of [EPG+07] called Daikon to infer DOM and JavaScript invariants. Groeneveld et al. released this plug-in composition for Crawljax version 1.9. Unfortunately the plug-in was not further maintained, and bringing it in a useful state for this thesis was out of scope. The scope of the research is to provide a mechanism to express application specific invariants without any Java knowledge, so that web developers and designer can also provide invariants to use during the use of Crawljax within Guidion.
4.5 Crawljax
The automatic testing approach proposed in [MvDR12] is based on the Java open-source crawler Crawljax [Cra14]. Crawljax is a crawler application targeting JavaScript based AJAX web appli-
15 cations, and automatically infers a state flow graph of the crawled web application. Crawljax was initially proposed by A. Mesbah, E. Bozdag and A. v. Deursen in the research about crawling AJAX web applications and inferring a finite-state-machine [MBvD08]. The approach used to infer a finite state-machine was initially proposed by [MBvD08]. In addition to automatically inferring a state flow graph during the crawling process, Crawljax fully supports client-side JavaScript execution. Its approach was further applied and improved in additional work by A. Mesbah et al. in [BMvD09], [Roe10], [RMvD10], [MvDL12], [MvDR12] and [NMvD14]. Those publications improved Crawljax, and also a small but active user community, who uses, tests, reports and contributes improved Crawl- jax. The version used for this thesis is Crawljax version 3.5.1 [Cra14], which was at begin of this thesis the latest stable release.
State flow graph implementation notes The current version of Crawljax [Cra14] uses a open- source library called JGraphT for the state flow graph implementation. The state flow graph itself is implemented as a directed graph. JGraphT supports state and edge creation, relation creation as well as graph related functions and algorithms. A supported algorithm is the K-shortest-path algorithm, which is used to determine all sinks in the inferred state flow graph for the test case generation. For more information about JGraphT and its implementation details please have a look at the vendors website [JGr], or its implementation in Crawljax [Cra14] in the source pack- age “com.crawljax.core.state”.
Browser supported crawling An advantage of Crawljax is the use of a browser instance for crawling, simulating user behaviour and execute JavaScript AJAX functionality. Crawljax uses the WebDriver API, which offers the ability to control natively the browser. The WebDriver API is a fully object-oriented Java library and is also used as basis in and since the Selenium-2.0 2 web application test framework release, which substitutes the JavaScript based approach of Selenium-1.0 to execute user behaviour in the browser. In addition to the natively supported browser control, does the WebDriver API supports several different browsers. Crawljax uses it to crawl the target web application and to execute client-side AJAX functionality. The WebDriver API is a well-known Java library, and now part of the Selenium project.
Crawljax - Console interface (CLI) In addition to include Crawljax as a library in a Java application, it comes with a console interface [Cra13], which offers global configuration parameter to specify the crawl behaviour via the console. This enables the user, to use the tool right away without any extra effort. The console interface is predestined to be used within a continuous integration systems.
4.6 Automatically Testing UI States of Ajax (ATUSA)
This section describes the approach proposed in the initial research called “Automatically Testing UI States of AJAX” also called ATUSA [MvDR12]. ATUSA is based on Crawljax and introduces besides improvements and features at the Crawljax core implementation also additional plug-ins targeting the challenge to automatically test AJAX based web applications.
4.6.1 Core functionality The features which improved the Crawljax core functionality are explained in this section.
Oracle comparators Oracle comparators in ATUSA are used to compare, determine difference or strip non-deterministic DOM elements before a states DOM gets further validated, compared and scanned. They target the oracle problem challenge in testing AJAX applications described in section 4.2. Using oracle comparators was at first introduced by [Roe10] and [RMvD10] and further used in the ATUSA stack in [MvDR12]. ATUSA provides a interface called “Comparator” which defines the following three methods: 2Selenium HQ website http://docs.seleniumhq.org/
16 • isEquivalent, validates the old state DOM with the the new state DOM.
• getDifferences, lists of the DOM differences, when equivalence check fails.
• normalize, strips non-deterministic or other DOM overhead.
The “Comparator” interface must be used when implementing own comparators, or the abstract “AbstractComparator” class can be extended. It provides default implementations of the comparator methods. Furthermore then offering the use of oracle comparators, ATUSA introduces a number of generic comparators, which are useful to ignore differences, strip DOM overhead before checking for DOM equivalence.
Generic comparators The generic comparators listed in the initial research include the following comparators, a White-space, Attribute, Style, Plain structure, List, Table, Regex (Regu- lar expressions) and XPath comparator. For example the provided plain structure comparator removes all element attributes, line breaks and the content before the DOM is used for compari- son. Since the publication of [MvDR12] and of ATUSA in 2012, Crawljax has improved but not the provided generic comparators.
Oracle Comparator pipeline The defined oracle comparators get pipelined in ATUSA, which means that every comparator passes its normalized result further to next comparator to eliminate as many differences as possible before state comparison. Further details about the comparator pipeline can be found in [MvDR12].
Oracle Comparator change The new version of Crawljax, version 4.0 3 will substitute the oracle comparators, with the so called DOM strippers. These DOM strippers have the same goal than the oracle comparators, but offer a simplified way to provide them. This will be achieved with the use of JSoup, a new Java HTML parser engine, which provides DOM element selection, traversal and manipulation as the JavaScript library JQuery4. Furthermore are the DOM strippers fully thread safe, and can be used with several browser instances. Their implementation just started during creation of this thesis and therefore not further included in the conducted case studies of this research.
Invariants The initial research [MvDR12] introduces with ATUSA the ability to use Crawljax for invariant based testing. With the use of invariants the DOM state or state-machine transitions can be validated against defined constraints. The invariants in ATUSA are distinguished into state invariants and state-machine invariants. Both variants offer a generic and application specific scope. Besides the scope lies the difference also in the manual effort required to implement those invariants in ATUSA. The section below describes the implementation of the state invariants, followed by a description about the state-machine invariants. For further information about invariants, their definitions and description about their goal are explained in section 4.4.
State Invariants ATUSA supports state invariants expression in two types, generic state invari- ants and application specific invariants. The difference between those two types is the scope of the invariants applicability, because some invariants can be in general used for all web applications and some are application specific. This describes the first type of state invariants and is defined as generic state invariants [MvDR12]. These invariants are useful to validate against every new detected state. The initial research describes the use of generic state invariants as oracles [MvDR12] which need to be satisfied on every new detected state. E.g. W3C HTML validation, which validates every state against the W3C HTML standard or Error message detection, to detect server-side produced error messages in the states DOM. The second type of state invariants in ATUSA are defined as application specific state invariants. These invariants include application specific DOM constraints targeting a specific web application in development. The application specific state invariants internally use the different generic conditions as constraint over the DOM. The initial research proposes five different
3Crawljax version 4.0 https://github.com/crawljax/crawljax/tree/crawljax-4.0 4JQuery http:/jquery.com
17 generic conditions inclusive logical operators for more flexibility over defining conditions, and these are Regular expression condition, XPath condition, JavaScript condition, URL condition and Visible condition. These conditions are not bounded to the use of application specific invari- ants, these invariants can change more often than generic state invariants. Considering the example, a invariant for a application specific table structure including content validation compared with the changeability of the W3C HTML standard. The initial research does not state any example or imple- mentation detail of these generic state invariants. These invariants need to be implemented in Java inheriting the Crawljax invariant base class, or for several use cases also via the plug-in system. For example a PostCrawlPlugin which could be used as generic state invariant to validate the states and their corresponding DOM-trees from the inherited state flow graph.
Invariant Error reporting - Implementation details Invariant violation reporting is supported by ATUSA’s plug-in system and plug-in hook API. The plug-in system with API is described in section 4.6.2. It offers the extensibility to customize error collection and crawl behaviour on invariant violations. A default invariant reporting mechanism is described in ATUSA’s error reporting plug-in in section 4.6.2.
State-machine Invariants State-machine invariants are distinguished between generic and appli- cation specific state-machine invariants as defined in 4.4. The main difference compared to the state invariants is that state-machine invariants are constraints on the complete inferred state-machine after the crawling phase. Two examples of generic state-machine invariants are detecting dead clickables, a validation over all states if all candidate elements got successfully fired and backtracking validation, a validation over all states if all edges to other states are bidirectional. These two state-machine invariants target two challenges in testing AJAX web applications described in section 4.2. In general these invariants follow the pattern “Source → Action → Target [MvDR12]”. State- machine related invariants require a implementation as plug-in, because the condition based invariant object and its generic conditions does not support state-machine validations. Furthermore are state- machine suited for validations after complete creation of the state flow graph. The initial research does not provide further information about the implementation of the state-machine invariants used for conducting the case studies [MvDR12].
4.6.2 Plug-ins The initial research proposed additionally a plug-in system with several plug-in hooks, to enable the plug-in development and provide additional features for Crawljax within the crawling phase. The plug- in API provides several entry points as plug-in types. These plug-in types are: OnFireEventFailed- Plugin, OnInvariantViolationPlugin, OnNewStatePlugin, OnRevisitStatePlugin, OnUrl- LoadPlugin, PostCrawlingPlugin, PreCrawlingPlugin, PreStateCrawlingPlugin. These plug-in types are used by the required plug-ins of ATUSA and are described in this section.
Overview plug-in Generates a overview report of the crawled web applications resulting state-flow graph, to validate crawling result for possible further adjustments to improve state coverage. The overview plug-in generates an HTML report with a graphical representation of the state-flow graph. In addition to the graphical representation it also displays detailed information for every state in the graph. Those information includes the state DOM, all detected candidate elements, a screen-shot including indication of detected and clicked candidate elements, the XPath of the detected candidate elements, simple edge statistics and a overview over the used Crawljax configuration for producing the overview output.
Testing Framework plug-in The test framework is one of the key plug-ins of the research. Sim- ply without this plug-in the whole automated testing approach of ATUSA would be missing. The framework provides a plug-in for test suite generation, a test execution controller for executing the
18 test suite and collecting test results and a error report plug-in. The error report plug-in collects the test result and generates a HTML report.
Test-suite generation For generating test-cases ATUSA uses the K-Shortest Path algorithm, this is a generalized version of the shortest-path problem. Which means where several path’s through the graph are thought with increasing order. ATUSA uses all sink nodes from the inferred state flow graph to search for the K-Shortest path from index node (State 0) to the sink nodes. In the initial research paper a short statement is given that loops in the state flow graph are included once. With help of Apache Velocity5, Java JUnit6 test cases are generated. Each test case includes the sequence of events from the index state to the sink state. The sequences include all events used to navigate from the index state to the target state. The test generation process is briefly discussed in [MvDR12], but it is a follow up research, on an earlier publication of A. Mesbah, A. van Deursen and D. Roest. The publication about regression testing [Roe10] and the pre created master thesis of D. Roest [Roe10] describe the test framework in more details.
Test-suite execution The initial research paper proposes with ATUSA also a solution to execute the generated test cases, without using an additional technique. The generated test cases make use of a test execution controller, which includes a embedded browser to navigate through the generated test cases. This means the browser navigates with the test case the new web application and tests its state with the test state. The test execution follows the specification in [MvDR12].
Test reporting Furthermore a test report controller generate a test report including failure and success information about the test execution. It also includes detailed information about, condition check results, invariant validation results and state differences. The test report is essential to see details about the actual error, the JUnit test report does only includes a global test method fail message as described in [Roe10].
4.7 Continuous Integration (CI)
Guidion is currently implementing its own testing environment. This environment is driven by a continuous integration system (CI ). CI enables that the software which is currently developed is continuously build after someone commits a change. Especially in large software projects developers test their own changes but not the whole application. Bringing the whole application in the state to use requires to much effort. CI helps here in automating testing, building and deploying the software to a testing or staging environment [HF10]. Guidion uses as CI system the open-source CI server Jenkins 7. The CI system builds the management system were the ATUSA stack should be implemented. At the moment of the thesis is one deployment pipeline implemented in Jenkins of Guidion’s web applications. The future vision here is that also the other web applications will follow and for each deployment pipeline a ATUSA testing phase exists. [HF10] states three preconditions before starting to integrate a software project into a CI system. 1. Use of a version control system. Guidion uses Bitbucket8 as Git version control system. 2. Use of an automated build. Most web applications of Guidion use a automated build. 3. Agreement of the team. The agreement of the team is asked within the survey research.
Jenkins Continuous integration server Jenkins is a full featured continuous integration server. It is a Java open-source CI server, with a wide variety of plug-ins, which can be used to create jobs to build, test and release a software project. It supports testing and building software continuously including monitoring and reporting. At Guidion the Jenkins server is installed on a own virtual server, and is fully customizable for integrating ATUSA. 5Apache Velocity http://velocity.apache.org/ 6JUnit http://junit.org/ 7Jenkins CI http://jenkins-ci.org/ 8Atlassian Bitbucket https://bitbucket.org/
19 Deployment pipeline A deployment pipeline is a manifested automated process to bring the software from the version control to a testing or staging environment. It include several steps to build, test and deploy the software project [HF10]. At Guidion the deployment pipeline consists of a few building steps and a few unit tests to deploy a system to a web server. The plug-in system of Jenkins provides additional support for reporting and uploading the build files to a test environment. After the web application is deployed to a test environment, it is fully prepared to get tested by the user. After the successful deployment to the test environment is the entry point of the ATUSA approach. This can help identifying errors before a group of tester validates the changes. The validation process of the group of tester can be saved when the ATUSA approach successfully test the web applications over a longer time period.
20 Chapter 5
Research
The replication of the case studies from the initial research [MvDR12] is divided into five sections. The first section provides a default configuration declaration of Crawljax / ATUSA. Followed by the first replication case study answering research question RQ1. The third section includes the case study replication answering research question RQ2 - RQ3. The following section includes the replication case study answering RQ1 - RQ4. Every replication section evaluates the correctness of the case studies described in the original values, and compares the findings of the replication with the findings of its original. The last section summarizes the replication findings and ends in a conclusion.
Goal of the case study replications The replications goal is to support Mesbah et al. with providing additional evidence on using the ATUSA approach for automatically testing AJAX based web applications. Additionally, it evaluates the described case study scenarios and the use and implementation of the proposed invariants, comparators and plug-ins. Its findings are compared with the findings of the initial research [MvDR12].
Additional Tools used for the replication For the evaluation and implementation of the repli- cation case studies additional tools were used. These tools are further explained in the section 5.10.3. In addition to these tools the Netbeans1 IDE was used as main development environment for imple- menting all required features.
5.1 Cases study setup and default configuration
The replication case study setup followed the described setup from the initial research. In general are two main runs were executed per replication subject system. One in default setup, with additional click elements, and the second run including the created invariants, comparators and plug-ins. Every main run is further executed two times with different browsers.
5.1.1 Crawljax - ATUSA default configuration The initial research uses Crawljax with its default configuration. The default configuration is not further described. The Crawljax evolution, especially [NMvD14] lead to a change in the default configuration which produces different replication results, and invariants e.g. the application specific state-machine invariants always fail. This is because the default activated clickOnce setting limits the state creation, through clicking identified elements only once. Setting the clickOnce to false solves this problem. Another identified key configuration is the default crawl strategy, which is defined as clickElementsInRandomOrder and set to false. Setting the clickElementsInRandomOrder to true produces different state coverage results, and is further described in paragraph 5.1.1. Through the fact that the default configuration was not further described within the initial research, the
1Netbeans https://netbeans.org/
21 configuration mentioned below was determined as default configuration during testing and running Crawljax with different settings before conducting the actual replication.
• Waiting time after event. 500 Milliseconds
• Waiting time after reload. 500 Milliseconds
• Maximum crawling time was set to 30 Minutes (If not otherwise specified in the initial re- search).
• Clicking Default elements enabled (A, DIV and INPUT elements).
• The overview plug-in is used for all the case studies.
• Maximum states is set to unlimited.
• Maximum crawling depth is set to 2.
• Random input generator is deactivated (false).
• Browser execution instances is always set to 1 to prevent multi-threading side effects through the use of the not thread safe oracle comparators.
Reliability of the use of different crawling strategies Crawljax can be configured to use two crawling strategies. The default crawling strategy is the breadth-first strategy, where identified clickables are executed in the order they are found in the states DOM. The problem of this crawling strategy in Crawljax is that, if a clickable leads to a state deeper than the allowed maximum depth, the clickable is skipped and the following identified clickables of the state also. This is already a known bug in Crawljax and will be fixed with the upcoming new version 4.0. Another crawling strategy is the random equivalent, it executes the clickables in a random order, and validates every clickable for itself to which depth it leads. This leads to different state coverage results. To reduce the state creation the default crawling strategy is the breadth-first strategy.
Default oracle comparator: simple comparator If no oracle comparator is specified, the de- fault SimpleComparator is used during the replication. This comparator removes white-spaces and line breaks, before comparing on state difference. This is the simplest form of an generic oracle comparator within Crawljax and always the default comparator if not otherwise specified. For devel- oping additional oracle comparators, is the provided Crawljax ApiDoc2 misleading in the aspect of the oracle comparators. The oracle comparator uses the stripped browser DOM of Crawljax which converts HTML elements in upper case.
Browsers used for the replication Some of the case studies use several browsers for execution, to compare their test results. The case study replications make use of three browser types. Firefox, Chrome and PhantomJS.
Default overview plug-in The overview plug-in provided from the Crawljax plug-in repository is used to generate the overview graph for candidate inspection and invariant search.
5.1.2 Required plug-ins for the replication The replication required beside the ATUSA plug-ins such as the TestSuiteGenerator, Error reporting and the Overview plug-in, several additional plug-ins. These plug-ins were not available within the Crawljax Source code repository. The test case generator plug-in [Cra12b] and error report plug-in [Cra12a] could be only found for Crawljax version 2.2 and both needed to be re-implemented to fit the new Crawljax architecture. For more details please have a look in section 5.10.1. These plug-ins were created and merged in the ATUSA stack, to give a complete suite ready for further use within Guidion to test their AJAX based web applications and as contribution of this thesis. The list below presents an overview about all plug-ins.
2Crawljax ApiDoc http://crawljax.com/apidocs/
22 1. (W3C) DOM Validator plug-in
2. Test-Suite-Generator plug-in
3. Test-Suite-Executor (Limited version)
4. Dead clickable detector plug-in (OnFireEventFailed plug-in)
5. Back-tracking plug-in (generic state-machine invariant AI1)
6. Error reporting plug-in
7. Overview plug-in
The back-tracking plug-in the generic application specific state-machine invariant AI1 in the initial research A1, which is used for every experiment. Implementation details of the plug-ins can be found in the the tool section 5.10.2.
5.2 Case study 1 Introduction
The goal of the first case study replication is to assess the creation and the scenarios described in the initial research. Therefore are four different web applications in four different scenarios are used.
Replication scenarios overview The experiments conducted within the replication use different scenarios for testing the effectiveness of the use of invariants. Every scenario uses an open-source web application as subject system and the systems per scenario are as follows:
1. The Organizer, Invariant creation and validation in normal setup mode.
2. TaskFreak, Invariant creation and validation in normal setup mode.
3. HitList, Invariant creation and validation with regression versions.
4. The Tunnel, Invariant creation and validation for JavaScript expressions.
Application preparation and reset Before every replication run, the database was restored to its initial state. The web applications, which use a authentication mechanism include a already created Crawljax test user. This ensured comparable results, because the server state was for every run identical.
5.3 Case study 1.A “The Organizer”
The first case study uses The Organizer as test web application. The Organizer is a J2EE web application using the prototype AJAX JavaScript library, the web application framework for J2EE and the Spring data JDBC library.
The Organizer installation The Organizer was installed on a local Tomcat3 web server instance.
Changes made to bring the web application in the testing state The Organizer web ap- plication required some extra work to get it ready for the replication. A first quick test of the web application revealed several issues with its old code base. These issues have a direct impact in the expected results of the case study. One issue is the logoff function, which seams to be a blocked or reserved word within the Browsers. Both test browsers (Firefox and Chrome), which were used to verify the blocking of the function. The logoff() function was renamed in logioffi(), and it worked as expected.
3Tomcat http://tomcat.apache.org/
23 Another issue is the “Fail%” regular expression invariant OI3. This invariant tests if a text message occurs in the DOM with the prefix “Fail%”. A full text search on the projects source code and a manual inspection of the JavaScript and JSP files to find a error message which fits the pattern ended negative. Invariant OI3 tested the DOM on a error message which never occurred in the application. Furthermore was the welcome alert message disabled. Cause of a bug in the Firefox remote driver, which threw an AlertUnhandledException. This led to the problem that Crawljax was unable to retrieve the DOM and the screen-shot from the embedded browser. The welcome alert message was no important issue and had no impact on the replication results. Furthermore was it a local driver problem and will be fixed with a driver update.
Seeded errors and changes to test on the expected results To be able to replicate the case study several changes were made before executing the replication. Every AJAX function within The Organizer uses an on Error function call, which is always the same JavaScript function. To provide a test basis for OI3, the alert was changed in a $(’#container’).html = ‘Failed to populate the list properly’ Error message as described in the initial research in table 5 on page 46. For the test with Firefox and the invariant OI5, the “logioffi()” body was replaced with a empty function body, that a call to this function result in the same state and not in the expected logoff state. This change was made to replicate the findings of the use of the application specific state-machine invariant such as OI5. This seeded error recreates the scenario that the logout fails with the use of the Firefox browser, as described in [MvDR12].
Required configuration Additionally to the default configuration The Organizer requires the configuration of an active random input generator, and the IMG tag element as clickable. (1 minute)
Required plug-ins For a complete overview the experiment used the following plug-ins.
• Authentication OnBrowserCreated plug-in
• Overview plug-in
• Error reporting plug-in
• Application specific state machine invariant OI4 plug-in
• Application specific state machine invariant OI5 plug-in
• Back-tracking (generic state-machine invariant AI1) plug-in
Case study scenario The replication was conducted regarding the following scenario.
1. Running ATUSA with basic configuration and Overview plug-in configured.
2. Analysing the generated overview graph for Invariants and input specification.
3. Configuring the input specification the generic and the two application specific invariants (OI1 - OI3), Backtracking plug-in (AI1), Error reporting and application specific state machine invariant plug-ins (OI4 - OI5).
4. Running ATUSA with the new configuration in Chrome.
5. Changing the logoff function and run ATUSA with the Firefox browser.
Created and evaluated generic and application specific state invariants The following generic and application specific invariants OI1 - OI3 were proposed, which are named O1 - O3 in the original research. [MvDR12]
• OI1 - XPath condition invariant, every state with //IMG[ contains( @src, ‘head yearView’ ) ] contains one ore more appointment items identified with //INPUT[ @id = ‘edit’ ].
24 • OI2 - Generic state regular expression invariant, with the expression “Failed to populate the list properly”.
• OI3 - Regular expression condition invariant. The appointment item view must match a given item view regular expression pattern.
Implementation of the application specific state-machine invariants Crawljax creates an “Eventable” (edge) when an event was successfully fired and leads to a new or existing state within the state flow graph. The application specific state machine validation plug-in validates the state flow graph transition after the proposed pattern, Source → Eventable / Action → Target from the initial research in section 5.4 on page 42. The Organizer case study requires two state machine invariants, OI4 - OI5.
• OI4 - Every clicked logoff button eventable via IMG[ contains( @id, ‘logoff’ )] is a transition to a state with a DIV[ contains( text(), ‘You have logged out’ ) ].
• OI5 - Every clicked image eventable via IMG[ @id = ‘X’] is a transition to a state with a IMG[ contains( @src, ‘head X’ ] element. X must be the same for the eventable and the target state.
Both application specific state-machine invariants are implemented in ATUSA as PostCrawling plug-ins. They check the completely inferred state flow graph on its existing transitions after the crawling process.
Manual effort measurement summary The measured manual effort to replicate The Organizer experiment was as follows.
• First run configuration of ATUSA. 7 minutes
• Analysing the graph including invariant evaluation and input specification search. 48 minutes
• Creating authentication plug-in. 12 minutes
• Creating input specifications. 30 minutes
• Developing the application specific state invariants (OI1-OI3). 1 hour and 19 minutes.
• Developing the application specific state-machine invariants. 4.5 hours.
• Configuration for the second run including input specification, invariants and plug-ins. 15 minutes
Replication findings and comparison with initial case study findings
• The proposed invariants were valid and their expression was possible within ATUSA. The manual effort required to configure ATUSA and express the different invariants required more time as for the authors of the initial study. This can be explained with the fact that the authors created Crawljax and ATUSA and knew required implementation details. The initial research stated less than 30 minutes as manual effort. The replication required 7 hours and 41 minutes
• To produce Application specific state machine invariants for Crawljax and the ATUSA stack require a good knowledge about the Java programming language and an advanced knowledge about the inferred state flow graph as well as its included data. The documentation of Crawljax is not always clear and the source code was conducted to collect further information about the state flow graph implementation.
• The expressed invariants were able to detect violations of OI2, OI4, OI5, the same as stated in the initial research [MvDR12]. OI4 was detectable after disabling the logoff method in the web application.
• The backtracking plug-in detected the violation of AI1.
25 5.4 Case study 1.B “TaskFreak”
TaskFreak is a simple task management web application implemented in PHP Taskfreak-0.6.4. Task- Freak uses XajaX4 and JsCalendar5 as additional JavaScript libraries to plain JavaScript.
TaskFreak installation and changes made to bring the web application in the testing state TaskFreak is based on PHP 5.0 and the web server of Guidion where it is hosted runs PHP version 5.4 and the changed error handling between these versions caused additional unexpected error messages. These could be simply fixed through the error handling settings in the configuration file of TaskFreak. Additionally was a administrator account created with the name “crawljax” and password “crawljax” which is used as test user.
Required configuration The original case study describes in addition to the default configura- tion, that table elements are candidate clickable. Therefore TD, and TH element tags are added. Furthermore was the random input generator configured for filling unspecified input elements with randomly generated text values. Also required TaskFreak an Authentication plug-in to login, which was implemented. (10 minutes)
Required plug-ins
• Authentication OnBrowserCreated plug-in.
• Overview plug-in.
• Error reporting plug-in.
• Backtracking (generic state-machine invariant AI1) plug-in.
Required oracle comparator TaskFreak uses a Date Time element, which displays the current Date including a ticking clock. The Date-time oracle comparator is used to strip the Date/Time before DOM comparison to detect a new state. This avoids the additional state creation and possible state-machine explosion. The Date-time oracle comparator substitutes the default Simple Oracle Comparator and is implemented as a regular expression comparator including all general date-time format patterns. (5 minutes)
Required custom input specifications TaskFreak required additional specified input specifica- tion to limit the state exploration.
• One valid and one invalid email address.
• Password specification for password change form. Including Multi-values to ensure that the second password entered is the same as the first one, as at section 8.2.3 on page 46 of the initial research.
Case study scenario The experiment replication was conducted regarding the following scenario.
1. Running ATUSA with basic configuration and overview plug-in defined.
2. Analysing the generated overview graph for invariants and additional input specifications.
3. Configuring the input specification, the additional DateComparator, the application specific invari- ants (TI1-TI2), Backtracking plug-in (AI1), Error reporting and application specific state machine invariant plug-in (TI3).
4. Running ATUSA with the new configuration.
4XajaX http://www.xajax-project.org/ 5JSCalendar http://sourceforge.net/projects/jscalendar/
26 Created and evaluated application specific state invariants The original research proposes the following application specific invariants: TI1 - TI2, in the original research T1 - T2.
• TI1 - Design XPath condition invariant, every states top level body contains of a DIV[ @id = ‘header’ ], a DIV[ @id = ‘container’ ] and 0 till 1 DIV[ @id = ‘calendar’ ] elements.
• TI2 - All states with a element DIV[ @id = ‘taskSheet’ ] which displays a table of tasks, match a table regular expression template invariant.
Created and evaluated application specific state-machine Invariants The original research proposes the following application specific state machine invariant: TI3 in the original research T3.
• TI3 - Every clicked reload button (eventable) via IMG[ @id = “frk status”] is a transition to a state with a DIV[ @id = “taskSheet”] element, displaying the current tasks.
The invariant is implemented as a PostCrawlingPlugin and checks the complete inferred state flow graph on its existing transitions after crawling.
Manual effort measurement summary The measured manual effort to replicate the The Orga- nizer experiment was as follows.
• First run configuration of ATUSA. 4 minutes
• Analysing the graph including invariant evaluation and input specification search. 28 minutes
• Creating authentication plug-in. 10 minutes
• Creating input specifications. 23 minutes
• Configuring oracle comparator. 5 minutes
• Developing the application specific state invariants (TI1-TI2). 25 minutes.
• Developing the application specific state-machine invariant plug-in (TI3). 30 minutes.
• Configuration for the second run including input specification, oracle comparator, invariants and plug- ins. 10 minutes
Replication findings and comparison with initial case study findings
• The proposed invariants were valid and their expression was possible within ATUSA. The manual effort required to configure ATUSA and express the different invariants required more effort as mentioned in the initial study. The initial research stated less than 1 hour as manual effort. The replication required 2 hours and 15 minutes.
• The application specific state-machine invariant was implemented in a shorter time, because of a better knowledge about the state flow graph. This was observed through the measured manual effort required to implemented TI3.
• With invariant TI1 it was possible to identify the slowly growing DOM as described in [MvDR12].
• The back-tracking plug-in detected the violation of AI1.
• Even if only evaluation of the invariants was included in the replication, does their validation require a good knowledge about the web application and their implementation.
Research inconsistencies To provide a complete replication report, the inconsistencies in the description of the case study from the initial research are listed and briefly explained here. For the application specific invariant TI3, the id of the click element is not “frk-status”, it is “frk status”.
27 5.5 Case study 1.C “HitList”
HitList is an open source PHP and AJAX based contact manager. It works on PHP 5.2, MySQL and uses JQuery with the LiveQuery plug-in. HitList was used in several publications of the authors and the details to replicate the HitList case study are taken from the following two publications [MvDR12] and [RMvD10].
HitList installation, changes made to bring the web application in the testing state HitList was installed on a Guidion web server with a PHP 5.4+ version. HitList uses a deprecated MySQL PHP library, which was not further supported by the web server. For a correct HitList version all MySQL related function calls were updated to the newer and supported MySQLI function calls.
HitList versions The details of the HitList case study are partially taken from the research RoestMD10. This case study targets the use and results of the test suite generation and execu- tion in regression versions of HitList. This case study is originally divided into three parts. To limit the scope of the replication, only the two versions are chosen which are essential to build the required scenario, to reproduce the findings described in the initial research and answer RQ1. These two HitList versions are: • Version one is the original HitList version, with the required changes made to bring the web application in a working state. • Version two is an altered version of HitList, where the phone number does not include any leading zeros in the contact overview state. This is done by a regular expression, which removes all zeros from the beginning of the phone number string.
Required basic configuration The basic ATUSA configuration includes the default configuration and additionally the specification of input tags with the attribute class = ‘add button’ as clickables. For limiting the state creation and testing of the required CrawlCondition, anchor tags with the attribute title = ‘Delete%’ and anchor tags with the attribute class = ‘delete button’ are configured as dontClick clickables. The manual effort was 25 minutes for the evaluation of the clickables and the creation of the configuration.
Required plug-ins • Test-Suite-Generator plug-in • Overview plug-in • Error reporting plug-in • Back-tracking (generic state-machine invariant) plug-in (AI1)
Required CrawlCondition The HitList case study replication makes use of a CrawlCondition to determine if the ‘Add Contact’ state can be crawled or not. The CrawlCondition for HitList uses one precondition and a condition. The precondition is a regular expression condition, which checks if the new state contains the text ‘Add Contact’. The condition checks if a contact with the name “John Doe” exists. This is done by a JavaScript condition using a JQuery AJAX call to the ajax/home.php page. The JavaScript expression proposed in the initial research was deprecated, so the expression was updated to a working version.
Required comparators For stripping the DOM before comparison the original case study proposes two comparators. The first one is a ListComparator to strip the menu list of HitList, and the second one is a TableComparator, which is used to strip the contacts tables from the contacts overview state before DOM comparison. Both comparators inherit from the given RegexComparator and are implemented as own classes, so they can be shared with the Test-Suite-Execution-Controller, see section 5.10.1. The comparators are only briefly documented and their implementation required more development and testing time than estimated, (6 hours).
28 Seeded leading zero error The leading zero bug was seeded in the second version of HitList. It removes the leading zeros from the contacts phone numbers. The error was seeded to evaluate the effectiveness of invariants within the execution of the generated test framework in a regression test scenario. The replication considered the regression case and it was included it in the replication scenario.
Case study scenario The replication was conducted regarding the following scenario.
• Running ATUSA with default configuration and Overview plug-in configured.
• Analysing the generated overview graph to evaluate the proposed invariant template, input specification creation and CrawlCondition.
• Configuring the additional list and table comparators, the contacts template invariant (HI1), Back- tracking plug-in (AI1), Error reporting and Test-Suite-Generator plug-in.
• Running ATUSA with the new configuration and additional input specifications.
• Changing the URL of the generated test suite to point to the regression version of HitList with the seeded leading zero bug.
• Running the generated Test Suite with the Test-Suite-Execution-Controller, including the same in- variants, comparators, wait-conditions and configuration as ATUSA used for the second crawling run.
Created and evaluated application specific state invariants The original research proposes the following application specific invariant: HI1 in the original research H1.
• HI1 - A regular expression condition, which validates the contact state DOM. It also includes regular expression patterns to validate the data displayed per contact targeting the leading zero bug.
Manual effort measurement summary
• First run configuration of ATUSA. 7 minutes
• Analysing the graph including condition evaluation and input specification search. 30 minutes
• Comparators creation. 6 hours
• Analysing, evaluation and creation of the CrawlCondition. 22 minutes
• Analysing, evaluation and creation of the contacts regular expression template invariant (HI1). 20 minutes
• Configuration for the second run including comparators, invariant and plug-ins. 9 minutes
• Changing URL of generated test suite. 30 seconds
Replication findings and comparison with initial case study findings
• The proposed invariant were valid and the expression was possible within ATUSA. The manual effort required to configure ATUSA and express the invariant was not included in the initial research. To provide a complete and comparable replication report, the required manual effort for the replication was 7 hours and 38.5 minutes.
• To develop the comparators required additional effort and time cause of their short documentation. The oracle comparators could be implemented and they reduced the identified states.
• With invariant HI1 was it possible to identify the seeded leading zero bug in a regression run.
• With the re-implemented ATUSA test framework was it possible to test a regression version of HitList.
• The backtracking plugin detected the violation of AI1.
29 5.6 Case study 1.D “The Tunnel”
The Tunnel is a JavaScript based arcade Game. In focus of this case study is “The Tunnel”, which is a arcade tunnel game implemented in JavaScript and JQuery. The mission in “The Tunnel” is to navigate a ship through a tunnel with moving walls.
Case study goal The case study replication of the tunnel focuses on the ability to express and test JavaScript expressions. Its origin case study does also only focuses on their expression, and mentioned its use with the WebScarab plug-in, which is based on the research by Mesbah et al. [GMvD10]. Unfortunately was the plug-in out of date since 2010. This replication evaluates the JavaScript invariants and their expression in ATUSA. The implemented error report plug-in provides a way to set and test those JavaScript expressions apart from the invariants. The case study replication with “The Tunnel” evaluated the invariants from the initial research description, their expression and implementation in ATUSA.
Case study scenario For the Tunnel replication, the original game was copied and deployed on the Guidion server instance. Furthermore the case study replication followed the following scenario.
1. Analyze “The Tunnel” JavaScript code
2. Evaluate correctness of the proposed invariants of the initial research
3. Express the proposed invariants in ATUSA
4. Run ATUSA test case example to test JavaScript expression invariants as regression scenario.
Created and evaluated application specific state invariants The original research proposes the following application specific invariants: TTI1 - TTI3 in the original research U1 - U3.
1. TTI1 - The global “alive” variable is true, as long the player does not fail. When he fails “alive” must be false.
2. TTI2 - The space ship position must be at all-time 32 times higher than the right wall.
3. TTI3 - The global background must be always between 0 and 20.
The invariants used for “The Tunnel” case study can be found in the original research [MvDR12] in Table 5 on page 46.
Summary and findings comparison ATUSA can be used to express and test JavaScript ex- pressions. With the support of the report plug-in and its added support to retrieve and read the browser console log. The Error Reporting plug-in includes a assertion mechanism to execute and log JavaScript expressions [RMvD10]. The evaluation of the invariants and expressing the invariants in ATUSA, took 43 minutes. This manual effort does not include the time required to provide the reporting mechanism as well as the added functionality to retrieve the browsers log (More details about the browser log retrieval can be found in the tool implementation section 5.10.1).
5.7 Case Study 2 “Tudu”
The second case study replication uses the open source web application Tudu as test subject. Tudu is a Java based web application to manage tasks. It uses the DWR framework (Direct Web Remoting)6 for a simplified way to use AJAX with Java. As additional client-side JavaScript libraries it uses Scriptaculous7 and PrototypeJS8. The goal of the case study replication is to assess the ATUSA approach on its scalability, fault revealing capabilities and required manual effort. The findings are used to answer RQ2 - RQ4
6DWR http://directwebremoting.org/dwr/index.html 7Scriptaculous https://script.aculo.us/ 8PrototypeJS http://prototypejs.org/
30 Tudu installation The Tudu web application was deployed on a local Tomcat server instance. Tomcat uses a self-generated server certificate to provide HTTPS connections. This caused in the DWR framework for an alert message, which warns for CSFR Security Errors. This was fixed with an additional initialise parameter in the web.xml.
Tudu application size The application size was measured with the Ohloh, Openhub9 source code line counter (LOC) ohcount10. A part of the replication concentrates on measuring client and server- side code coverage of the generated test suite. For this purpose the application size (LOC) of Tudus client and server-side implementation was recounted. The Tudu version used for the replication has the following application size.
• LOC server-side is in total, 2849 LOC.
• LOC client-side is in total, 10100 LOC.
• LOC client-side external is in total, 9755 LOC.
• LOC client-side internal is in total, 345 LOC.
Tudu application state for the replication Before every replication run, Tudu was removed and redeployed to the Tomcat server, to ensure a similar database with only the shipped sample data. In addition a user was registered with the name “crawljax” and the password “crawljax”. These login credentials were further used in a authentication plug-in for ATUSA.
Required basic configuration To crawl and test Tudu, ATUSA uses the default configuration. In addition to the default configuration ATUSA was further configured to use the provided EditDis- tanceComparator and a authentication plug-in.
Required plug-ins The Tudu replication required the following plug-ins.
1. Authentication plug-in (Using the OnBrowserCreated and not the PreCrawlingPlugin interface)
2. Test-Suite-Generator plug-in
3. Overview plug-in
4. DOM Validator plug-in (generic state invariant)
5. Performance plug-in (Not included in manual effort measurement)
6. Back-tracking plug-in (generic state-machine invariant AI1)
Required oracle comparators The oracle comparator used for the replication is the EditDis- tanceComparator. This Comparator existed already and was directly ready to use. (5 minutes). The EditDistanceComparator uses a threshold for the state comparison. In the original case study the threshold used is 0.89, which was also defined for both replication runs. The EditDistance- Comparator uses the Levenshtein Edit Distance to determine whether two states are equivalent [MvDR12].
Required tools For answering RQ2 of this case study the initial research measures the coverage of the client-side and server-side implementations of the generated test suite. The coverage was also measured within the replication. JSCover11 was used to measure the client-side JavaScript implementation. JSCover substitutes JSCoverage from the initial research. It is the official further developed branch of JSCoverage and fully implemented in Java. In addition to measure client-side
9Openhub https://www.openhub.net/ 10ohcount https://github.com/blackducksw/ohcount 11JSCover http://tntim96.github.io/JSCover/
31 JavaScript, the presentation layer of Tudu was instrumented with Clover12 from Atlassian (with a 30 day trial license). Clover is the same coverage measurement tool as used in the original case study. For the actual instrumentation the Clover Maven plug-in was used. Before instrumenting the presentation layer, the JSP files needed to be pre-compiled. This was done with the JSPC compiler in combination with the Tomcat JSPC compiler. The process is further explained in a Clover ANT example13, which was a good reference to understand and rebuild the process with Maven.
Seeded Errors For the last run (Step 7) ten errors were seeded in Tudu. These errors were also from the same nature as the errors mentioned in [MvDR12]. Unfortunately Tudu is no longer maintained and the bug-tracking list of Tudu14 was empty, this was the basis for the seeded errors of [MvDR12]. Therefore the errors were manually seeded in Tudu. These errors include, DOM errors, longer input values then allowed by the persistence unit, adding dead clickables, adding duplicate to- do items and removing all list items instead of one. These errors were seeded in the web applications DOM and in a altered version of the generated test-suite, e.g. removing all list items or add a duplicate one.
Change and impact declaration on replication results The initial research describes the case study only briefly. No published implementations of the plug-ins are available. This means their implementation cannot be verified for correctness against the plug-ins used in the original research. Another factor of influence is the use of a newer Tudu version. The original case study does not mention a version. The Tudu version used is for the replication is the last published version available. The use of a newer version of Tudu can influence the case study results. Furthermore for the measurement of the client-side JavaScript code, JSCover is used, which is an improved version of JSCoverage and offers a better coverage measurement as its predecessor15.
Replication scenario The replication was conducted regarding the following scenario.
1. Running ATUSA with the EditDistanceComparator, Authentication, Overview and TestSuiteGenerator plug-in.
2. Analysing the generated overview graph and specification of input specifications for every single data entry point of Tudu.
3. Configuring the additional DOM Validator plug-in, Backtracking plug-in, Performance and Error Reporting plug-in.
4. Running ATUSA with the new configuration and additional input specifications.
5. Instrumenting Tudu client-Side implementations with JSCover and server-side implementation with Clover. (Clover is configured to measure only coverage of Tudus JSP presentation layer through pre-compiling of the JSP’s)
6. Running the generated test suite against the instrumented Tudu version.
7. Seeding ten errors in Tudu, and run the generated test suite to evaluate the fault revealing capabilities.
Manual Effort measurement The case study replication required the following manual effort.
• Development of Authentication plug-in. 23 minutes
• First run configuration of ATUSA. 7 minutes
• Basic configuration and setup of ATUSA. 12 minutes
• Additional configuration (Step 3). 1 minute
12Clover https://www.atlassian.com/software/clover/overview 13Instrument JSP with Clover https://confluence.atlassian.com/display/CLOVER/Instrumenting+JSP+files 14Tudu bug-tracking https://github.com/jdubois/Tudu-Lists/issues 15JSCover versus JSCoverage http://tntim96.github.io/JSCover/manual/manual.xml section 7.1.4
32 • Analysing data entry points and creating the input specifications. 38 minutes
The case study replication uses the following plug-ins, which needed to be completely implemented. The measurement is not taken into account in the original research. To provide a complete replication report the following plug-ins required a manual effort of 11 days. These plug-ins are the DOM validator plug-in and the performance plug-in. Further details about their implementation can be found in paragraph 5.10.2 and 5.10.2 of the tool implementation.
Replication findings and comparison with initial findings The table below shows the findings after the second run compared with the findings of the original research [MvDR12].
Name Replication Original DOM string size 13682 byte 24908 byte Detected candidate clickables 861 332 Detected clickables (edges) 215 42 Detected states 57 34 Detected data entry points 4 forms and 18 inputs 4 forms and 21 inputs Back-tracking False False Generated test cases 55 32 DOM violations 0 (Errors) , 0 (Warnings) 182 (Errors) ,0 (Warnings) Coverage server-side 81% 73% Coverage client-side 28% (ext.) and 78% (int.) 35% (ext.) and 75% (int.) Detected faults 90% 80% Manual effort 74 (minutes) 26.5 (minutes) Performance 12.2 (minutes) 5.6 (minutes) The differences in the detected candidate clickables, detected states and edges can be explained through the evolution of Tudu and Crawljax. The version of Tudu is not mentioned in the initial research, but since the publication date and the time of the replication was Tudu still in active development. Furthermore was Crawljax improved in detecting and states, edges and candidate clickables through the research of [NMvD14]. The server-side and client-side coverage of the generated test suite is in the same region as the original coverage. The different coverage result can be explained through a better coverage measurement implementation within the coverage tools used, e.g. especially on the client-side through the use of JSCover. The generic invariants were able to detect back-tracking faults and several DOM violations.
• The Oracle comparator EditDistanceComparator exists and works as expected.
• The finding of the original research can be confirmed that ATUSA can help in revealing generic errors such as DOM violations automatically.
• RQ4 - Overall manual effort to configure ATUSA for the replication as described took in total 1 hour and 14 minutes. The manual effort stated in the initial research was 26.5 minutes.
• RQ2 - The generated test suite was able to detect 9 of 10 errors. The test suite of the initial research detected 8 of 10 errors. The seeded errors of the replication are from the same sort as stated in the initial research.
• RQ2 - ATUSA was able to reach a coverage of 78 % int. and 28 % ext. client-side JavaScript implementation and 81 % of the server-side presentation layer.
• RQ4 - The generated test suite was able to support finding bugs in regression versions. The generated test suite provides a good basis to be extended for more test scenarios.
• RQ3 - The performance of ATUSA was also very acceptable. ATUSA identified almost double as much candidate clickables, edges and states. This in a time frame of 12.2 minutes, which was a bit more than double the time of the original research 5.6 minutes.
33 JSCover The use of JSCover was very helpful to get insights in the actual use of the JavaScript libraries in use of the web application. Its Java based implementation and its good documentation leaded to a plug-in creation for the generated test suite. The test suite automatically creates a server in JSCovers proxy mode and sets the desired browser capabilities with the proxy configuration for the embedded browser of ATUSA before the test execution. The complete execution of the test suite gets measured, and its result gets automatically published. For further details please see section 5.10.1.
5.8 Case Study 3 “Finding real bugs”
This “Finding real bugs” case study targets the research questions RQ4 and RQ2 and evaluates them in a real world scenario at Guidion. The web application chosen for this case study is one of Guidions own portfolio, which was during the research in a active development phase. This replication evaluates and asses the the fault revealing capabilities RQ2 of ATUSA and the manual effort RQ4 required to prepare ATUSA.
Case study test subject web application The original case study used a commercial web ap- plication with the name “CJZ, Coach Yourself”. Due to its commercial nature and its closed source, “Coach Yourself” was not available as test subject for the case study replication. Therefore a Guidion web application was chosen as test subject system. Guidion has a few different web applications, which are currently in an active development phase. The web application chosen is the in-house de- veloped logistics web application called Cargo. Cargo is a PHP based web application, which makes use of the JavaScript libraries, JQuery and Bootstrap. It is a complex web application and its correct functionality is very important for Guidion. The system combines a web shop with a product-stock management and track and trace system, and is used by the field force for ordering and by the depots and staff for product and stock management.
Case study focus The development team from Cargo consists of two developers, one full-time developer who is responsible for most of the AJAX UI functionality and one half-time developer who is responsible for the server-side implementation. For the case study replication, a new and complete task of the upcoming development sprint was chosen. This task was to implement a new dashboard component. Every administrative user monitors his work relevant critical values with help of a central dashboard. The old version was a static dashboard, which displays for every authenticated user the same information, regardless if they are important for the user or not. The new dashboard component is an interactive dashboard, which can be configured and saved for every user individually. The requirements of the dashboard are the following. • CD-REQ-1 - Dashboard container design is same as in the old static version. • CD-REQ-2 - All possible dashboard panels, shall be selectable via one concise modal settings dialogue. • CD-REQ-3 - The settings dialogue, includes a button for saving and resetting the dashboard. • CD-REQ-4 - The dashboard settings are added to the user settings table via AJAX. • CD-REQ-5 - The data displayed in the dashboard gets delivered via a new dashboard web service. AJAX is required to request and update the dashboard panels. • CD-REQ-6 - The dashboard panels consists of one name column and one value column. • CD-REQ-7 - The dashboard panels value column displays the values as a link. The link leads to a overview page with more detailed information.
Differences with the original case study The difference to the original case study is, that the developer has already predefined requirements and can only use the given JavaScript library JQuery. Furthermore had the developer no Java experience and no Java friendly development environment installed. In this case the observer expressed the delivered invariants in ATUSA for the case study. The web developer was asked to document suitable invariants for the design, the requirements and his implementation.
34 Case study set-up and implementation This case study took place after a introduction presen- tation of ATUSA and invariants in the scope of the invariant research explained in section 5.9. As a additional support got the web developer an introduction of possible invariants, to stimulate his invariant creation process. These example invariants were a collection of the invariants used during the first two case study replications explained in section 5.2 and 5.7. After the completion of the new dashboard component for Cargo, the identified invariants were expressed in ATUSA. Because of the missing Java knowledge of the web developer, the implementation of the proposed invariants was executed by the experiment observer. ATUSA was executed after deployment of the new dashboard component to the Cargo test server and user acceptance.
Application specific invariants The web developer proposed four application specific invariants.
1. CI1 - A XPath expression invariant to validate the design of the dashboard panel. A dashboard panel always consist of a main element DIV[ @class = “dbContainer”], and three children. A DIV[ @class = “dashboard header”] element, a DIV[ @class = “dashboard content”] and a DIV[ @class = “dashboard footer”] element. 8 minutes
2. CI2 - A XPath visible condition invariant to validate that the dashboard panel is visible in a defined time frame of maximal 1.5 seconds. A element with DIV[ @class = “dbContainer”] must be visible under 1.5 seconds. 5 minutes
3. CI3 - A application specific state-machine invariant to validate transitions. Every clicked button (eventable) via A[ @class = “no text button”] and [ @data-url = X] is a transition to a state with a DIV[ @class = “dbContainer”] with a [ @data-url = X] element. X must be the same for the eventable and the target state. 18 minutes
4. CI4 - Regular expression template for validating the data within a dashboard panel. The regular expression template validates the structure of the element DIV[ @class = “dashboard content”] and the displayed data. The DIV element contains a table with two columns. A name and value column. The regular expression also includes a validation of the displayed data within columns. The name column via [a-zA-Z]+3,30 were the name has at least three letters and maximal thirty letters, and the value column via [0-9]+. 9 minutes
Generic invariants The initial research [MvDR12] has no explicit section about the generic in- variants used for the case study. According to the reported errors found on page 49 in table 7, three additional generic plug-ins were used for the initial case study [MvDR12]. In addition to the proposed generic invariants one additional invariant was proposed by the web developer to check every DOM for server side or AJAX related error messages. This was done with regular expression invariants to check the DOM for error messages, e.g. text which begins with “Warning” or “Error”. In total were four generic invariants configured for the case study. Three generic invariants are implemented as plug-ins and one as a regular expression invariant. The generic invariants used by this case study replication are:
• Error reporting plug-in with predefined regular expression invariant.
• DOM validator plug-in.
• Dead clickables plug-in (generic state-machine invariant).
• Back-tracking plug-in (generic state-machine invariant).
The configuration of ATUSA to use these four generic invariants required 7 minutes.
Additional required plug-ins The users of Cargo need for the shop environment and the admin- istration environment valid authentication credentials. Therefore a authentication plug-in was created for ATUSA. Cargo uses the Google+16 service for authentication. The authentication plug-in was implemented as an OnBrowserCreated plug-in in 27 minutes. In addition to the authentication plug-in the overview plug-in was also included in less than 1 minute.
16Google+ Signin https://developers.google.com/+/web/signin/
35 Required configuration Cargo is a complex system with a lot of different application states. For the case study ATUSA was configured using the default configuration and a set of click exclusions. The exclusions were necessary so ATUSA stays on the dashboard page and executes only dashboard relevant candidate elements. These exclusion rules were:
• Exclude all menu children of the element DIV with the attribute id = ”bs-example-navbar- collapse-1” .
• Exclude all breadcrumb children of the element DIV with the attribute class = ”breadcrumb” .
• Exclude the click element for the Atlassian widget to report a bug, with the element tag A and the attribute id = ”atlwdg-trigger” .
Other relevant clickables to test the dashboard were already included in the default configuration. Furthermore was no additional input specification required for this case study. The manual effort to configure the exclusion click rules for ATUSA was 3 minutes.
Required oracle comparators The framework of Cargo identifies a logged in user via a token which is added after successful authentication. This token is also added to every link reference which requires a authenticated user. For limiting the state creation, a oracle comparator was created to remove the token variable from every link which contains the token. The oracle comparator was named TokenComparartor and uses XPath queries e.g. //*[@href]/@href to identify possible attributes. For every attribute value which contains a token=, the token gets removed via a regular expression, it starts with the text token= and the token value is a 32 character long hash code. The oracle comparator was added after a first test of the configuration, because of the missing back- tracking functionality in Cargo, ATUSA logs out by navigating back and after login the token changed and causes additional state creation. The implementation of the TokenComparartor required 38 minutes.
Results and detected failures After the development of the new dashboard component and its deployment to the Cargo test environment, ATUSA was used to test the new AJAX dashboard. The generic and application specific invariants were added as described in paragraph 5.8 and 5.8. The three generic invariants identified several errors in Cargo. The DOM validator plug-in found six different types of DOM violations.
• Used HTML5 attributes and tags are not supported by XHTML 1.0 Strict standard. E.g. the use of data attributes.
• INPUT element as a descendant of a A element is not allowed according the standard.
• Unsupported icon attribute on BUTTON elements.
• Unsupported async attribute in script elements.
• Missing type attributes for SCRIPT tags.
• DIV tags in UL list elements are not allowed according to the standard.
The back-tracking plug-in registered that every navigation back was violated and resulted in the login state. This was already earlier identified, therefore the TokenComparator was implemented and added to the configuration of ATUSA. With the generic error reporting plug-in it was possible to locate two panels which displayed error messages during the testing process. These two errors occurred because of missing database columns which were forgotten by the deployment. The server- side implementation generated own error messages, and the AJAX calls executed their error handler and rendered the text “Error”. In this case ATUSA was able to identify these two errors, on server- side and client-side. Error handling with clear error messages was not a specified requirement, but as result from the test these error messages need to be more clearly especially for a root cause analysis. Two application specific errors were found through two different application specific invariants. One error was found with the regular expression template CI4. The invariant followed the example from
36 the HitList case study in section 5.5. A regular expression template which validated the dashboard panels presented data including the dynamic links of the different panels. The error was that one panel displayed in the result column a question mark (?) instead of a valid number. This question mark was shown because of a typing error in the function which extracts the received data from the server. The JavaScript library generated in this case a question mark. The second identified error was through the application specific invariant CI2. The web developer defined a time frame in which the panels need to be loaded. The invariant CI2 checked after each click if the new panel is visible within 1.5 seconds. One panel violated the invariant because of a missing filter statement in the AJAX call to the server-side implementation, which produced a query over all closed orders. These errors found during the Cargo case study are summarized in the table below.
Failure Cause Violated invariant Invariant type Missing back navigation No history function used Back-tracking Generic Six different DOM errors DOM validity not taken into DOM validator Generic account Unexpected error messages Missing database columns Error reporting Generic after deployment Error in displaying data in Coding error CI4 Specific the panels value column Loading limit exceeded Missing filter criteria CI2 Specific
Manual effort The initial case study [MvDR12] states that the invariant creation in ATUSA was easy, but no actual time measurement was stated. Mesbah et al. states in [MvDR12], that the invariants could be expressed in a few lines of Java code. For providing a complete case study report the manual effort to setup ATUSA to the test subject application took in total 1 hour and 56 minutes. The invariant creation for ATUSA required 40 minutes. The manual effort to setup and configure ATUSA was not included in the initial research. This was inconsistent compared with the manual effort measurement of the Tudu case study. For answering research question RQ4 the whole ATUSA configuration was measured and taken into account and not only the measured manual effort for the invariant creation.
Replication findings and Conclusion During the real bugs case study replication, the web de- veloper was observed in identifying and creating the invariants. One additional observation was that the web developer used the example invariants as starting point. According to the web developer were the examples very helpful to see how invariants can look like. Based on the further observations during the case study we concluded that:
• RQ4 - Web developer without Java and ATUSA (including Crawljax) knowledge cannot express in- variants in ATUSA. The observation was that the UI developer could not express invariants in ATUSA without Java and ATUSA knowledge and a Java friendly development environment (minimal a IDE and Apache Maven installed). The web developer used Netbeans as IDE but only with the PHP and web development features (JavaScript, CSS and HTML5).
• RQ2 - The generic invariants were able to detect several faults, which were not known at Guidion. Imple-menting a history for a correct navigation through the web application was not taken into account during development of Cargo. The test with ATUSA has shown that a incorrect back navigation leads to a new login and a new token. This means that when the end user uses several tabs the new login leads to a new security token. The token is always added to the URL as get parameter and a new login in one tab logs all other open tabs also out. In addition to the back-tracking impact the DOM validator revealed six different sorts of DOM validation. Validating the DOM was never done before for Cargo. ATUSA is able to validate the DOM per state and list the errors.
• RQ1 - RQ2 - The application specific invariants are able to detect errors. The errors detected could have been also detected through a complete acceptance test. The tester was a power user of Cargo, and he did only test the panels relevant for his work. This problem is earlier described in the problem section 2.2. The panels with the errors were not tested during the user acceptance test at Guidion but the feature was accepted. ATUSA was able to identify the errors through a complete coverage of all possible states for this case study.
37 • RQ4 - The manual effort to configure ATUSA took in total 1 hour and 56 minutes and to create the invariants for ATUSA took 40 minutes. The manual effort to express the invariants was for this case study not minimal and this with existing Java and ATUSA knowledge, which was not given by the web developer.
5.9 Invariant research
This section targets the research question RQ5: “How can invariant expression be supported for web developers in ATUSA?”. To answer the research question, the research was divided into two sections. The first analyzes the possibilities to express application specific invariants in ATUSA. The second is a survey research conducted with the web developers at Guidion, to collect information about the web developers knowledge about their current testing habits to validate user interface functionality, and invariant expression related knowledge (e.g. XPath).
Use of Invariants for testing The original research and the results of the replication show that the use of ATUSA including its invariant based testing approach supports identifying DOM or state flow graph related errors. Further research was conducted to infer application specific state invari- ants as already described in section 4.4. These approaches automatically infer DOM and JavaScript invariants, unfortunately these approaches were to time and resource consuming [PZ10] and with the approach of Groeneveld et al. [GMvD10] the open-source plug-ins for Crawljax were out of date, and needed to be adapted to fit the new architecture of Crawljax. Automatically inferring invariants in both approaches require that the crawler run several times, which has an significant impact on the execution times in a testing environment. Both approaches reach a high coverage of inferred invariants but their effectiveness require more empirical evidence. In the scope of this thesis a survey research is included to support web developers in creating invariants for ATUSA. It concentrates on the finding of [MvDR12] to support the invariant creation for web developers.
Generic state and state-machine invariants The generic state and state-machine invariants are used to validate the different states of a web application. The generic state invariants created for the replication are already a sufficient set of generic state and state-machine invariants. This set is already sufficient, because they test aspects of the Guidion web applications which are not included in the manual testing approach of Guidion, e.g. the generic DOM validation invariant. Therefore these generic invariants provide for Guidion additional information about their web applications. Through their implementation during the replication part of this thesis, lies the main focus on simplifying the creation of application specific state and state-machine invariants for the web developers.
Application specific state invariant expression in ATUSA Besides the fact that invariants need to be created in Java for ATUSA, they are provided in a variety of possible expressions. In- variants can be expressed in ATUSA using predefined generic condition types. These condition types form an invariant. Its basic signature is {invariant (condition, preconditions)}. The invariant condition will be validated if all preconditions are satisfied for a state. ATUSA already provides a range of conditions, e.g. a Regular expression condition, JavaScript condition, Visible condition, and XPath condition. Furthermore provides ATUSA a implementation of the logical operators with NOT, AND and OR, which can be used to build preconditions. The conditions supported by ATUSA are categorized into following three condition categories.
1. Expression-based. Conditions using string expressions. These condition types are: JavaScript con- dition, Regular expression condition, URL condition and XPath condition.
2. Identification-based. Conditions using the Crawljax Identification object. The identification ob- ject combines an identification method and its value, to denote an element. ATUSA provides one identification-based condition the Visible condition. The Visible condition can be used to check if after loading of a state, a certain element is visible or not.
38 3. Counter-based. Conditions using a counter combined with other conditions. The provided condition is the Count condition. It counts how many times a condition is satisfied, and if the test condition is less than the specified maximum value it return true.
These generic conditions form the basis for the invariants in ATUSA, one condition as test constraint over the DOM-tree with a optional set of preconditions. Mesbah et al. states in [MvDR12] in section 8.4.3 that expressing invariants in ATUSA is simple and only done in a few lines of code. Even when the invariant expression in ATUSA only cost several minutes, as the replication case study shows. The expression still requires basic Java knowledge and the use of a Java friendly development environment.
Application specific state-machine invariant expression in ATUSA Application specific state-machine invariants require an expression as a PostCrawling plug-in for ATUSA. The replica- tion section already concludes that the development of the application specific state-machine required a good knowledge about the state-flow graph and its implementation in ATUSA. Two types of applica- tion specific state-machine invariants are useful for validating the complete inferred state-flow graph. ATUSA does provide the inferred state flow graph after the crawl process, which can be checked against constraints, but has no simple configurable way to express them. The two types of identified application specific state-machine are the transition constraint and the pattern constraint.
• Transition constraint - The transition constraint validates if a source state with a given element has en existing edge to an expected state (e.g. OI4 - OI5 in section 5.3).
• Pattern constraint - The pattern constraint validates, if a source states with a set of elements matching a pattern, includes transitions to a target states where the pattern recurs (e.g. TI3 in section 5.4).
Temporal logical model checking algorithms for state-flow graph validation A further approach can be to implement logical model checking algorithms to validate the inferred state-flow graph. The case study replication has shown that the use of application specific state invariants can identify state-flow graph transition errors. No invariant-based research addresses this sort of invariants in detail and their diversity needed to be analyzed within a longer case study to validate the need of an implementation with advanced temporal model checking algorithms. For the prototype and the scope of this thesis the two types of application specific state-machine invariants are sufficient.
Possible use of a graph database A Graph Database was considered as possible way to support better transition checks. A graph database is query able, and is made for asking questions against a given graph. It is optimized to deliver fast results, and offers a perfect basis to outsource the application specific state-machine validation. Unfortunately during a small proof of concept with a graph database for state flow graph validation and the test report generation technical difficulties were encountered which required too much time to be further investigated and the graph database option was not further analyzed.
Invariant violation reporting Another observation of the case study replication was, that a complete error reporting mechanism was not given by the plug-ins. The provided overview plug- in includes only a counter for failed events, but no extra information about the failed event. The created error reporting plug-in only includes error reporting about invariants. Therefore a reporting mechanism is required to provide complete information about invariant violations, failed events, back- tracking and DOM violations. These information include details about the states where a violation occurred, its execution trace and additional details e.g. the invariant. This supports the validation of the correctness of the invariant and it stimulates providing more invariants when their use identifies errors before live deployment.
Web developers survey research Determining how the web developers can be supported in pro- viding invariants for ATUSA a questionnaire was created for the web developers. This questionnaire included questions about the current testing habits and tools used for developing and testing. Fur- thermore includes the survey several questions about the web developers knowledge about invariant
39 specific expression possibilities, e.g. regular expressions and XPath. The questionnaires results sup- ported the decision where the invariant support is integrated. The target group for the survey research consists of four web developers, which are responsible for all front-end related development at Guidion. These web developers have an academic background but no explicit testing knowledge. At Guidion these web developers execute the first tests during development. The web developers were chosen as research group because they design and implement the AJAX functionality which alters the DOM. As stated in [MvDR12] in section 9.4, the decision which application specific invariants to adopt is directly connected to the design. Furthermore have web developers not necessarily Java knowledge which prevents the actual invariant implementation as observed in case study three in section 5.8.
Survey research goal The survey research had two main goals, the first one was to identify the tools the web developers use at Guidion and their testing habits. The second goal was to collect infor- mation about invariant expression related knowledge. The invariant related knowledge means knowl- edge about the expression variants for ATUSA, which are regular expressions and XPath. Furthermore collects the survey research additional information to provide a overview about used JavaScript event handlers, used JavaScript libraries, common errors after live deployment. These main goals deliver more information to answer RQ5 and can be summarized as the following research aims of the ques- tionnaire.
1. What are the tools the web developers use and what are their testing habits.
2. Invariant expression knowledge (Java, Regular expressions, XPath expressions).
3. How is error reporting handled.
4. What are common errors the web developers fix after deployment.
In addition to the main goals of the questionnaire, it includes questions to validate if they are open for a automated deployment and testing process like continuous integration. As described in section 4.7, one key to the success to integrate a automated CI process. These information are not relevant for the invariant research question RQ but deliver more evidence about the overall acceptance of ATUSA and CI.
Questionnaire design The design of the questionnaire followed the guidelines of Burgess [Bur03]. The questionnaire consist in total of twenty questions, six are open questions, six are questions with given choices, five are with given choices and a open choice to specify and three range questions. The questions were formulated clearly and as simple as possible and the questions were sorted as groups per topic. The questionnaire can be found in the appendix in section 7.1 and as PDF document at https://drive.google.com/file/d/0B-7eCETi8-g1YkU1eGZkMU5tZFE/view?usp=sharing.
Web developers survey research setup and implementation At Guidion developers can plan so called “Coding Sessions” were new technologies or tools are presented by a developer to other interested developer of the IT department. Before the coding session the web developers were asked to take part in the coding session, with the information that a questionnaire will be handed out at the end of the coding session. In the scope of a coding session a small presentation was held to introduce Crawljax with ATUSA. The presentation also included a detailed introduction of invariants in ATUSA. Which expression variants are possible, what kind of invariant types exist, how they can be used to validate the DOM-tree and how they can be used to validate the state flow graph. After the presentation a short demo was shown, questions were answered and the questionnaire was distributed among the web developers. The survey research followed the scenario below.
1. Presentation about Crawljax and ATUSA, including a detailed section about invariants.
2. Short demo of ATUSA.
3. Answering questions of the web developers.
4. Hand out of the questionnaire, including collection of the filled out results.
40 The web developers were asked to completely fill out the questionnaire. Every web developer of Guidion took part at the coding session and fully answered the questionnaire.
Questionnaire results After completion of the coding session the questionnaires were controlled on completeness, and evaluated. The main results of the questionnaire are the following.
• Web developers at Guidion have a common knowledge about invariant expression relevant knowledge. The knowledge about XPath ranges from 2 - 4. Where 0 is no knowledge and 10 is expert knowledge. The regular expression knowledge ranges from 2 - 5. JavaScript knowledge was not asked, because every web developer at Guidion uses JavaScript for their web development.
• Only the browser is used as testing tool to validate locally developed functionality. At Guidion every web developer uses the Chrome browser, because every work station at Guidion only has the Chrome browser installed.
• Web developers at Guidion have no or little Java knowledge ranging from 0 - 2.
• The common errors which are fixed by the web developers after deployment are: Linking errors, missing closing tags in DOM, AJAX method call errors as side effects from re-factoring. These errors are all detectable through ATUSA. This supports the integration of ATUSA at Guidion.
• The JavaScript event handler mostly used is the click event handler.
• The web developers mainly use JQuery, AngularJS and Bootstrap as JavaScript libraries.
• Every web developer uses a different IDE or Editor for development. These IDE’s and Editors are: Netbeans, Eclipse, and Sublime Text.
• The web developers use source code repositories for their projects and are all open for a automated testing and deployment process. This finding is only important to validate if the web developer are open for a CI deployment pipeline. This finding supports the acceptance of CI and ATUSA within Guidion.
• Error reporting occurs in two manners according the web developers. The first is via email then often without enough information to reproduce the error. The second is via personal sessions with the tester to test the web application. The need for a generalized error report was mentioned by two web developers in the last open question.
Conclusion The web developers at Guidion have a common understanding of regular expressions and XPath, but only little knowledge about Java development. Therefore the creation of invariants need to be separated from ATUSA and its Java based expression. The web developers use only one testing approach to validate their developed functionality, and this is via the Chrome web browser. Another conclusion of the questionnaire is that the ATUSA should generate a report with usable information about the web application test. In addition to the questionnaire the observation of the third case study in section 5.8 has shown that the invariant creation process is stimulated through different invariant examples. In addition the supporting tools used for the replication in section5 are also included to support the invariant expression process. The knowledge measurement from the questionnaire shows that the web developers knowledge about XPath and regular expressions is expandable.
Requirements for the proof of concept The analyzed requirements for the proof of concept are listed. These requirements are a combination of the analyzed requirements for the invariant support tool and for the integration of ATUSA as testing phase in a deployment pipeline. The requirements for the invariant collector are the following.
• Integration in the current testing process (Chrome plug-in).
• Creating application specific state invariants without Java knowledge. Generic support for the three identified types of invariant expression for ATUSA.
• Creating application specific state-machine invariants without Java knowledge. Generic support for the two identified application specific state-machine invariant types.
41 • A centralized support for invariant creation, update, delete, disable and view, so several web developer can delivery invariants for the same web application.
• Support for system connected invariant creation. This enables the delivery of applications specific invariants for several systems.
• Invariant loading mechanism.
• Generic invariants plug-ins used during case study replication as configurable default generic invariants.
• Report generation of tested invariants.
The requirements for a integration in a Jenkins deployment pipeline are:
• Preconfigured ATUSA with plug-ins and executable as CLI.
• Automatic invariant loading mechanism.
• Report publication.
• Email notification.
• Separated job template, for manual single execution or automatic execution via a job chain.
5.10 Tool Development
This chapter describes all required implementations to conduct the research and the replication. These compositions of plug-ins build the basis test framework for Guidion and the case study to answer RQ6. Furthermore based on the results of RQ5, this section proposes an invariant collector/provider application, which simplifies the invariant creation, a reporting mechanism for reporting on violated invariants and a Java component to retrieve the created invariants from the provider. At the end of this section the Guidion CLI application is described which is able to test the Guidion subject systems, Cargo and the Social network Sonar. The Guidion CLI builds the basis for answering RQ6. Guidion CLI is configured in a ready to use continuous integration job, which is also included and described in this section.
Technical details and requirements The developed tools require the Java runtime environment, and the Apache Maven build and dependency management system. This is already introduced by the Crawljax development community and further used for the proposed tool implementations.
Crawljax Version The Crawljax version used is the version SNAPSHOT-3.5.1, this version was identified as the last version with oracle comparators as described in the background section 4.5.
MySQL Database The invariant collector web server uses a MySQL Database in the community edition to persist the created invariants.
Chrome Web browser Guidion uses mainly the Google Chrome Web browser, for using the Invariant collector Chrome extension the Chrome Web browser is required.
Continuous integration requirements Guidion uses Jenkins as continuous integration system in the Version 1.554. Jenkins makes use of the FTP, GitHub, ANT and Email notification plug-ins for the proposed continuous integration process.
42 Additional third party libraries The developed plug-ins introduce the following new third party libraries. • W3C Validation API wrapper library (jcabi-w3c17)
• Apache Velocity18
• JUnit419
• JSCover20
• Spring Framework21
5.10.1 ATUSA stack reimplementation In order to replicate the case studies and have a working ATUSA stack. As described before in section 4.5 Crawljax evolved and several features proposed with ATUSA were merged with Crawljax. Unfortunately the projects main development team further maintains only the Overview plug-in. This section describes all re-implemented plug-ins to build the ATUSA stack.
Test-Suite Generation and execution (AtusaTestSuiteGeneratorPlugin) implementation The Test Suite Generator plug-in was since the version of Crawljax 2.2 not further maintained. The whole Test Suite Generator needed to be rewritten to fit the new Crawljax architecture. The plug-in uses the same libraries as described in [MvDR12] and [Roe10]. The Test Suite generator uses Apache Velocity as template engine for generating the test suite. The Java test Framework was updated to JUnit4, and JSCover was additionally integrated into the test suite to measure JavaScript code coverage. Furthermore the persisting functionality was restored for the crawled states, crawled edges and the crawl configuration. The implementation followed the description of [MvDR12] and [Roe10] and includes nearly all features. As reporting engine uses the Test Suite execution controller of the AtusaReportingPlugin to generate a test report. The figure 5.1 below shows a example of a generated test method.
Figure 5.1: Example of a generated test method by ATUSA
17W3C API Wrapper http://www.yegor256.com/2014/04/29/w3c-java-validators.html 18Apache Velocity Project http://velocity.apache.org/ 19JUnit4 http://junit.org/ 20JSCover http://tntim96.github.io/JSCover/ 21Spring Framework http://spring.io/
43 Test Suite and execution controller and its limitations The execution controller holds all test relevant configuration settings from ATUSA for the generated Test Suite. One limitation is that the execution controller cannot execute all plug-in invocation hooks within the crawl process. The plug-in types require a fully build Crawljax instance only for running the plug-ins. Only the OnBrowserCre- ated and OnUrlLoad plug-in hooks are directly supported, to ensure that the generated test suite can use the authentication plug-ins. These plug-ins get called in the @BeforeClass and @Before anno- tated methods of the JUnit4 test framework. Additional plug-in invocation hooks are covered by the report engine, this includes the OnNewState, OnFireEventFailed and OnInvariantViolation plug-in hooks. The remaining plug-in types were not used during the case study replication and are also not required for the Guidion test framework. These unsupported plug-in types are the OnRivisitState, the PreStateCrawling, the PostCrawling and the PreCrawling plug-in.
Error Report plug-in (AtusaReportingPlugin) The error report plug-in, can be used as a stand-alone plug-in and is also integrated in the the generated test suite to report about the test failures. An initial version of the report plug-in could be found in the GitHub repository of Crawljax for version 2.2 [Cra12a]. The whole plug-in was since version 2.2 not further maintained and required a complete re-factoring to bring the plug-in in a usable state. The AtusaReportingPlugin includes the same features as described by Roest in [Roe10], and reports about detected state difference, invariant violations and failed events during the test suite execution. The plug-in also includes a new mechanism to retrieve the browsers console log, to show JavaScript errors which may otherwise be swallowed by the browsers AJAX engine. This feature is only in a Beta state of the selenium WebDriver and is only supported for the browsers Firefox and Chrome. The figures 5.2, 5.3, 5.4 and 5.5 show different examples of the generated report.
Figure 5.2: Example of a reported failed event and its original DOM
44 Figure 5.3: Example of a reported failed event and its highlighted DOM (The image was merged cause of the lengths of the displayed DOM)
Figure 5.4: Example of a reported failed event and its path to failure
45 Figure 5.5: Example of a reported failed event and its screen-shots
The details of reporting vary by reporting type and execution level, if it is with ATUSA or within the test suite, but for invariants and the failed events the details in the report are the same.
5.10.2 Plug-in implementation required by the replications This section describes the plug-ins required and implemented for the case study replication.
• Back-tracking plug-in
• DOM validator plug-in
• Dead clickables plug-in
• Error detector plug-in
• Performance plug-in
Back-tracking generic state machine plug-in Implementation The Backtracking plug-in is implemented as an OnRevisitedState, OnUrlLoad and OnNewState plug-in. When a state gets revis- ited the plug-in checks if the current state is equal to the last known state. The last and current state gets updated when a new state is detected. Additionally is also the OnUrlLoad hook in use, because it gets called when a backtracking is not possible and the browser goes back to the index state.
W3C DOM validator, generic state plug-in Implementation The DOM validator plug-in is implemented as an OnNewState and PostCrawling plug-in. It uses the “jcabi-w3c” Java library, which wraps the communication with the W3C validation SOAP API. The plug-in creates on every new identified state a HTML file with the DOM of the state. The location gets registered inside the plug-in and when the crawl process is finished, every file and its content get checked against the W3C validation web-service. This plug-in invocation differs from the described version of the initial research in Figure 9 on page 44, which validates the DOM directly when a new state is discovered. The validation process within the crawl process has a too huge impact on the crawl performance. Persisting the DOM for a PostCrawling validation. The DOM validator can be configured to use the limited public W3C validation web-service or to use a local installed validation server instance. The result of the W3C validation is added to the report of the AtusaReportingPlugin. A example is shown in the figure 5.6.
46 Figure 5.6: Example of the reported W3C validation results
Dead clickables detector, generic state machine plug-in implementation The dead click- ables detector is implemented as an OnEventFiredFailed plug-in. The dead clickables detector is a part of the AtusaReportingPlugin, and every failed event gets reported including its state and path to failure.
Generic error detector, generic state invariant plug-in implementation The error detector state invariant plug-in uses the OnNewState plug-in for checking the new discovered state DOM on error messages. These error messages are configurable and are expressed as regular expressions. The error message can also be expressed and tested as invariants with no preconditions, these invariants get validated with every states DOM.
Performance plug-in implementation The performance or statistic plug-in measures the fol- lowing data during and after the crawl process. Average DOM size, discovered candidate clickables, clicked events (edges), discovered states, time used for crawling and its crawling close reason. It is implemented as OnNewState, PreStateCrawling and PostCrawling plug-in.
Plug-in composition merge into the ATUSA reporting plug-in The plug-ins listed above were merged into the Reporting plug-in, because all of the plug-ins require an output mechanism. Additionally to the plug-ins mentioned above, does the error reporting also include violated invariants. This is possible through the plug-in invocation hook OnInvariantViolation.
Generic Authentication plug-in, with support for Google The plug-ins created for authenti- cation during the case study replication, were re-factored to one generic authentication plug-in. This plug-in is implemented as an OnBrowserCreated plug-in, and can be configured to use username and password credentials or to use the Google+ authentication service to login into a web application with a Google account. The plug-in is fully configurable, username, password, and input elements.
5.10.3 Invariant provider, collector and violation reporting This section describes the proposed tool implementation to support a simplified way to express ap- plication specific invariant expression for ATUSA. The tool is divided into two main components, an invariant provider and an invariant collector.
Invariant provider implementation The invariant creation process was integrated into the cur- rent testing method of the web developers at Guidion. The result of the questionnaire has shown that Google Chrome was the main validation tool for manual testing and validation of the developed features. Google Chrome offers with its plug-in system an API to develop own plug-ins. The invariant provider is divided into two components, the first is a Google Chrome plug-in and the second is a
47 small Java based web server with a REST web service API with simple Create, Read and Update functionality. The web server side is implemented in Spring MVC and provides a simple Front end for Test System registration, invariant activation and invariant deletion. The web server connects to a locally installed MySQL database with help of the Spring JPA Framework. It uses the XML based persistence configuration of the Spring JPA Framework to enable a simple configuration basis, without rebuilding the web server. Furthermore includes the server the Spring Boot extension, which enables the server to run as a own web server. The web server invariant creation and administration interface is displayed in the figures 5.7, 5.8 and 5.9 below.
Figure 5.7: Target system selection and invariant initialisation
Figure 5.8: Condition creation
48 Figure 5.9: Optional pre-condition creation
The invariant creation follows a several steps in wizard mode, which is implemented in the web front-end as well as in the Chrome plug-in. The web front-end offers more features than the Chrome plug-in, these features are a visual status of the created invariant, and the possibility to edit or delete invariants, where the Chrome plug-in only offers a quick creation possibility. The invariant providers REST web service API is used by the web front-end and the Google Chrome plug-in, which needs to be connected via a configuration page. The Google Chrome plug-in is the second component of the invariant provider. It is implemented as a browser action plug-in, which can be activated via the status bar and opens a popup, which includes input fields for the invariant creation.
Figure 5.10: Invariant creation with the Chrome plug-in
49 Figure 5.11: Invariant creation with the Chrome plug-in
The figures 5.10 and 5.11 shows the Google Chrome plug-in and its invariant creation possibility. The creation flow is read from the left to the right in the figures. After selecting a system, a invariant can be created with defining a invariant name, a mandatory condition and optional one or more pre-conditions.
Tool integration The figure 5.12 below illustrates how the proof of concept tool integration works together with ATUSA. Web developer’s can deliver Invariants in two manners, through the Web front- end of the Invariant provider, or through the Invariant Collector Chrome plug-in. Both manners use a REST web service to save the created invariants in a MySQL database. After starting ATUSA via the Guidion CLI application, the CLI collects the created invariants from the MySQL database and injects them into the ATUSA configuration, so that these invariants are used during the crawling phase. The ATUSA stack is able to use different web browsers for the crawling phase, it generates a Test Suite from the crawl result and saves it on the hosts file system. Furthermore does the ATUSA stack use the W3C validation service for DOM validation. After the crawling phase and generation of the result artefacts, the generated result report gets published to a Apache web server.
50 Figure 5.12: Proof of Concept Tool integration
Guidion ATUSA CLI The Guidion ATUSA application wrapper inherits and extends the Crawl- jax CLI project, and adds additional support for connecting and loading invariants from a database. The CLI [Cra13] offers the ability to configure Crawljax / ATUSA via the command line with all necessary configuration options, like Waiting times, Crawl Time, test subject URL, plug-in output directory. Only two additional parameters were specified in the Guidion ATUSA CLI.
• -key with the System Key in the invariant provider database, to ensure the correct loading of the invariants. If no key is specified the Invariant loading will be skipped.
• -x, to load a predefined set of configuration settings like oracle comparators, click elements and click exclusions for the Guidion web applications Cargo and the Social Intranet.
51 The invariant loading from the database uses also the Spring JPA Framework and its XML based configuration It connects to the invariant provider database and collects all application specific state, state-machine invariants for the test subject system (indicated by the system key). The Guidion CLI uses as default browser Chrome, because Google Chrome is the default Browser used by Guidion and is installed on every workstation. The default configuration of the replication is used as default configuration of CLI if not otherwise specified. Furthermore includes the CLI in default mode the following plug-ins.
• Overview plug-in.
• AtusaTestSuiteGeneratorPlugin (with JSCover activated).
• AtusaReportingPlugin (with additional DOM validation activated).
Limitations, Oracle Comparators Click and dontClick settings are not configurable The configuration of the oracle comparators was on hold because of the upcoming change to the DOM strippers 4.6.1. The oracle comparators will be substituted with the DOM strippers and then their configurability will be implemented in the Guidion ATUSA application wrapper. At the moment of this thesis only the predefined oracle comparators for the Guidion web applications are supported. Their use can be activated with the parameter x cargo or x social to activate the predefined set of oracle comparators created for the two web applications. These oracle comparators are hardcoded in a comparator builder class. When no oracle comparator parameter is specified the default Simple- Comparator 5.1.1 will be used. For a complete list of the default supported parameters can be found at the Crawljax-CLI application GitHub page22.
Invariant violation reporting The AtusaReportingPlugin was extended to support the OnInvari- antViolation invocation hook from ATUSA. The Invariant violations get in the same manner reported as the Invariant violations of the AtusaTestFramework. The invariant violations are reported includ- ing the state, the states DOM in stripped and original version. Furthermore was the validation of the Application specific state-machine invariants included. These invariants were not included by the initial report plug-in proposed by [Roe10].
Additional third party support for invariant creation RQ5 concludes that the invariant creation gets stimulated when the web developer has a starting point with a few invariant examples. Therefore the invariant collector plug-in includes a help dialogue including the invariants of the case study replication as examples. During the case study replication were also the following tools and resources very helpful to create the XPath expressions and regular expression invariants.
• Regex101.com. A complete regular expression editor, including regular expression validation and test string-matching visualization. Regex101.com does also provide complete regular expression documen- tation and the ability to save templates for later use after account creation.
• XPath Helper. XPath Helper is Google Chrome Extension, which is open source and free to use. It support XPath validation, and also identification by holding Shift + Ctrl + X and selecting the desired element inside the browser.
• W3CSchool XPath documentation. The W3CSchools XPath documentation explains the main syntax and function of XPath including understandable examples.
These tools and documentation references were also linked and briefly explained in the Chrome extensions help dialogue to provide the web developer with a small tool box to get started.
22Crawljax-CLI https://github.com/crawljax/crawljax/tree/master/cli
52 5.10.4 CI integration The ATUSA stack with invariant collector is wrapped by the Guidion CLI application. The Guidion CLI application is integrated into the Jenkins server of Guidion. For this thesis is the Jenkins job not yet integrated into the development pipeline, due to server resource problems, but it is fully prepared for integration. This section describes the configured components in Jenkins for the Guidion CLI integration, report publication and development team notification.
ANT integration The Guidion CLI gets executed via a simple configured ANT task within Jenkins. The ANT task creates or cleans the required folder structure for the generated test suite, overview report and test report. It executes the Guidion CLI for the test subject system. Currently supported are only the logistics web application Cargo and the social network of Guidion.
Report publication For the report publication is the FTP publisher plug-in23 configured. The report gets uploaded to the web server of Guidion. The whole output of the Guidion CLI gets uploaded into a folder which includes the build number given by Jenkins. The build number identifies the build number with the report result.
Email notification The email notification plug-in email-ext24 is configured to inform the developers that a new test report was published.
Manual execution The integration in the development pipeline is prepared through the ANT script and the configured plug-ins. The single job can easily be integrated into the development pipeline with simply selecting the Jenkins job as downstream job after web application deployment to the test environment.
23FTP publisher plug-in https://wiki.jenkins-ci.org/display/JENKINS/FTP-Publisher+Plugin 24Email-ext plug-in https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin
53 5.11 Case study “Effectiveness of the integration”
The last case study conducted in the scope of this thesis, was the case study about the effectiveness of the integration of ATUSA with its additional features at Guidion. This case study provides more empirical evidence for the research questions RQ2 and RQ4, evaluates the proof of concept and provides evidence to answer RQ6 of this thesis.
Case study subject system - “Sonar” The subject system chosen for the case study is Guidions own developed social network platform called “Sonar”. Sonar is implemented in Wordpress25, a open- source PHP based blogging web application. The social networking components for Wordpress are provided through a plug-in suite called Buddypress26. Buddypress extends Wordpress with additional social networking features such as e.g. time-lines, posts, comments, likes and personal user profiles. Sonar is used to stimulate the communication between teams and the field force specialists in the Netherlands. At the moment of the case study, Guidion just enabled full access for the teams and the whole field force which are in total around 700 active users. For Sonar the Buddypress plug-in was customized to fit Guidions social network needs. All of these customizations were implemented using AngularJS27 with AJAX, the Buddypress and Wordpress REST API. During the conduction of the case study, Sonar was in a active development sprint and two web developers were involved in the development of new features. In addition to AngularJS used Sonar also the JavaScript libraries, JQuery28 and Twitter Bootstrap29. The active development sprint of Sonar involved in general several bug fixes, and performance tuning.
General case study configuration The basis configuration chosen for the case study, was the same default configuration, as used within the case study replications. The main difference in the configuration is that the configuration option clickOnce was set to true. So that every identified candidate clickable was clicked only once during the crawl process. Furthermore the random input generation was disabled and several input specifications were defined to reduce the state creation. These specifications included, selected filter tags, selected filter groups, test posts and comments. Also included in the general configuration were click exclusions rules to reduce the crawl time. These rules included exclusions for the top level navigation menu which included the log-out button, the breadcrumb menu and a notification sidebar which displays a list of the recent activities including links for users, post and comments. This configuration was set in the Guidion CLI application. The required manual effort for the configuration, defining input specifications and the click exclusion rules was 25 minutes.
General case study plug-ins Sonar uses a authentication via the Google plus SignIn API, there- fore the generic authentication plug-in was configured to use Google with specified email and password for authentication. The test user was extra created including additional test data. The additional plug-ins are the following.
• AuthenticationPlugin 7 minutes.
• AtusaTestGeneratorPlugin, for the test case generation.
• AtusaReportingPlugin, using all included generic invariants (e.g. DOM, back-tracking).
• Overview plug-in.
These plug-ins were set in the Guidion CLI application and the manual effort required to configure those was 9 minutes. 25Wordpress https://wordpress.org/ 26Buddypress https://buddypress.org/ 27Google AngularJS https://angularjs.org/ 28JQuery http://jquery.com/ 29Twitter Bootstrap http://getbootstrap.com/
54 Sonar oracle comparators Three oracle comparators were created for Sonar to reduce the identi- fied states by ATUSA. Sonar used a global search field with an auto complete feature, which requests possible search terms during typing. This search function can be used for searching of users, posts and articles and according to its search scope the list of search term suggestions changes (e.g. the list of users displays the user name and his profile picture and the article list only article headlines). The first oracle comparator the SearchListComparator targets the search fields suggestion lists and removes the list of every state. The following implemented and used oracle comparator is the PostComparator. The PostCompara- tor targets all post listings. These post listings are e.g. the time-line, all time-line related search results and the last posts displayed on the profile and group pages. A post in Sonar consists of the content, likes, reads and comments. The PostComparator uses a pre-condition which checks via XPath if the post container exists in the current state before applying the oracle comparator. The third oracle comparator implemented was the ActivityComparator. The ActivityComparator targets the activities in the Sonar side bar, which displays the last recent activities of all groups and posts according to the user rights. Furthermore displays the group overview page of a certain team or group a user belongs to also the last recent activities of the current displayed group. These activities are removed before state comparison. The ActivityComparator also uses pre-conditions to determine if the oracle comparator should be applied on the current state. The implemented comparators cover the dynamic parts of the test web application and can be used within the crawling process and are prepared for further regression testing. The data in the test database grows during every crawling process and also during manual testing through the developers and testers. These comparators strip the dynamic parts of Sonar before state comparison and the manual effort required to implement the described oracle comparators is listed below. • SearchListComparator 3 minutes. • PostComparator 5 minutes. • ActivityComparator 3 minutes. The manual effort to configure create these application specific invariants took 11 minutes.
Jenkins integration The ATUSA execution was configured for this case study as Jenkins job in a prepared test Jenkins server at Guidion. The Jenkins template was used to create the ATUSA test job for Sonar. Sonar had at the time of the case study not a existing deployment pipeline, therefore was the ATUSA job created to be executed manually after successful manual database and code deployment. The created single job can be simply connected to a later created deployment pipeline. The ATUSA test job used the features and limitations described in 5.10.4. The whole process was implemented using a ANT build script, which handles the output folder structure preparation and calls the Guidion CLI with the system -x sonar parameter. The parameter indicates that the Sonar configuration shall be used for ATUSA. After the ATUSA execution via the Guidion CLI and after publication of the generated test report, overview graph and test suite, an email is send as notification to the web developers.
Case study invariants creation design and implementation For creating the invariants, one web developer were asked to create application specific state invariants for Sonar during the following development sprint. These invariants were limited for the case study to six different state invariants to limit the required effort during development. The web developer was asked to create the invariants with the Chrome plug-in which was installed in the developers browser. In addition to the installation was the plug-in and the web front-end with example invariants explained in detail. To cover all main states of Sonar, the web developer was asked to create invariants also for the main states, e.g. groups, profiles and time-line. Sonar was during the case study in a active development sprint. Both web developers which were assigned to the project were involved in the development of the features but only one created the invariants. Therefore the one web developer had good knowledge about the developed features and improvements. In addition to the invariant creation via the invariant collector Chrome plug-in or web front-end was the web developer asked to measure and document the time required to create those invariants.
55 Case study scenario preparation The case study preparation followed the steps listed below.
1. Configuring ATUSA including input specifications and click rules.
2. Configuring the AtusaReportPlugin with all activated generic invariants, the AtusaTestSuitePlugin, Overview plug-in and Authentication plug-in.
3. Preparing Guidion CLI.
4. Setting defined for the Guidion CLI. Including the AtusaReportPlugin with all activated generic invari- ants. Additionally the AtusaTestSuitePlugin and the Overview plug-in.
5. Creating and configuring the Jenkins job, with all plug-ins activated (see section 5.10.4) and the ANT build script.
Case study scenario The conducted case study scenario followed the steps listed below.
1. Invariant delivery via the web developer during active development.
2. Running the prepared Jenkins job after sprint completion and successful code and database deployment.
3. Executing (automatically) the Guidion CLI via the Jenkins job.
4. Executing (manually) the generated test suite including JSCover for client-side code coverage measure- ment.
After execution of the ATUSA crawling and plug-ins invocation, the generated report was uploaded via the Jenkins FTP plug-in to the Guidion test web server. The generated test suite was also transferred as backup. The build script of the job cleans the local build folder before execution of the Guidion CLI. After completion of the job including publication, a notification email about a new published test report was send. For more insight about the client-side coverage of ATUSA the generated test suite was manually executed with JSCover activated.
Created application specific invariants The web developer created the following six application specific invariants via the invariant collector Chrome plug-in for testing Sonar.
• GSI1 - XPath condition invariant, time-line item design decision contract.
• GSI2 - Not http://localhost link regular expression invariant. A common error during the last deployments were forgotten changes from static test URL’s to dynamic ones.
• GSI3 - Not 404 error, not 504 error, not empty list message, regular expression invariant.
• GSI4 - Groups state regular expression template, including data validation.
• GSI5 - Profile state regular expression template, including data validation.
• GSI6 - Visible condition invariant for the user time-line. The invariant validates if a time-line list identified by UL [@class= “cpb tmtimeline list” ] is fully loaded within 1000 milliseconds.
The web developer documented only the total manual effort required to create the invariants via the invariant collector Chrome plug-in, which was 41 minutes. The time measurement provided by the web developer included the invariant expression and identification.
Included generic invariants The configured AtusaReportingPlugin provided the following generic invariants and included their result in the generated output report.
• Error detector generic invariant.
• DOM validator generic invariant.
• Back-tracking generic state-machine invariant.
• Dead clickable generic state-machine invariant.
56 Identified errors ATUSA reported the following errors:
• GSI3 - The message detector invariant identified two different errors during the crawling process. The first one is that every group state has a change user button, for changing users from the current group. Clicking this button leads to a 404 error page. The reason for this error was that the button was shown without user permission validation. After clicking the permissions are checked and result in a 404 error page. This error was already known within Guidion, but was not fixed during the sprint. The second identified error was that after displaying more posts for a filter tag filtered time-line the result is a empty time-line list, which was not allowed. This error was not known before at Guidion, not a tester or a user has earlier reported this error. The error can be hard to identify by the tester because of the dependent stateful web application state.
• DOM violations - Several DOM violations were identified. Most of the identified errors are Angu- larJS related. AngularJS requires additional HTML attributes and allows their declaration in several forms, e.g. ”ng-controller” to specify the usage of a AngularJS controller. For HTML5 conformity these attributes need to be declared as ”data-ng-controller”. This was not earlier known and HTML conformity was not earlier validated at Guidions web applications. The other DOM violations found were not allowed element combinations e.g. span element is not allowed as children of list elements.
• GSI4 - The groups regular expression template invariant identified a counter panel error on several group states. The groups page includes a counter panel which shows a animated counter for post and a counter for articles of the group. If a group has zero articles or posts the zero is not shown as expected. The error occurred because of coding error in the counters animation function.
• Inconsistent back-tracking Sonar uses a inconsistent back-tracking. Several states lead to another state than expected during the navigation through the history. AngularJS supports the history registra- tion, the states with the incorrect back-tracking occurred because of the missing registration of selected values within the AngularJS based implementations.
Findings and conclusion Based on the observations made during the case study we concluded that:
• RQ2 - The delivered application specific invariants were able to detect unknown errors (GSI3 and GSI4). These errors were not included in the web applications Jira bug-tracking system. ATUSA and the generated test suite covered 54% of the developed JavaScript code of Sonar. A simple analysis of the coverage result and the uncovered functions revealed that several functions have no usage and can be removed. The generic invariants revealed errors back-tracking and several DOM errors. These functional and non-functional attributes were not earlier validated within Guidion. In addition to coverage and identified errors ATUSA was able to detect a average DOM size of 274.181 Kb, which is in the average range of enterprise applications according [MvDR12].
• RQ4 - The required manual effort to configure ATUSA for testing Sonar and the usage through the Guidion CLI was 45 minutes. The configuration and the Guidion CLI can be used as basis for further testing approaches. The separation of the complete configuration would enable the current ATUSA stack usage without re-compilation after the configuration changes. The creation and expression of the invariants via the invariant collector required 41 minutes. The invariant collector enabled the web developer to express invariants for ATUSA without Java knowledge. The invariant collector completely separates the invariant creation and usage and enables the invariant delivery without recompilation of the ATUSA configuration. The integration of the Guidion CLI in a sample Jenkins job enables the web developers to execute the ATUSA testing phase per one click without any additional local Java command line commands.
• RQ6 - ATUSA with the invariant collector addition and its sample integration in Jenkins build a effective basis for testing the Guidion web applications. ATUSA is capable to test the web applications with generic and application specific invariants provided via the invariant collector. These invariants help to validate developed web application functionality and help to identify unknown web applications errors. The generated report helps in the reproduction of errors, in fixing errors and in the validation of the correctness of the reported errors and invariants. The generated test suite builds a good basis for further regression testing and cross browser compatibility testing which was not given at Guidion. The included JSCover measurement helps in identifying unused functions and provides additional insights of the used JavaScript code. The Jenkins template and proof of concept job enables Guidion to integrate
57 the ATUSA testing phase in a deployment pipeline, to execute ATUSA automatically. The invariant collector Chrome plug-in and sever enable the invariant creation by the web developer. Its integration in the natural development process of the web developer, which stimulates the creation of invariants. These invariants can be also used for validating loading times, which were not earlier tested at Guidion.
58 Chapter 6
Analysis and Conclusions
This chapter summarizes the research questions and their answers, states the threats to validity, future work and ends with a final conclusion about ATUSA. This thesis replicated the research of Mesbah et al. [MvDR12] and investigated the use of ATUSA and its possibility of automated invariant based testing of modern web applications. ATUSA uses as basis the open-source crawler Crawljax with additional plug-ins, which needed to be reimplemented to conduct the case study replications. This included the test-suite generator plug-in as well as the advanced error reporting plug-in. The ATUSA related plug-ins are publicly available as contribution of this thesis via Bitbucket1 for everybody who is interested to use the ATUSA stack free to use. Furthermore this thesis includes research about a simplified way to provide invariants for ATUSA. The results of the research form a prototype to enable a web developer to deliver application specific invariants for ATUSA. The final case study uses the prototype and ATUSA in a Jenkins job to test its effectiveness by testing the Guidion social network web application Sonar.
6.1 Conclusion
Required Crawljax knowledge The plug-in and oracle comparator implementation required good knowledge of the implementation of Crawljax. This could be only acknowledged through the study of the implementation itself, the implemented tests and the API documentation. Unfortunately the documentation was not always helpful and the actual implementation needed to be analyzed.
RQ1 - Invariants The invariant creation process was separated from the invariant configuration for ATUSA. The invariant collector Chrome plug-in and the web front-end centralize the invariant persistence per test system. The web developers can deliver invariants without Java knowledge on two ways. The created invariants add additional application specific documentation and can be helpful for the web developers to get inspired for new invariants. The Chrome plug-in integrates the invariant creation process in the actual web developer process. This enables the web developer to create invariants, after manual local testing of a new feature.
RQ2 - Fault revealing capacities The invariants created within the replication were able to find the faults reported in the original study [MvDR12]. In addition the created invariants were also able to find unknown errors in the web applications of Guidion. The approach of invariant based testing with ATUSA and its provided generic invariants build a basis for testing general behaviour and testing the web application against standards. The application specific invariants were useful in detecting application specific faults. ATUSA is suited for testing web applications as a part of a testing strategy, it does not substitute other testing approaches like functional testing of server side functionality.
RQ3 - Performance and Scalability ATUSA provides configurable options to reduce the risk to reach the well known state space explosion problem. The options to reduce the state space are
1Thesis contribution repository: https://bitbucket.org/sroh/atusa-plugins/
59 input specifications, oracle comparators and crawl and click rules. The most effective options are the crawl rules e.g. crawl depth and maximum states and the oracle comparators, which are used to strip dynamic parts of the state before state comparison. For improving the performance Crawljax provides a multi-threaded crawling option. This option was set to one, because of the thread unsafe oracle comparators. In newer versions with the thread safe DOM strippers and the multi thread crawling option of Crawljax can increase the crawling performance. Another important factor is the DOM-tree size, Crawljax analyzes every new state DOM for candidate clickables, a DOM-tree with more elements and a size of e.g. 0.25 Mb. requires more time to be fully analyzed than a DOM-tree with fewer elements and a size of 0.48 Kb. Another performance impact was introduced through the W3C DOM validation functionality. The plug-in uses the W3C validator web service and validates every state DOM and reports its violations. For a huge amount of different state DOMs the validation process can take several minutes.
RQ4 - Automation scope The automation scope finding of the initial research [MvDR12] in section 9.1 was confirmed through this thesis. ATUSA augments the manual user testing approach. The user testing approach can also validate the correctness of the visible user interface functionality before and after clicking them, e.g. correct toggle of table elements or display of user activity items. ATUSA focusses on finding errors manifested in the DOM tree or in the navigation of the inferred state-machine. The ATUSA approach is capable to identify these errors.
RQ4 - Manual effort The manual effort measurement included the same steps as described in the initial research [MvDR12]. These steps consist of Crawljax crawl configuration, plug-in development and configuration, crawl rules specification, oracle comparator implementation and configuration, invariant expression and input specification. The separation of the invariant expression process as Chrome plug-in enables the web developer to create invariants during the development of new features. The provided help functions, tools and the browsable invariant collector invariant stimulates the invariant creation process. Both the invariants and the invariant collector simplify or enable the invariant for the web developer. The provided invariant collector extension also reduces manual effort of building, packaging and deploying a new ATUSA configured version jar.
Testing strategies ATUSA is capable to be used as a simple smoke testing tool. The reimplemented test suite generation enables ATUSA to provide a basis for different testing strategies, e.g. regression testing or selective regression testing. To the best of our knowledge ATUSA is still the only open- source crawler which generates a test suite for further testing. The generated test suite was used in three different conducted case studies of this thesis and can be used for regression testing, cross browser testing and selective regression testing. The generated test suite implementation has its limitations but forms a good basis for further research in the field of automated regression testing. Furthermore the generated test suite builds a good basis to create a selective regression test suite where only important paths of the web application are included and the others are removed.
6.2 Threats to validity
Several factors where identified as threats to validity if this research would be replicated. These threats are divided into internal and external threats and described in the following subsections.
6.2.1 Internal threats to validity Application specific state machine invariants Application specific state invariants are only described as constraints on state complete inferred state flow graph. As state flow graph transition validations with the following pattern, “From state” → “Eventable / Edge” → “Target state”. The application specific state machine invariant plug-ins, are developed as described in the initial research, but without a definitive code example or guideline this part is declared as a thread to validity.
60 Different configuration As described earlier, the configuration of Crawljax has a significant impact on the crawl result. The thesis provides per case study a detailed list of the used configuration to avoid configuration based changes.
Different Crawljax version As an active open-source project Crawljax will be improved further, using Crawljax with improved features, e.g. a different crawl strategy or a improved state comparison algorithm, can have impact on the crawl results.
Different dependency versions Besides the different version of Crawljax, also the dependencies can change. One example is the used dependency XMLUnit. The library dependency was updated during this project. The newer version also considered ordering of the attributes within a element as difference, where the old version ignored the ordering. This had an impact on the state creation through Crawljax. Further improvements can change the crawl and test results.
6.2.2 External threats to validity Provided invariants The invariants created for the case study replication only evaluate their correctness, expression and result after use. These invariants were not extended to provide additional evidence for the replications.
Different web application for the real bugs case study replication The web application used for the real bugs case study replication in section 5.8 is a Guidion own web application with closed source. Running the same real bugs case study requires a different test subject system. This will produce a different outcome for the real bugs case study replication.
Different web application for the effectiveness case study replication The web application used for the effectiveness case study replication in section 5.11 uses a Guidion own web application with closed source. Running the same real study requires a different test subject system. This will produce a different outcome for the case study replication.
6.3 Contributions
1. Reimplemented and re-engineered ATUSA plug-ins (ATUSA stack). This includes the AtusaReport- Plugin which combines the used generic invariants such as DOM validator and Back-tracking and their reporting. Furthermore is the AtusaTestSuitePlugin included, which includes the test suite generator, test suite execution and the JSCover coverage measurement. The whole ATUSA plug-in stack is public available at Bitbucket [Roh15].
2. Reimplemented reporting functionality with additional features, see section 5.10.1. This is a plug-in which provides a generic and extendible reporting function. The reporting function is used to report on failed events, error states and violated application specific invariants. The report plug-in also includes the whole reporting from the generated test suites. The reporting plug-in and its full stack of features is also publicly available at Bitbucket [Roh15].
3. Replication of the case studies. The case studies conducted by Mesbah et al. in [MvDR12] were completely replicated and their results are included within this thesis. This supports the validity of the research by Mesbah et al. and the testing approach with ATUSA.
4. Delivering additional empirical evidence for the research [MvDR12] and the ATUSA approach by Mes- bah et al., through the test subject change of the third case study replication and the last case study conducted within the scope of this thesis.
5. Deployed proof of concept tool at Guidion. The proposed approach with the invariant collector and provider as well as the Guidion CLI is integrated in a small Continuous integration pipeline and is ready to use for further web application testing. The tools were presented and web developers are able to create invariants for testing their developed features.
61 6. Invariant creation support for web developers. The identified problem that web developers are not able to deliver invariants without Java knowledge was fixed, through the proposed invariant collector and provider. These two manners help the web developer in a simple way to create invariants integrated in their development process.
6.4 Future Work
CLI fully configurable The CLI uses a set of predefined configurations for the Guidion web application Cargo and Sonar. For reducing the required Java knowledge the configuration needs to be fully separated from the code, e.g. an XML based configuration as used for the test execution.
Update ATUSA stack to support the upcoming Crawljax-4.0 release Crawljax 4.0 intro- duces the thread safe substitution for the oracle comparator, the so called DOM-stripper. For enjoying the full benefits of the DOM-strippers and further Crawljax related updates, the current ATUSA stack needs to be updated to support these features.
Enable use of Graph-Database A graph database can be used for crawling and additional ap- plication specific state-machine validation. Furthermore an inferred state flow graph can be queried, validated and compared with earlier or newer versions of the graph.
Regression testing Further research can be conducted in the field of regression testing with ATUSA as described in [RMvD10]. The contributed implementation supports further research.
Improvements invariant collector (Chrome plug-in) The current version deployed at Guidion is a simple create extension, with a configuration page and a page to create invariants. For avoiding switching to the web based version to maintain invariants, the extension needs to be updated to support the same functionalities.
Reimplement Crawljax core in Scala Crawljax is implemented in Java. During the last years additional programming languages were introduced which are also able to run in the Java Virtual Machine (JVM). One of this programming language is Scala. During tests with the current release of Crawljax, the core multi-threaded crawling mechanism can be substituted through a Scala based multi threaded implementation.
Basis recreated for further replications of Crawljax related research The ATUSA approach is further used in other research by Mesbah et al. [MvDR12]. Through the reimplementation of the ATUSA stack these research can be replicated with help of ATUSA. Future work can be to replicate these research in the field of regression testing, cross browser compatibility testing or JavaScript mutation testing.
62 Bibliography
[AS12] Anuja Arora and Madhavi Sinha. Web Application Testing: A Review on Techniques, Tools and State of Art. International Journal of Scientific and Engineering Research, 3(2):1–6, February 2012. [AS13] Anuja Arora and Madhavi Sinha. Avoiding State Explosion Problem of Generated AJAX Web Application State Machine Using BDD. In IC3, pages 381–386, 2013. doi:10.1109/IC3.2013.6612224. [Ber05] Sebastian Bergmann. PHPUnit Pocket Guide. O’Reilly Media, 1st edition, 2005. [BMvD09] Cor-Paul Bezemer, Ali Mesbah, and Arie van Deursen. Automated Security Testing of Web Widget Interactions. In ESEC/SIGSOFT FSE, pages 81–90, 2009. doi:10.1145/ 1595696.1595711. [Bur03] Thomas F. Burgess. A general introduction to the design of questionnaires for survey research, May 2003. URL: http://iss.leeds.ac.uk/downloads/top2.pdf. [Car10] Jeffrey C. Carver. Towards Reporting Guidelines for Experimental Replications: A Proposal. In Proceedings of the International Workshop on Replication in Empirical Software Engineering Research, RESER, May 2010. [Cra12a] Crawljax error report plugin Github repository, 2012. URL: https://github.com/ crawljax/errorreport-plugin. [Cra12b] Crawljax test case generator plugin Github repository, 2012. URL: https://github. com/crawljax/testcasegenerator-plugin. [Cra13] Crawljax console interface (CLI) Github repository, 2013. URL: https://github.com/ crawljax/crawljax/tree/master/cli. [Cra14] Crawljax version: 3.5.1 Github repository, 2014. URL: https://github.com/ crawljax/crawljax/releases/tag/crawljax-3.5.1. [EPG+07] Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. The Daikon system for dynamic detection of likely invariants. Sci. Comput. Program., 69(1-3):35–45, 2007. doi:10.1016/j.scico. 2007.01.015. [Gar05] Jesse James Garret. Ajax: A New Approach to Web Applications, February 2005. URL: http://www.adaptivepath.com/ideas/ajax-new-approach-web-applications/. [GMBCM13] Vahid Garousi, Ali Mesbah, Aysu Betin-Can, and Shabnam Mirshokraie. A System- atic Mapping Study of Web Application Testing. Information & Software Technology, 55(8):1374–1396, 2013. doi:10.1016/j.infsof.2013.02.006. [GMvD10] Frank Groeneveld, Ali Mesbah, and Arie van Deursen. Automatic Invariant Detection in Dynamic Web Applications. Software Engineering Research Group Technical Report Series 37, Delft University of Technology, 2010. URL: http://resolver.tudelft.nl/ uuid:df700052-3b1b-4e5c-98f1-5538138e44af.
63 [HF10] Jez Humble and David Farley. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation. Addison-Wesley Professional, 1st edition, 2010.
[JGr] JGraphT, A Open-Source Java Graph library. URL: http://jgrapht.org/. [MBvD08] Ali Mesbah, Engin Bozdag, and Arie van Deursen. Crawling AJAX by Inferring User Interface State Changes. In ICWE, pages 122–134, 2008. doi:10.1109/ICWE.2008.24. [Mes09] Ali Mesbah. Analysis and Testing of AJAX-based Single-page Web Applications. Phd thesis, Delft University of Technology, Delft, The Netherlands, 2009. URL: http: //www.st.ewi.tudelft.nl/~arie/phds/Mesbah.pdf. [MRT08] Alessandro Marchetto, Filippo Ricca, and Paolo Tonella. A Case Study-based Compar- ison of Web Testing Techniques Applied to AJAX Web Applications. STTT, 10(6):477– 492, 2008. doi:10.1007/s10009-008-0086-x. [MvD07] Ali Mesbah and Arie van Deursen. An Architectural Style for AJAX. In WICSA, page 9, 2007. doi:10.1109/WICSA.2007.7. [MvD09] Ali Mesbah and Arie van Deursen. Invariant-based Automatic Testing of AJAX User Interfaces. In ICSE, pages 210–220, 2009. doi:10.1109/ICSE.2009.5070522. [MvDL12] Ali Mesbah, Arie van Deursen, and Stefan Lenselink. Crawling AJAX-Based Web Applications through Dynamic Analysis of User Interface State Changes. TWEB, 6(1):3, 2012. doi:10.1145/2109205.2109208. [MvDR12] Ali Mesbah, Arie van Deursen, and Danny Roest. Invariant-Based Automatic Testing of Modern Web Applications. IEEE Trans. Software Eng., 38(1):35–53, 2012. doi: 10.1109/TSE.2011.28. [NMvD14] Alex Nederlof, Ali Mesbah, and Arie van Deursen. Software Engineering for the Web: The State of the Practice. In ICSE Companion, pages 4–13, 2014. doi:10.1145/ 2591062.2591170. [PZ10] Karthik Pattabiraman and Benjamin G. Zorn. Dodom: Leveraging DOM invariants for web 2.0 application robustness testing. In IEEE 21st International Symposium on Software Reliability Engineering, ISSRE 2010, San Jose, CA, USA, 1-4 November 2010, pages 191–200, 2010. doi:10.1109/ISSRE.2010.17. [RMvD10] Danny Roest, Ali Mesbah, and Arie van Deursen. Regression Testing AJAX Applica- tions: Coping with Dynamism. In ICST, pages 127–136, 2010. doi:10.1109/ICST. 2010.59. [Roe10] Danny Roest. Automated Regression Testing of AJAX Web Applications. Master’s thesis, Delft University of Technology, Delft, The Netherlands, 2010. URL: http: //resolver.tudelft.nl/uuid:1b97ab0f-03ac-4199-b2d0-183f34a5d624. [Roh15] Sven Rohde. ATUSA plug-ins, thesis contribution repository, 2015. URL: https: //bitbucket.org/sroh/atusa-plugins/. [Sma11] John Ferguson Smart. Jenkins: The Definitive Guide. O’Reilly Media, 1st edition, 2011. [vDM10] Arie van Deursen and Ali Mesbah. Research Issues in the Automated Testing of AJAX Applications. In SOFSEM, pages 16–28, 2010. doi:10.1007/978-3-642-11266-9_2. [Yin02] Robert K. Yin. Case Study Research: Design and Methods. SAGE Publications, Inc, 3rd edition, 2002.
64 Chapter 7
Appendix
7.1 Questionnaire
The Questionnaire is included in the following four pages.
65 Web developer survey Master thesis of Sven Rohde
Web developer survey
Please choose one or more answers per question. Please mark them with an X or Circle. If you have any questions, please ask the survey observer.
Question 1: What for editors or integrated development environments (IDE‘s) do you use? ______
Question 2 (Multiple answers are possible): What kind of JavaScript libraries do you use in your projects?
[ ] – JQuery [ ] – Others: ______[ ] – Bootstrap ______[ ] – AngularJS ______
Question 3 (Multiple answers are possible): What sort of testing do you use during development, to test and validate your JavaScript functionality?
[ ] – JSUnit (or other JavaScript unit test framework). [ ] – Selenium (e.g. recorded user behavior simulation) [ ] – Manual testing via your browser (Click, View, Validate with Chrome). [ ] – Other: ______
Question 4: What for JavaScript event handlers do you mainly use?
Question 5: Do you map events in JavaScript files or directly in the HTML tags?
Question 6: Which HTML elements do you prefer for mapping JavaScript events? (E.g. , ,