Automated Acceptance Testing Tools for Web Applications Using Test-Driven Development
Total Page:16
File Type:pdf, Size:1020Kb
Marian JURECZKO1, Michal MLYNARSKI2 Technical University of Wroclaw (1), University of Paderborn (2) Automated acceptance testing tools for web applications using Test-Driven Development Abstract. In the last years the software engineering community pays a strong interest in agile development methods. Those methods place software testing for example the Test-Driven Development method as an important task of the development process. Agile projects rely on good test automation tools. In this paper we evaluate five test automation tools for their usage in acceptance testing for web applications using Test-Driven Development. Streszczenie. Rozdział prezentuje analizę porównawczą wybranych narzędzi umożliwiających tworzenie zautomatyzowanych testów akceptacyjnych, czyli Fitnesse, Proven!, jfcUnit oraz Selenium. Badana jest możliwość stosowania tych narzędzi do testowania aplikacji internetowych wytwarzanych w procesie opierającym się o podejście programowania przez testy. Aplikacje internetowe są bardzo specyficzną grupą aplikacji. Do ich uruchomienia potrzeba serwera aplikacji. W związku z tym nie każda aplikacja nadaje się do przeprowadzania na nich testów akceptacyjnych. Dodatkowe komplikacje pojawiają się, jeżeli wymagać, aby testy powstały przed napisaniem kodu źródłowego, co ma miejsce w przypadku programowaniu przez testy. (Zautomatyzowane testy akceptacyjne dla aplikacji internetowych w programowaniu sterowanym testami) Keywords: test-driven development, test automation tools, acceptance testing, web application testing. Słowa kluczowe: programowanie przez testy, automatyzacja testow, testy akceptacyjne, testowanie aplikacji webowych. Introduction As shown in Fig. we first choose a tool and create test Nowadays software engineering pays more attention to cases with the test automation tool. Then we implement the so-called agile methods. Those methods place automated code according to the test case definition. After step 1) we tests as one of the important tasks in software engineering. answer the questions defined for all subgoals as described One of the famous agile methods is test-driven earlier. To give a better understanding of our evaluation development (TDD), where code is implemented according process we first describe the TDD-like activities of step 1) to prior created test cases. The industrial practice shows, and then briefly introduce our goals, questions and value that test automation is not a trivial task. Especially in TDD ranges for step 2). the popular test automation tools using the Capture&Replay method, where the interaction between the user and system under test (SUT) is recorded, cannot be applied because the system is not implemented at the time of test case design. Different application types need different test automation methods. For example the tests of web applications are hard to automate because of the complex architecture and several technologies used within it. Speaking of acceptance tests, a web application has to be tested in a customer-like environment. This means, that the application has to be deployed on an application server. In order to conduct acceptance tests in a test-driven development project, the test automation tool issue has to be clarified. The goal of this article is to evaluate five test automation tools for their usage in acceptance tests conducted with TDD. We use a GQM-like method, where tools are evaluated according to predefined goals. For each tool, we first create a test case and then implement code for a small user story example based on the Spring framework. This article begins with the description of the evaluation process. Then we introduce our running example, selected tools and we evaluate each tool according to the evaluation Fig. 1. Tool Evaluation Process process. At the end we summarize the evaluation and Test-Driven Development as described in [2] and the select the most appropriate test automation tool. movement of Extreme Programming [3-4] gains high Evaluation process attention in the software engineering community. Different as in typical iterative development models, the test activities As mentioned in the introduction we define a clear start before and not after the code is implemented. As test evaluation process for the tool comparison. For this purpose cases are used as the specification of the system, the first we first adjust the Goal Question Metric (GQM) introduced activity in the development process is test case design. in [1]. We define the main goal and several subgoals. For Next, code needed to pass the created test case is being each subgoal we define questions which have to be implemented. If needed a refactoring of the created code answered. Each question has to be answered within a can be conducted in the last step. predefined value range. Different as in GQM we do not use According to [14] TDD can be used on two test levels. The concrete metrics but predefined answer values of the first one is unit testing, where test cases are mostly created questions for each subgoal. with tools from the xUnit family. The second and important The overall evaluation process is divided into three parts: 1) for this article test level is acceptance testing. Based on tool usage 2) question answering and 3) tool comparison. 198 PRZEGLĄD ELEKTROTECHNICZNY (Electrical Review), ISSN 0033-2097, R. 86 NR 9/2010 agile models like [5-6] we define the following activities executing steps 1) and 2) for all selected tools, we compare needed to conduct TDD-like acceptance tests: them according to the values defined for each answer. 1. Select and understand a User Story for the implemented system. Running example 2. Create new automated test cases and add to existing For our evaluation, we choose the wide-known web test suites. application framework Spring. The framework authors 3. Execute test cases. According to the red-green bar provide documentation, where a very basic inventory patterns [2], these test cases should fail. As the functionality management system can be implemented with it [13]. For needed to fulfil the selected User Story has not been this article we use the Change Price user story. It can be implemented, the test cases cannot pass. described as follows: 4. Implement the User Story functionality needed to pass A user is able to change the price of a selected product by the created test cases. providing the percent change value. If the price could be 5. If some alternative scenarios of the User Story can be changed, a message with the new price should be shown. If identified, then activities 2-4 should be reexecuted. the user provides negative or null values, then an error In Fig. 1 we summarize those activities with Create test message should be shown. case and Implement code. Knowing how the first step of our Knowing what use case has to be implemented we briefly evaluation process is done, we now show what questions introduce the tools chosen for our evaluation process. have to be answered in order to evaluate a test automation Evaluated tools tool. There are several test automation tools on the market. The As mentioned earlier, our main goal is to evaluate test evaluation of all available tools is not the scope of this work. automation tools for their usage in TDD-like acceptance We have chosen five tools, which can be categorized in testing. Based on the literature [2,3,5,6] and our open-source and proprietary. From the fist category we observations, we define the following subgoals G1-G10, selected the following tools: their appropriate questions Q and answers A: JFCUnit [7] is a JUnit extension for testing graphical user G1. Test first interfaces. Through an XML-based engine it is possible to Q: Can the automation tool be used before or after the capture and replay the needed test cases. Additionally test code is implemented? cases can also be implemented within XML or Java. A: easy, difficult, not possible JFCUnit is restricted to testing GUIs based on Java Swing. G2. SUT type Selenium [8] can be used as a browser plug-in for Q: Can the automation tool be applied for web Capture&Replay. The Selenium API supports the manual applications? implementation of test cases in several programming A: web application, non web application languages. Further, it supports the transformation of G3. Test environment recorded test cases into predefined languages. This way Q: Can the created test cases be executed in a client- the integration with for example JUnit is possible. like environment? FitNesse [11] is a test automation tool based upon the Fit A: client-like, development-like test framework as described in [12]. It offers all features G4. Development integration known from Fit and enhances it with self-editable wiki pages Q: Can the automation tool be integrated in the for test case specifications. Typically test cases created development environment? with FitNesse use directly the application business logic A: possible, not possible omitting the GUI. The test cases are specified with tables G5. Test implementation representing all user action steps. Through several Q: How test cases for GUI tests are created? extensions called fixtures, it is possible to use the GUI or A: coding, Capture&Replay even the underlying database in test case specifications. G6. GUI usage AutoIt [9] is a BASIC-like scripting language designed for Q: Does the automation tool use id´s or position automating the Windows GUI. It uses a combination of coordinates of GUI elements? simulated keystrokes, mouse movement and A: id-based, position-based window/control manipulation. Unfortunately AutoIt supports G7. Database interaction neither web applications nor Java Swing programs but on Q: Can the automation tool interact with the database? the other hand it is very popular and the application of A: possible, possible through integration with other tools, AutoIt in the context of test-driven development may be not possible very interesting. G8. Error checking For our evaluation we also selected one proprietary tool Q: Can the automation tool access and check error called Proven! [10].