Highly Automated Agile Testing Process: an Industrial Case Study
Total Page:16
File Type:pdf, Size:1020Kb
e-Informatica Software Engineering Journal, Volume 10, Issue 1, 2016, pages: 69–87, DOI 10.5277/e-Inf160104 Highly Automated Agile Testing Process: An Industrial Case Study Jarosław Berłowskia, Patryk Chruściela, Marcin Kasprzyka, Iwona Konanieca, Marian Jureczkob aNetworkedAssets Sp. z o. o. bFaculty of Computer Science and Management, Wrocław University of Science and Technology [email protected] Abstract This paper presents a description of an agile testing process in a medium size software project that is developed using Scrum. The research methods used is the case study were as follows: surveys, quantifiable project data sources and qualitative project members opinions were used for data collection. Challenges related to the testing process regarding a complex project environment and unscheduled releases were identified. Based on the obtained results, we concluded that the described approach addresses well the aforementioned issues. Therefore, recommendations were made with regard to the employed principles of agility, specifically: continuous integration, responding to change, test automation and test driven development. Furthermore, an efficient testing environment that combines a number of test frameworks (e.g. JUnit, Selenium, Jersey Test) with custom-developed simulators is presented. Keywords: software engineering, testing process, agile software development, case study 1. Introduction bility [5], the role of testers [6] or test automa- tion [7]. Software testing is a very costly part of the soft- The goal of this research is to extend the body ware development process, it is sometimes esti- of knowledge concerning agile testing by docu- mated to make 50% of the whole development menting a real life software testing process. This cost [1], [2]. It is one of the main activities (at paper presents a case study of a medium-size soft- least should be) in agile software development ware project with special factors that affects the methods. Beck and Andres [3] claimed it to be aforementioned process, i.e. requests for unsched- a measure of project progress and the main mean uled releases and high complexity of project envi- of assuring software quality. On the other hand, ronment. The project is a Java Enterprise system applying agile software development methods sig- in the telecommunication domain and its main nificantly affects the testing process. The agile purpose is to ease network devices controlling and methods usually require to test early and to have management. Thus, there is a number of features the tests automated. In consequence, testing is that concern integration and communication with not left to the final phase, but requires sustain- other systems and devices. Functional tests of able investments during the whole process (in- such features involve items that are outside of the cluding after-release maintenance of automated tested system, but are necessary for a successful test cases). Some of the authors even recommend test execution. Therefore, the test environment to start with testing [4]. The agile testing has is complex and its management can consume been studied for several years, however, there a considerable amount of resources, specifically are still unanswered questions regarding scala- in the case of automated tests, where all changes 70 Jarosław Berłowski et al. that come from test execution should be verified size, namely the number of Lines of Code (LOC). and reverted in order to ensure tests repeatability. The project investigated here is described as On the other hand there are many, unscheduled a medium-size software project. The small, release requests that concern the newest version medium and large scale is not a precise term, it of the developed system. Those releases regard is very intuitive (e.g. more intuitive than LOC). presentations for end users or deployments into The investigated project has been classified as an environment for acceptance tests. Nonetheless, a medium-size one, since it seems to be a bit the released version must satisfy company quality smaller than projects described in [5] and [6] standards and must be ‘potentially shippable’ [8]. which are reported as large ones. The authors The unscheduled releases could be explained as refer to costs of development and test. This releases in the middle of a Sprint (the project should be considered as the amount of time com- is developed using Scrum), that are announced mitted by the development team during project one or two days in advance and are requested related activities. One of the investigated aspects to contain some features from the current Sprint of the testing process is availableness for un- and, of course, all the features developed in pre- scheduled release. A possibility of making an vious Sprints. The unscheduled releases create application release is evaluated using results of challenges. It is critical to ensure that the ‘old’ the functional tests (Selenium tests and REST functionality still works correctly and there are tests) executed in a continuous integration system very limited resources for regression tests, since (100% success of unit tests is a prerequisite of the there are the ‘in-progress’ features that must be functional ones). The application is truly ready straightened up before the release. The solution for release if 100% of the functional tests are is somewhat obvious – test automation. In order passed. The application fits for a presentation in to support the unscheduled releases a complex the case when at least 90% of the tests are passed. set of automated regression tests must be present If more than 10% of the tests failed, no form and frequently executed. In consequence, it is pos- of release shall be considered. The investigated sible to evaluate the current version of software testing process was assessed using the concept of instantly and decide if it is ‘potentially shippable’. the level of agility that evaluates the adoption The paper describes how the challenges are sat- of different agile principles using the concept of isfied, in which ways they affected the testing the level of adoption. Both aforementioned process and what results were obtained. concepts are borrowed from [5], further details The case study presents an insider perspec- about them can also be found in section 2.3. tive, i.e. it is conducted by members of the team The rest of this paper is organised as follows. that develops the project. Therefore, it is possible The next section presents the goal and study to get a really deep insight, but unfortunately design as well as a detailed description of the the risk of biased observation or conclusion grows study context. The obtained results are docu- simultaneously. In consequence, the case study mented in section 3. In section 4, the threats to is focused on a quantitative approach, which is validity are discussed and in section 5, related associated with lower risk of subjectivity instead work is summarized. Finally, the discussion and of the qualitative one, which is usually employed conclusions are given in section 6. in similar case studies. The study is designed according to guidelines given by Runeson and Höst [9]. 2. Goal and Study Design A number of concepts is used in this paper. Let us present explanations of them in order to 2.1. Context clarify possible ambiguities. The authors refer to project size. The concept is in fact the metric The context is described according to the guide- which Mall [10] identified as one of the two most lines given by Petersen and Wohlin [11]. It corre- popular and the simplest measures of project sponds with the suggested description structure Highly Automated Agile Testing Process: An Industrial Case Study 71 as well as the recommended elements of the con- Customisation. It is custom software develop- text. ment. The product is tailored to the requirements of a customer (a telecommunication vendor). 2.1.1. Product Programming language. The system is devel- oped mostly in Java. The investigated project regards the development of a system which is used in the management of 2.1.2. Processes telecommunication services delivery. The main functionalities of the product are: Figure 1 presents the testing process. In fact, no – managing customer access devices, defined process is used, and what the diagram – managing and configuring network delivery shows is the result of the ‘definition of done’ devices, that is being employed. New user stories are – managing resources of IP addresses from the first identified, then acceptance criteria for them public and private pools. are defined and stories are estimated. When the The following technologies are used in the soft- Sprint begins, the development team makes the ware development: Sprint planning and later software development – GWT – Google Web Toolkit is a develop- and unit tests. At the same time a specification ment toolkit for building and optimizing for functional tests must be prepared. If pair pro- complex browser-based applications (https: gramming was not carried out, than a code review //developers.google.com/web-toolkit/); is needed. Before the end of the Sprint, functional – Spring – a comprehensive programming and tests must be conducted. When there is time, au- configuration platform for modern Java-based tomated functional tests are prepared, in other enterprise applications without unnecessary case, automation is postponed to the next Sprint. ties to specific deployment environments Test automation is not a part of the ‘definition (http://www.springsource.org/); of done.’ It might seem strange as automation – Hibernate – It is a framework which allows is very important in agile processes, but unfortu- to provide the service of the data storage in nately automation is very time-consuming and a database, regardless of the database system for technical reasons sometimes must be done af- (http://www.hibernate.org/); ter implementation (this remark does not regard – Oracle DB – Relational database manage- unit tests). In consequence, it was not possible ment system which is commonly used in cor- to always do implementation and automation in porations because of data maintaining capa- the same Sprint.