e-Informatica Software Engineering Journal, Volume 10, Issue 1, 2016, pages: 69–87, DOI 10.5277/e-Inf160104
Highly Automated Agile Testing Process: An Industrial Case Study
Jarosław Berłowskia, Patryk Chruściela, Marcin Kasprzyka, Iwona Konanieca, Marian Jureczkob aNetworkedAssets Sp. z o. o. bFaculty of Computer Science and Management, Wrocław University of Science and Technology [email protected]
Abstract This paper presents a description of an agile testing process in a medium size software project that is developed using Scrum. The research methods used is the case study were as follows: surveys, quantifiable project data sources and qualitative project members opinions were used for data collection. Challenges related to the testing process regarding a complex project environment and unscheduled releases were identified. Based on the obtained results, we concluded that the described approach addresses well the aforementioned issues. Therefore, recommendations were made with regard to the employed principles of agility, specifically: continuous integration, responding to change, test automation and test driven development. Furthermore, an efficient testing environment that combines a number of test frameworks (e.g. JUnit, Selenium, Jersey Test) with custom-developed simulators is presented. Keywords: software engineering, testing process, agile software development, case study
1. Introduction bility [5], the role of testers [6] or test automa- tion [7]. Software testing is a very costly part of the soft- The goal of this research is to extend the body ware development process, it is sometimes esti- of knowledge concerning agile testing by docu- mated to make 50% of the whole development menting a real life software testing process. This cost [1], [2]. It is one of the main activities (at paper presents a case study of a medium-size soft- least should be) in agile software development ware project with special factors that affects the methods. Beck and Andres [3] claimed it to be aforementioned process, i.e. requests for unsched- a measure of project progress and the main mean uled releases and high complexity of project envi- of assuring software quality. On the other hand, ronment. The project is a Java Enterprise system applying agile software development methods sig- in the telecommunication domain and its main nificantly affects the testing process. The agile purpose is to ease network devices controlling and methods usually require to test early and to have management. Thus, there is a number of features the tests automated. In consequence, testing is that concern integration and communication with not left to the final phase, but requires sustain- other systems and devices. Functional tests of able investments during the whole process (in- such features involve items that are outside of the cluding after-release maintenance of automated tested system, but are necessary for a successful test cases). Some of the authors even recommend test execution. Therefore, the test environment to start with testing [4]. The agile testing has is complex and its management can consume been studied for several years, however, there a considerable amount of resources, specifically are still unanswered questions regarding scala- in the case of automated tests, where all changes 70 Jarosław Berłowski et al. that come from test execution should be verified size, namely the number of Lines of Code (LOC). and reverted in order to ensure tests repeatability. The project investigated here is described as On the other hand there are many, unscheduled a medium-size software project. The small, release requests that concern the newest version medium and large scale is not a precise term, it of the developed system. Those releases regard is very intuitive (e.g. more intuitive than LOC). presentations for end users or deployments into The investigated project has been classified as an environment for acceptance tests. Nonetheless, a medium-size one, since it seems to be a bit the released version must satisfy company quality smaller than projects described in [5] and [6] standards and must be ‘potentially shippable’ [8]. which are reported as large ones. The authors The unscheduled releases could be explained as refer to costs of development and test. This releases in the middle of a Sprint (the project should be considered as the amount of time com- is developed using Scrum), that are announced mitted by the development team during project one or two days in advance and are requested related activities. One of the investigated aspects to contain some features from the current Sprint of the testing process is availableness for un- and, of course, all the features developed in pre- scheduled release. A possibility of making an vious Sprints. The unscheduled releases create application release is evaluated using results of challenges. It is critical to ensure that the ‘old’ the functional tests (Selenium tests and REST functionality still works correctly and there are tests) executed in a continuous integration system very limited resources for regression tests, since (100% success of unit tests is a prerequisite of the there are the ‘in-progress’ features that must be functional ones). The application is truly ready straightened up before the release. The solution for release if 100% of the functional tests are is somewhat obvious – test automation. In order passed. The application fits for a presentation in to support the unscheduled releases a complex the case when at least 90% of the tests are passed. set of automated regression tests must be present If more than 10% of the tests failed, no form and frequently executed. In consequence, it is pos- of release shall be considered. The investigated sible to evaluate the current version of software testing process was assessed using the concept of instantly and decide if it is ‘potentially shippable’. the level of agility that evaluates the adoption The paper describes how the challenges are sat- of different agile principles using the concept of isfied, in which ways they affected the testing the level of adoption. Both aforementioned process and what results were obtained. concepts are borrowed from [5], further details The case study presents an insider perspec- about them can also be found in section 2.3. tive, i.e. it is conducted by members of the team The rest of this paper is organised as follows. that develops the project. Therefore, it is possible The next section presents the goal and study to get a really deep insight, but unfortunately design as well as a detailed description of the the risk of biased observation or conclusion grows study context. The obtained results are docu- simultaneously. In consequence, the case study mented in section 3. In section 4, the threats to is focused on a quantitative approach, which is validity are discussed and in section 5, related associated with lower risk of subjectivity instead work is summarized. Finally, the discussion and of the qualitative one, which is usually employed conclusions are given in section 6. in similar case studies. The study is designed according to guidelines given by Runeson and Höst [9]. 2. Goal and Study Design A number of concepts is used in this paper. Let us present explanations of them in order to 2.1. Context clarify possible ambiguities. The authors refer to project size. The concept is in fact the metric The context is described according to the guide- which Mall [10] identified as one of the two most lines given by Petersen and Wohlin [11]. It corre- popular and the simplest measures of project sponds with the suggested description structure Highly Automated Agile Testing Process: An Industrial Case Study 71 as well as the recommended elements of the con- Customisation. It is custom software develop- text. ment. The product is tailored to the requirements of a customer (a telecommunication vendor). 2.1.1. Product Programming language. The system is devel- oped mostly in Java. The investigated project regards the development of a system which is used in the management of 2.1.2. Processes telecommunication services delivery. The main functionalities of the product are: Figure 1 presents the testing process. In fact, no – managing customer access devices, defined process is used, and what the diagram – managing and configuring network delivery shows is the result of the ‘definition of done’ devices, that is being employed. New user stories are – managing resources of IP addresses from the first identified, then acceptance criteria for them public and private pools. are defined and stories are estimated. When the The following technologies are used in the soft- Sprint begins, the development team makes the ware development: Sprint planning and later software development – GWT – Google Web Toolkit is a develop- and unit tests. At the same time a specification ment toolkit for building and optimizing for functional tests must be prepared. If pair pro- complex browser-based applications (https: gramming was not carried out, than a code review //developers.google.com/web-toolkit/); is needed. Before the end of the Sprint, functional – Spring – a comprehensive programming and tests must be conducted. When there is time, au- configuration platform for modern Java-based tomated functional tests are prepared, in other enterprise applications without unnecessary case, automation is postponed to the next Sprint. ties to specific deployment environments Test automation is not a part of the ‘definition (http://www.springsource.org/); of done.’ It might seem strange as automation – Hibernate – It is a framework which allows is very important in agile processes, but unfortu- to provide the service of the data storage in nately automation is very time-consuming and a database, regardless of the database system for technical reasons sometimes must be done af- (http://www.hibernate.org/); ter implementation (this remark does not regard – Oracle DB – Relational database manage- unit tests). In consequence, it was not possible ment system which is commonly used in cor- to always do implementation and automation in porations because of data maintaining capa- the same Sprint. Having the choice to either do bilities, security and reliability (http://www. longer Sprints or postpone automation it was oracle.com); preferable to postpone the automation as there is Maturity. The project was conducted from May the requirement for unscheduled releases, which 2011, the first release was carried out in Autumn does not correspond well with long Sprints. Post- 2011. The study was conducted in January 2013. poning test automation is not a good practice, but Quality. The main means of ensuring the prod- conducting functional tests manually [12,13] or uct quality are tests. Further details regarding automating after the sprint [5,14] is not unique in the testing process are presented in the next software development. Puleio [14] even invented sections. a name for not doing the test automation on Size. The total number of Lines of Code is time, i.e. testing debt. At the end of the Sprint 327387 (calculated by StatSVN – all lines are the software is potentially shippable as the new counted). features which are not covered by automated System type. Enterprise system with a web tests (if there are such) are tested manually. As UI. The development team makes use of func- presumably it already emerged from the above tionalities accessible as REST services and also description, the employed development process integrates a them with existing systems. is Scrum. 72 Jarosław Berłowski et al.
Figure 1. Testing process
2.1.3. Practices, Tools, Techniques opers, testers (one tester performs also the role of a business analyst), the Scrum Master CASE tools (who takes part in software development) and Eclipse STS (http://www.springsource.org/sts) the Product Owner (who does not take part is used as the primary development environment. in the software development): Furthermore, the development team uses a set – Software developers (5), of frameworks for test automation which are em- – Scrum Master (1), ployed in a continuous integration system (details – Testers (2). in the next section). Practices and techniques 2.2. Experience – Time–boxing. – Frequent (in fact continuous) integration. Some software developers have longiterm experi- – Frequent, small releases. ence in the IT field, i.e. two people 5+ years – Early feedback. of experience (one of them is a Scrum Mas- – Test automation. ter), the Product Owner 15+ years of experi- ence and one of the testers who is also a busi- 2.1.4. People ness analyst 10+ years of experience. For the rest of the team, this is the first job. They Roles take part in the project, learn technical skills • Product Owner – comes up with new ideas, and expand knowledge using different tools. The updates priorities and chooses the most impor- entire team has completed bachelor’s or mas- tant issues, he is also responsible for contact ter’s degree studies in computer science. In or- with customers. der to improve the qualifications, the members • Scrum Master – takes control of the Scrum of the development team have participated in process and enables team members to reach some trainings and certification programs, for the goal of the Sprint by helping them with example ISTQB (International Software Testing all impediments. For a short period of time Qualifications Board) – both testers, Professional there were two Scrum Masters, as the project Scrum Master I – the Scrum Master, CISCO was developed by two different teams. certificates (CCNA/CCNP, CCAI) – one of the • Team – responsible for reaching the goal of testers. the Sprint. The development team currently The development process was configured by consists of 8 people including: software devel- the experienced team members who also take care Highly Automated Agile Testing Process: An Industrial Case Study 73 of the introduction of their younger colleagues. also with the customer, however, with respect Thus, there are no reasons to believe that there to testing process the most relevant commu- are issues in the process resulting a lack of experi- nication is between developers and testers ence. Additionally, the trainings and certification (weight = 2). program were used to ensure high development – Pair programming – pair programming does standards and to avoid shortcomings. not relate to testing directly. Nevertheless, it affects the quality of a source code – could be 2.3. The Level of Agility of the considered as on-the-fly code review (weight Investigated Testing Process = 1). – Root-cause analysis – it is one of the quality The paper is to be focused on agile testing process. related eXtreme Programming corollary prac- Hence, it is important to take carefully analyze tices. It forces complex analysis for each of the the principles of agility and assess the process. identified defects. Not only the defect should Otherwise, there would be a considerable risk of be resolved, but also the software development investigating other phenomena, not the ones that process should be improved to avoid similar should be investigated. defects in future. The root-cause analysis is not recommended when some of the essen- 2.3.1. Survey Design tial eXtreme Programming practices are not adopted. Therefore, there are agile testing The issue is addressed with a survey conducted processes that do not employ it (weight = 1). among all the staff that is or was involved in – Working software (over comprehensive docu- project development. The survey is designed ac- mentation) – one of the Manifesto for Agile cording to a set of weighted criteria suggested Software Development rules that may have by Jureczko1 [5], i.e. the following criteria are strong influence on tests (weight = 1). used (each of them can be met fully, partly or – Responding to change (over following the not at all): plan) – another rule from the aforementioned – Test driven development – TDD is one of manifesto. In the context of a testing process, the most remarkable improvements, which this rule is mainly considered with respect to has been brought to testing with the agile the flexibility of test plans (weight = 1). methods [15], [3]. Presumably not every agile The above listed criteria are extracted from prin- testing process uses TDD, on the other hand ciples suggested in eXtreme Programming [3] TDD affects the process and system design and Agile Manifesto. Each of them is somehow in such a significant way, that it cannot be connected with testing and as the Authors be- ignored when evaluating the level of agility lieve they make a rich enough subset of all agile (weight = 3). principles to offer a good understanding of the – Test automation – most of the agile methods project reality with respect to the testing process. move the focus from manual to automated The questionnaire was generated in a paper form, tests, e.g. [3]. Furthermore, the automation nonetheless, it is available on-line: http://purl. is often considered an essential practice [16], org/MarianJureczko/TestingProcess/Survey. [14] (weight = 3). – Continuous integration – what is the use of 2.3.2. Survey Results automated tests when they are not executed frequently? (weight = 2). The questionnaire was completed by all present – Communication – it is considered at two lev- team members and those past members that are els, i.e. not only between team members but still working for the company. The results are dis-
1 The author argued the usage of weights by stating that the principles of agility have different relevancy from the testing perspective. It is also noteworthy that some of the agile software development methods, e.g. XP [3], identify categories of principles that are based on the relevancy of their adoption. 74 Jarosław Berłowski et al. cussed in detail in the forthcoming subsections son (http://hudson-ci.org/) system to support and presented in Figure 2. Each of the sub figures automatic compilation, testing and deployment shows which respondents recognised given levels of the developed system. of adoption. It is worth mentioning that there is The unit tests and some of the functional no single ‘I do not know’ response. Presumably tests are executed after each commit. The func- it is a consequence of the fact that each of the tional tests are split into two groups. The first of respondents actively participates or participated them contains test cases that do not need a great in the investigated project. amount of time for execution, which in fact means Test driven development More than half of no interaction with graphical interface – these the respondents recognised test driven develop- tests are executed after each commit. The second ment as partly implemented and the rest of them group of tests contains GUI related tests and for as fully implemented (Fig. 2a). TDD has been performance reasons it is executed nightly. used in the project for a long time, however, The Hudson continuous integration system not all features are developed in this way. It is supports also releases. After a Sprint Review always the developer’s responsibility to choose Meeting, i.e. when the sprint ends, a Hudson job the most efficient way of the development and is executed and performs the following actions: there are no repercussions for not doing TDD. – The version number of the developed system Regardless of the selected development method, is updated. the unit tests are obligatory and hence TDD – A SubVersion ‘TAG’ is created for the new is often a good choice. Nonetheless, there is version of the developed system. also a number of features that are closely cou- – Release notes regarding newly implemented pled to moderately documented, external ser- features are generated. vices which do not make a perfect environment – A new version of the developed system is for TDD. deployed to Apache Maven repository and Test automation Almost all respondents recog- saved on a FTP server. nised test automation as fully adopted (Fig. 2b). – The new version is installed automatically in test automation is very important in the investi- the customer acceptance tests environment. gated testing process. The automated unit tests Communication All respondents recognised are explicitly written in the project’s ‘Definition the Communication principle as partly adopted of Done’ [8]. The automation of functional tests (Fig. 2d). The communication was considered at is also obligatory, however, sometimes it is post- two levels, namely among team members, specifi- poned and it is not conducted in the very same cally between testers and developers and between sprint as the functionality which has to be tested. team members and the customer. The communi- Typically, there is one automated test case per cation between team members is acknowledged a user story, but there are also some very com- to be effective. Testers and developers work in plex stories that are tested by more test cases the same location. The communication is mostly and some complex test cases that cover more verbal (but all major issues are reported in an is- stories. All the automated tests cases are used as sue tracking system) and supported by the Scrum regression tests and are executed in a continuous meetings, e.g. Daily Scrum Standup Meeting. Fur- integration system which brings us to the next thermore, the roles (i.e. testers and developers) evaluation criterion. are not fixed, thus a team member has an oppor- Continuous integration Each of the respon- tunity to do developing as well as testing tasks. dents acknowledged that the continuous integra- The communication with customer was not tion principle was fully adopted (Fig. 2c). There assessed so well. The eXtreme Programming ‘on is only one development line, and as far as it is site customer’ principle [3] has not been installed. possible the development team is avoiding using Furthermore, customer representatives do not branches and promotes frequent commits. More- participate in Planning and Review meetings over, there is a number of jobs defined in the Hud- [8]. Communication channels usually go through Highly Automated Agile Testing Process: An Industrial Case Study 75
(a) Test driven development (b) Test automation (c) Continuous integration
(d) Communication (e) Pair programming (f) Root-cause analysis
(g) Working software (over com- (h) Responding to change (over following prehensive documentation) the plan)
Figure 2. The level of agility in testing process
Product Owner, which sometimes makes the com- questionnaires (Fig. 2f). Definitely the root-cause munication inefficient. analysis is not executed for all identified defects, Pair programming Most of the respondents only a fraction of them is so closely investigated. recognised pair programming as partly adopted On the other hand, there are Sprint Retrospective (Fig. 2e). The usage of pair programming in the Meetings [8] which address the most important is- project is limited – more often informal code sues and give the team an opportunity to identify reviews are used instead, usually in the form of remedies. a code walk-through. Nonetheless, presumably Working software (over comprehensive each team member experienced this practice since documentation) Most of the respondents ac- it is extensively used as a knowledge transfer tool. knowledged that the working software is valued New employees work in pairs with the more ex- over comprehensive documentation (Fig. 2g). As perienced ones in order to improve their learning a matter of fact the customer is not interested curve. in technical documentation. He is provided only Root-cause analysis Presumably there is with the user and installation guide. Therefore, a confusion over what it means to adopt this the team could decide which technical documents principle since no clear message comes from the to use and there are no reasons for preparing doc- 76 Jarosław Berłowski et al. uments that are not useful. The Agile Modeling per presents how the testing process was tuned principles [17] are followed and in consequence with respect to the aforementioned requirements, the number of created documents is limited and hence it can be considered as a descriptive or always corresponds with one of two purposes, i.e. exploratory study [9]. The Authors believe that a model to communicate or a model to under- other practitioners who operate in a similar con- stand. The Product and Sprint Backlogs are used, text will find this work helpful and use it as an but neither of these documents is delivered with example of a well working testing process. The the released product. study objective can be defined as follows: Responding to change (over following the Describe and evaluate an agile testing plan) Most of the respondents recognised that process with support for unscheduled re- responding to change is valued over following the leases in development of a software system plan (Fig. 2h). With regard to test plans the prin- that depends on a number of external ser- ciple is fully adopted since test plans are never vices and devices. fixed and there is always room for an update. The investigated testing process deals with It looks slightly different in the case of the set two challenges. Rge first of them comes from of features (user stories) that are chosen for im- business, i.e. there are often unexpected opportu- plementation during a Sprint Planning Meeting. nities which must be addressed immediately, i.e. The plan created during the aforementioned meet- within one or two days depending on a release, ing shall not be changed according to Schwaber otherwise they are missed. In consequence, some- [8]. However, there is a possibility to terminate times it is not possible to wait with the release a Sprint and plan a new one (which has been till the end of a Sprint. The second challenge done several times during the project). Further- comes from the project domain. The developed more, sprints are not long (for several months one system operates in a complex environment that week sprints were used, currently they have been consists of a number of different network devices extended to two weeks), thus waiting for a new and services. Thus, it is a typical case when the sprint with an unplanned change is not painful. configuration of the surroundings requires more It should also also stressed that there is no fixed time that the test execution itself. This chal- long term plan. Each new sprint is planned from lenge significantly affected the testing process, scratch and hence unexpected changes can be and therefore the Authors believe that it is a very easily handled. important part of the project’s big picture and Conclusion The results were evaluated in the must not be ignored in the case study. way suggested in [5], i.e. value 1 was used for full adoption of an agile principle and 0.5 for 2.5. Research Questions and Data partial adoption, then weighted average was cal- Analysis Methods culated (the weights are given in teh previous subsection) and the value of 76.6% was obtained. RQ1: To what extent is the requirement The value is higher than the one calculated for for unscheduled releases satisfied? The un- the project investigated in [5], which could be scheduled releases are one of the main drivers of interpreted as a slightly better adoption of agile the definition of testing process. Thus, it is critical testing principles in the described in this study to study this aspect. It will be evaluated using process. a quantitative approach. The continuous integra- tion server will be employed as a data source and 2.4. Objective the results of the builds that execute functional tests (builds with unit tests are a prerequisite) The study is focused on a testing process in will be used as a measure of the possibility of a medium-size software project with challeng- a release. In order to quantify the data we as- ing requirements for unscheduled releases and sumed the a release is possible when all tests are complex infrastructure to deal with. The pa- passed. Additionally, we assumed the possibility Highly Automated Agile Testing Process: An Industrial Case Study 77 of a release with acceptable risk of failure when fore, it is possible to address the third research not more than 10% of tests failed. Such a risk can question by mining the collected data and ex- be accepted only when the release is conducted tracting information regarding efforts connected exclusively for presentation purposes. The 10% with creating new automated functional tests as threshold is provided to give insight into the vari- well as with maintaining the existing ones. The ability of the continuous integration outcomes. It results are presented using descriptive statistics also corresponds with the business requirements and statistical tests. as 90% of available functionality is usually enough to conduct a presentation. The Authors assumed that the release is possible when before 1 PM 3. Results the build is successfully finished according to the aforementioned criteria. The time, i.e. 1 PM, was selected as it was the latest possible hour that 3.1. To what Extent is the Requirement enables installation in the customer environment for Unscheduled Releases Satisfied? or preparation of a presentation (depending on the goal of the unscheduled release) during the Research regarding availableness for an unsched- same working day. uled release was conducted over a period of two RQ2: How is the test automation per- months. The results are shown in the Figure 3. formed? Test automation is the main mean to For 47.4% of the time, the application was ready address unscheduled releases as it is the only for the release. A period of release with accept- way to quickly assess the current software qual- able risk – 10,5%. The application was not ready ity and decide if it is acceptable for a release. for release for 42.1% of the time. The obtained Furthermore, automation has a crucial role in results can be claimed as satisfactory with respect the testing process and requires significant ef- to availableness for an unscheduled release. forts. Specifically, in a project that operates in Kaner et al. [18] considered test coverage in a complex environment that creates non-standard the context of requirements-based testing, which requirements for test configuration and in conse- is very similar to the REST and Selenium func- quence out-of-the-box solutions are not sufficient. tional tests, as they are intended for proving that Hence, it is vital to describe in detail how the the program satisfies certain requirements. There test automation is done. This research question is at least one automated test per requirement, will be addressed using the qualitative approach. which the investigated project is expressed us- All the employed testing frameworks, tools and ing user stories. Therefore, there is 100% test home-made solutions will be described with re- coverage at the requirement level, i.e. each of spect to their role in the project. the requirements is tested. However, it is an RQ3: How much effort does the test au- overoptimistic interpretation as not all corner tomation require (with respect to different cases in some of the user stories are automated types of activities and tools)? Automated and thus there are places where errors cannot tests play a crucial role in deciding about an be detected using the REST or Selenium func- unscheduled release and since two different test tional tests. The tests execution results may also frameworks were used it could be very interesting misleadingly show errors in the case of database to evaluate the process from a business perspec- malfunction or overload of a machine on which tive as well. Hence, the study presents data re- the test environment is located. Adding further garding costs of test automation that allow to jus- tests would decrease the probability of releasing tify whether it is worth making the investments a low quality version of the system by limiting and which framework should be be chosen to have the number of not covered corner cases but it support for unscheduled releases. The company would also increase the probability of blocking tracks data about committed efforts using the a high quality release which can happen as a re- issue tracking system (Atlassian JIRA). There- sult of the aforementioned continuous integration 78 Jarosław Berłowski et al.
Figure 3. Results obtained from the continuous integration system system malfunctions. Additional tests would also couplings to external systems or dependencies in increase the costs of tests maintenance. unit tests.
3.2. How is Test Automation Performed? 3.2.2. Automated Selenium Tests
The results of the execution of automated tests Automated regression tests with Selenium Web- are used as the primary citerion in making deci- Driver are used to test whether high level func- sions regarding unscheduled releases. In order to tionalities and a graphical user interface work ensure that tests results correspond with sys- and react properly to input. These tests are com- tem quality, the team uses a wide variety of bined into a suite and executed nightly within tools and frameworks. This includes also sev- the continuous integration system (a new build eral self-developed solutions which contribute to of the system under test is deployed at the end of conducting test automation in complex environ- every day on a dedicated server and its database ments. is set to a predefined state). These tests are not executed after each commit, as it is in the case of 3.2.1. JUnit Test Framework JUnit and REST tests, since, due to their com- plexity, the execution takes more than one hour. JUnit is a test framework for the Java language. It is more than the recommended 10 minutes [3] On its own it provides the basic functionality and thus would have decreased the comfort of for writing tests, which can be further enhanced developers’ work. by other extensions. JUnit tests also serve as Tests using Selenium WebDriver are func- an entry point to nearly every type of test in tional, they simulate a user action on the interface the project development (including the aforemen- and check whether the output matches expecta- tioned self-developed solutions). It is a first class tions. Typically a test covers exactly one user citizen with regard to the way the unit testing and story, however, there are exceptions that span test driven development are conducted. The unit multiple test cases and user stories. Additionally, tests are a part of the build, and thus the system for tests where a connection to an external device cannot be built (and in consequence released) is required (and it is not feasible to keep a real when some of the tests do not pass. There is also device connected to a test machine at all times), a positive side effect, i.e. developers are forced to the team developed simulators which can be set instantly fix issues detected by unit tests. up to respond like this particular device would EasyMock class extensions. EasyMock pro- (see 3.2.5). The development team strives to have vides Mock Objects for interfaces and objects at least one automatic functional test for every through class extension which is used to replace user story. Highly Automated Agile Testing Process: An Industrial Case Study 79
3.2.3. Jersey Test Framework response should be given to request, is configured in an XML settings file. Tests prepared using Jersey Test Framework are The SSH Simulator can be used in two ways. referred to as the REST tests for short. The It is possible to launch it as an SSH server or as framework enables functional tests for REST ser- a temporary service for a single JUnit test. The vices. It launches the service on an embedded SSH Simulator tool is configured using XML files. container, sends HTTP requests and captures The XML configuration file contains the expected the responses. requests and responses that will be generated by The REST tests are used to test function- the simulator: ality of the system (each user story is covered SSH Simulator configuration file by at least one Selenium or REST test). The
Executing a simulated WebService for test pur- public Result op(@WebParam(name = "NAME"...) poses String name...) throws OperationFault_Exception { public class WebServiceExecutor { try { private static Endpoint endpoint; assertEquals("PCSCF", name); private static ExecutorService executor; } catch (Throwable t) { ... log.error(t.getMessage(), t); public WebServiceExecutor(NaWebService ws) { reportError(t); if (endpoint != null && } endpoint.isPublished()) { return = new GenericResponseType.Result(); endpoint.stop(); } } } endpoint = Endpoint.create(ws); } The presented example shows how to test the /∗∗ Starts the web service ∗/ value of a WebService parameter. WebService public void publish() { simulators are usually employed in the REST if (executor == null) { tests and are useful in testing WebService clients. executor = Executors.newSingleThreadScheduledExecutor(); 3.3. How Much Effort Does the Test } Automation Require (with Respect endpoint.setExecutor(executor); to Different Types of Activities and endpoint.publish(WS_URL); Tools)? } /∗∗ Closes the web service and verifies To answer this question, it is necessary to refer execution results ∗/ to historical data about the efforts made by team public void shutdown() throws Throwable { members to create and maintain existing auto- endpoint.stop(); mated tests. For this purpose, data from the issue assertTrue(webService.isOk()); tracking system used by the company (Atlassian } JIRA) were collected. } Then the data about all automated tests have been divided into two categories according to The web service has its own thread, but the the used tool: Selenium tests and REST tests. ‘shutdown’ method is called from the JUnit con- This distinction has been made due to the fact text. Therefore, it is possible to assess what calls that these tools tests different layers of the soft- the received WebService. It is done by using the ware. The REST tests are focused on the REST ‘isOk’ method which should be implemented by services, while the Selenium test examines the each of the simulated WebServices: correctness of the operations of GUI which in Example of a WebService implementation turn may use the aforementioned services. The @WebService(name = "WS", difference between these two types of tests is serviceName = "WSService") also noticeable in in their maintenance because public class MyWs extends NaWebService of time needed to activate them. The Selenium ... tests are started periodically at a specified time @WebMethod(operationName = "OP", by the continuous integration system (or manu- action = "urn#OP") ally by a developer in their environment), which @WebResult(name = "Result", prolongs the response time to errors in tests. targetNamespace = "http://...") The REST tests are executed in a continuous @RequestWrapper(...) integration system after each commit and IN @ResponseWrapper(...) A local environment when developers build the Highly Automated Agile Testing Process: An Industrial Case Study 81 application, so tests errors are usually spotted tests a two-sample t-test was used to verify the immediately. following null hypothesis: Team effort has been measured based on the The average effort committed to cre- time which has been logged on specific tasks ation of a Selenium test is equal to the which regard the creation of automated tests and average effort of a REST test creation. their subsequent maintenance. Figure 4 presents versus an alternative hypothesis: the measured percentage of time spent on cre- The average effort committed to cre- ation and maintenance related to the total time ation of a Selenium test is larger than in of the REST tests. A relatively low percentage the case of a REST test. of the REST tests maintenance (18%) is due to Table 1. Efforts committed to the creation of the fact that fixing is usually easy to perform. the Selenium and REST tests statistics So the overwhelming amount of time is spent on the implementation of new test cases. Figure Selenium REST 5 presents the percentage of time spent on the tests Tests creation and maintenance of the Selenium tests. Figure 6 shows the average time spent on the Mean 13.46 5.83 Variance 146.89 5.67 implementation and maintenance task of the Se- F 24 lenium and REST test. The average time spent P (F ≤ f) one-tail 0.0004 by a developer on Selenium test creation (13.46h) t-Stat 3.2 was more than two times longer than the cre- P (T ≤ t) one-tail 0.0014 ation of a REST test (5.53h). An even greater difference between average times was observed Since the alternative hypothesis is asymmet- between the maintenance of the Selenium and ric, the one-tail version of the t-test was used. a REST tests. The average time of Selenium test The hypotheses are tested at the significance level maintenance tasks (6.75h) was more than three α = 0.05. The F -Test was used to test whether times longer than the average time logged on the variances of two populations are equal and REST tests maintenance. It results from the fact according to the obtained value of P (F ≤ f), that repairing Selenium tests is usually difficult i.e. smaller that 0.05, the two-sample assuming to perform (in most cases there is a need to fix unequal variances version of t-test 2 was used. the code on both, the client and the server sides). P (T ≤ t) value equal to 0.0014 was obtained, Figure 6 shows the differences between the effort thus the null hypothesis can be rejected and the committed for the REST and Selenium tests. This alternative one accepted. In other words, the ef- is largely due to the difference in the complexity fort needed to create a REST test is significantly of these tests. On the other hand, the Selenium smaller than it is in the case of the Selenium tests (which generally require more effort) de- tests. tect errors in both, the server and the client It is not possible to present analogous statis- side code. tics for the efforts related to the maintenance of In the case of the creation of both the Sele- the tests. The maintenance task almost always nium and the REST tests, it is possible to present spans across multiple test cases, hence there are further statistics, see Table 1. The average num- no data regarding individual tests and the average bers of hours committed to automation of a test values have already been reported. case are reported once more but also the vari- The execution of the automated test cases is abilities and results of analysing the differences done in the continuous integration system, hence between the Selenium and the REST tests are it does not involve additional efforts. The project given. In the case of the REST tests not only the compilation and build process, which is frequently mean effort but also the variance is noticeably executed by developers as a part of their work, smaller. In order to compare the two types of includes only unit tests and REST tests and in
2 Satterthwaite’s approximate t-test, a method in the Behrens–Welch family. 82 Jarosław Berłowski et al.
Figure 4. Time spent on the REST tests Figure 5. Time spent on the Selenium tests
Figure 6. Average time spent on implementation and maintenance of a single Selenium and REST test consequence it is below the 10 minutes suggested There is one more type of tests in the investi- by Beck [3]. Therefore, the Authors do not con- gated project, i.e. the manual tests. These tests sider test execution as a source of additional effort are outside of the scope of RQ3 as they are not for a development team. appropriate for supporting unscheduled releases, The next logical step in answering this re- but still could be interesting. Unfortunately, it search question leads to the unit tests. Unfortu- was not possible to correlate the manual tests ef- nately, there are no empirical data in this area. forts per particular requirement and according to As in the case of test driven development, the the development team there is a significant frac- unit tests are created simultaneously with the tion of untracked efforts. Therefore, the Authors code. There were no dedicated tasks regarding decided to report the subjective interpretations of unit tests that carry effort related information. team members involved in manual testing. The Furthermore, it is not possible to decide which interviewed team members estimated the cost part of the tracked effort was committed to the of implementing and maintaining an automated production and which to the test code. However, test scenario in the investigated project to be test driven development considers the prepara- up to 20 times higher than the cost of manual tion of a unit test as an integral part of the execution. Furthermore, the overall testing effort development process, hence decisions regarding (it covers both manual and automated tests) was unit tests should not be driven by cost and effort estimated by the members of the development related criteria in an agile testing process. Highly Automated Agile Testing Process: An Industrial Case Study 83 team to be close to 25% of the overall develop- Test automation efforts were evaluated using ment effort. data stored in the issues tracking system. There are no serious doubts about the quality of the data, but when it comes to completeness the situ- 4. Threats to Validity ation changes. There is a considerable probability that a fraction of efforts was not tracked. There In this section the most trustworthy results are could be small issues that were not reported at all evaluated: to what extent they are true and not or issues that were missed by the developers, e.g. biased toward the Authors’ subjective point of if they forgot about tracking their efforts. We did view. not find a way to evaluate which fraction of the collected data suffers from such problems. How- 4.1. Construct Validity ever, the data filtering was done manually. The developers, who were assigned to the test automa- Most of the employed measures are indirect. The tion tasks were approached and asked how the level of agility was assessed using questionnaires, collected data correspond with their real efforts. hence there is a risk of misunderstanding the questions or giving biased answers due to unin- 4.2. Internal Validity tentional company influence or context. To ad- dress the risk a meeting was organized, where all According to Runeson and Höst [9] the internal concepts from the questionnaires were explained validity is a concern when casual relations are and all questions that were answered. The pos- examined. This study does not provide such a re- sibility of unintentional influence is connected lation explicitly. Nonetheless, it may create an with the fact that all of the questioned people (as impression that following the principles that are well as the Authors) were involved in the project used in the project described here should lead to development. Therefore, they have a wide knowl- similar results, which the Authors in fact believe edge about the object of study, but simultane- is true. However, there is still a possibility that ously they cannot have an objective point of view some relevant factors are missing as they were not which is necessary to spot the influence. In conse- identified by the Authors. To mitigate the risk of quence, it must be stated that the level of agility this threat, the context was described according was assessed only from a subjective perspective. to guidelines suggested by Petersen and Wohlin Moreover, the Authors’ involvement creates an [11] and the rest of the study was designed fol- even greater threat to validity in terms of bias. lowing Runeson and Höst [9] recommendations To mitigate the issue quantitative measures were for a descriptive and to some extend exploratory preferred over qualitative ones in the study de- case study. Specifically, the Authors used the sign as the latter ones are more vulnerable to suggested research process, terminology, research influence. instruments and validity analysis. The ability of making an unscheduled release was assessed from the perspective of automated 4.3. External Validity tests and the continuous integration system. The tests results carry information about the qual- The context of this case study is described in ity of the developed system. Nevertheless, using Section 2.1 and there is no evidence that the them exclusively is a simplification of the sub- same or similar results could be achieved in a an- ject. There could be some unmeasurable factors other context. Nonetheless, the Authors believe involved (like team members intuition). The test that the environment complexity (e.g. the need results obtained from the continuous integration for simulators) increases the efforts related to system is the most convincing, quantitative mea- test automation and hence better results may sure we were able to come up with. be obtained when there are fewer couplings with external systems. 84 Jarosław Berłowski et al.
Specifically, it must be stressed out that the were investigated and as a result the Authors comparison of efforts related to different test concluded that agile practices: frameworks has very limited external validity. – tend to allow more time for testing activates, Statistical tests were employed, but the investi- while the total time for the project remains gated data were mined from only one project. the same, Therefore, it is not clear whether the results are – smooth the load of test resources, true for other projects and it cannot be empir- – require stakeholders to understand and con- ically verified which project specific factors are form to the practices in agile methods, relevant for the comparison results. A plausible – are usually supported by internal customers, explanation is presented in the ‘Discussion and – allow faster reaction time for change. conclusions’ section, however, data from addi- The findings advocate agile testing but do not tional projects are required to confirm it. correspond with the goals of the process investi- gated in this study, i.e. support for unscheduled 4.4. Reliability releases and complex infrastructure. Jureczko [5] investigated the level of agility According to Runeson and Höst [9] reliability in a testing process in a large scale financial is concerned with the extent to which the data software project. The studied project is of differ- and the analysis are dependent on specific re- ent size and comes from another domain, never- searchers. The conducted analyses are rather theless, the suggested criteria for agility evalua- straightforward and several perspectives have tion have been used to assess the testing process been presented to ensure the reliability. Nonethe- described in this study. Therefore, a compari- less, the Authors do not publish raw data, as son is possible. The process from [5] is signif- they contain business critical information, and icantly outperformed in the scope of continu- that can be an obstacle in replicating the study. ous integration, pair programming, root-cause Additionally, the Authors were involved in project analysis and working software (over comprehen- development and thus the observations as well as sive documentation), however, it is underper- conclusions may be biased – the issue is discussed formed in the field of communication. The over- in Subsection 4.1. all assessment is higher in the process studied in this work. Let us elaborate the development and testing process of this work. The project 5. Related Work was planned for 150 working years, but later the duration time was lengthened. More than The agile methods have been used for a cou- 150 high skilled developers, testers and man- ple of years. Thus, a number of case studies agers were involved. The project was divided with regard to the testing process have already into five sub-projects and the paper is focused been conducted. Nonetheless, the Authors are on only one of them. The sub-project was de- not familiar with any works that analyze the veloped by a group of about 40 people. The agile testing process with respect to unscheduled project is a custom-build solution that supports releases. On the other hand, the complexity of more than 1500 initially identified case scenarios. the developed system is always a factor taken The system is based on a well defined techni- into account, however, it but seldom becomes the cal framework, called Quasar. The development object of a study – none of the works reported team frequently delivers new releases with new in this section consider a similar approach to functionalities. There are two major releases per handling system complexity. year: in spring and autumn. They are used to Kettunen et al. [2] compared testing pro- deliver large changes and bulks of possible bugs. cesses between software organizations that ap- The two major ones are supplemented by hot- plied agile practices and employ traditional plan fix releases that target the most important bug- driven-methods. Altogether twelve organizations fixes only. The employed testing process forces Highly Automated Agile Testing Process: An Industrial Case Study 85 practices borrowed from the V-Model. Testers was limited overlap with the criteria employed work on test concepts once the specification of in our study (i.e. only Working software (over a requirement is ready. Subsequently develop- comprehensive documentation) and Responding ers write a source code and perform manual to change (over following the plan) are considered tests when testers write new automated tests. in both studies). Each of the tests is immediately added to the Talby et al. [6] described installation of ag- regression test set that is executed daily. Sub- ile software testing practices in a real large-scale system tests are performed after the integration. project in a traditional environment. The authors They are usually manual and focused on new pointed out that in the investigated case study features. the agile methods dramatically improved quality The role of test automation in a testing pro- and productivity. The testing process was ana- cess was empirically evaluated by Karhu et al. lyzed and described in four key areas: test design [7] in five software organizations among which he and activity execution, working with professional was developing a complex system with a number testers, planning, and defect management. The of interfaces to customer-specific external systems investigated project is a sub-project of a larger and hence creates challenges in test automation. one. The larger one was developed by 60 skilled The Authors identified a number of interesting developers and testers. Thus, there is a consider- consequences of automation. Quality improve- able probability that the sub-project is similar in ment through better test coverage and increase size to the project described in this study. More in the number of executed test cases were noted details about the software project investigated in among benefits, whereas costs of implementa- [6] can be found in [29]. tion, maintenance and training were pointed out A lesson learned from a transformation from as the main disadvantages. Moreover, according a plan-driven to agile testing process is docu- to Berlino [1] great emphasis is put on this is- mented in [14]. Extreme programming and Scrum sue and the methods of extending the degree of were adopted, which makes the process similar attainable automation are in the focus of test- to the one described here. Hence, the challenges ing research. Unfortunately, there is a dichotomy described by Puleio [14] may arise when trying to with regard to the industrial reality, i.e. many adopt adopt the principles advocated in this pa- companies, which claim that they have adopted per in a traditional (i.e. not agile) environment. XP, practice the automation in a limited scope [19]. Considerable research has been conducted on the test automation [16,20–23] and in general 6. Discussion and Conclusions this practice is strongly recommended. Among the aforementioned works especially [21] is note- This paper contributes to the body of knowledge worthy as it is conducted it the context of the in two ways. The Authors provide a detailed case Scrum method. Another commonly investigated study of an agile testing process, which can be agile testing practice is test driven development. used in further research and in combination with There is evidence for its usefulness in the indus- other studies to help make general conclusions. trial environment [24, 25], reports of controlled The documentation of the testing process in this experiments are also available [26, 27]. project could also be beneficial to practitioners Winter [28] investigated the challenges regard- interested in installing an agile testing process in ing testing in a complex environment, however, a similar environment. Especially, a number of the complexity came from evaluation usability ready to use testing tools are recommended (e.g. for a variety of end users. One of the areas of the simulators). interest was the agility of a testing process and A survey was conducted in order to assess the balance between the formal and informal the level of agility and the results showed a high approaches. The agility was considered in the level of adoption, i.e. 76.6%. Furthermore, each of context of the Agile Manifesto, and hence there the assessment criteria was compared against the 86 Jarosław Berłowski et al. project reality in a qualitative approach which essary listings and references to external sources gives an insight into the way the agile principles are given in Sec. 3.2.5). work. A detailed analysis indicated some areas of The overall results are satisfactory with re- possible improvement, e.g. communication with gard to the project goals. Therefore, we would like customer and pair programming. to recommend following the same rules (in simi- The project preparation for the ‘on short no- lar projects), i.e. adopt the principles of agility, tice’ release was analysed and assessed according specifically assure high quality and coverage of to test execution results in the continuous inte- automated tests and employ a continuous inte- gration system. The results gave a value of about gration system for automated builds. 47% of time in which the project was ready to be released and 10% of time it could have been released with the acceptable risk. Hence, the Au- References thors can conclude that the requirement for un- [1] A. Bertolino, “Software testing research: Achieve- scheduled releases is supported to a considerable ments, challenges, dreams,” in Future of Software extent. Engineering, FOSE ’07. IEEE, 2007, pp. 85–103. Comparative evaluation of the cost of the [2] V. Kettunen, J. Kasurinen, O. Taipale, and REST and Selenium tests was conducted. It mea- K. Smolander, “A study on agility and testing sured how much time is necessary in both test processes in software organizations,” in Proceed- frameworks for the implementation of new test ings of the 19th international symposium on Software testing and analysis. ACM, 2010, pp. and maintenance of the existing ones. A signifi- 231–240. cant disadvantage was found in the case of the [3] K. Beck and C. Andres, Extreme programming Selenium tests, which we believe is a result of explained: embrace change. Addison–Wesley Pro- using Google Web Toolkit for the graphical in- fessional, 2004. terface – this framework uses dynamic identifiers [4] L. Koskela, Test driven: practical TDD and ac- and generates complex (at least from the Sele- ceptance TDD for Java developers. Manning Pub- nium point of view) web pages. This extra cost lications Co., 2007. is counterbalanced with an ability to detect bugs [5] M. Jureczko, “The level of agility in the testing process in a large scale financial software project,” in GUI which is covered only by the Selenium in Software engineering techniques in progress, test. Nonetheless, the big difference in costs en- T. Hruška, L. Madeyski, and M. Ochodek, Eds. courages reconsideration of the testing approach. Oficyna Wydawnicza Politechniki Wrocławskiej, A brief description of all tools that were used 2008, pp. 139–152. to implement automated tests is provided. The [6] D. Talby, A. Keren, O. Hazzan, and Y. Dubinsky, main contribution in this area regards the sug- “Agile software testing in a large-scale project,” gestion for mitigating environment complexity IEEE Software, Vol. 23, No. 4, 2006, pp. 30–37. [7] K. Karhu, T. Repo, O. Taipale, and K. Smolan- with simulators. There is a detailed description der, “Empirical observations on software test- of an open-source project called SSH-simulator ing automation,” in International Conference which was co-developed by the Authors, in fact, on Software Testing Verification and Validation, it is officially presented in this paper for the first ICST ’09. IEEE, 2009, pp. 201–209. time. Furthermore, the Authors also suggested [8] K. Schwaber, Agile project management with a straightforward solution for simulating WebSer- Scrum. Microsoft Press, 2004. vices in a test environment. The paper contains [9] P. Runeson and M. Höst, “Guidelines for con- listings that show how to mock a WebService ducting and reporting case study research in software engineering,” Empirical Software Engi- in the context of the JUnit tests. The Authors neering, Vol. 14, No. 2, 2009, pp. 131–164. believe that the detailed descriptions of those [10] R. Mall, Fundamentals of software engineering. simulators will help other practitioners who face PHI Learning Pvt. Ltd., 2009. similar challenges during test automation since [11] K. Petersen and C. Wohlin, “Context in indus- the presented solutions are ready to use (the nec- trial software engineering research,” in Proceed- ings of the 3rd International Symposium on Em- Highly Automated Agile Testing Process: An Industrial Case Study 87
pirical Software Engineering and Measurement. [22] X. Wang and P. Xu, “Build an auto testing frame- IEEE Computer Society, 2009, pp. 401–404. work based on selenium and fitnesse,” in Inter- [12] S. Harichandan, N. Panda, and A.A. Acharya, national Conference on Information Technology “Scrum testing with backlog management in agile and Computer Science, ITCS 2009, Vol. 2. IEEE, development environment,” International Jour- 2009, pp. 436–439. nal of Computer Science and Engineering, Vol. 2, [23] T. Xie, “Improving effectiveness of automated No. 3, 2014. software testing in the absence of specifica- [13] K.K. Jogu and K.N. Reddy, “Moving towards tions,” in 22nd IEEE International Conf. on agile testing strategies,” CVR Journal of Science Software Maintenance, ICSM ’06. IEEE, 2006, & Technology, Vol. 5, 2013. pp. 355–359. [14] M. Puleio, “How not to do agile testing,” in Agile [24] N. Nagappan, E.M. Maximilien, T. Bhat, and Conference. IEEE, 2006, pp. 305–314. L. Williams, “Realizing quality improvement [15] K. Beck, Test driven development: By example. through test driven development: results and Addison–Wesley Professional, 2003. experiences of four industrial teams,” Empirical [16] M. Jureczko and M. Mlynarski, “Automated ac- Software Engineering, Vol. 13, No. 3, 2008, pp. ceptance testing tools for web applications us- 289–302. ing test-driven development,” Electrical Review, [25] A.P. Ress, R. de Oliveira Moraes, and M.S. Vol. 86, No. 09, 2010, pp. 198–202. Salerno, “Test-driven development as an innova- [17] S. Ambler, Agile modeling: effective practices tion value chain,” Journal of technology manage- for extreme programming and the unified process. ment & innovation, Vol. 8, 2013, p. 10. Wiley, 2002. [26] L. Madeyski, “The impact of pair programming [18] C. Kaner, J. Bach, and B. Pettichord, Lessons and test-driven development on package depen- learned in software testing. John Wiley & Sons, dencies in object-oriented design—an experi- 2008. ment,” in Product-Focused Software Process Im- [19] D. Martin, J. Rooksby, M. Rouncefield, and provement. Springer, 2006, pp. 278–289. I. Sommerville, “‘Good’ organisational reasons [27] L. Madeyski, “The impact of test-first program- for ’bad’ software testing: An ethnographic study ming on branch coverage and mutation score of testing in a small software company,” in 29th indicator of unit tests: An experiment,” Infor- International Conference on Software Engineer- mation and Software Technology, Vol. 52, No. 2, ing, ICSE 2007. IEEE, 2007, pp. 602–611. 2010, pp. 169–184. [20] M. Catelani, L. Ciani, V.L. Scarano, and A. Ba- [28] J. Winter, K. Rönkkö, M. Ahlberg, and cioccola, “Software automated testing: A solution J. Hotchkiss, “Meeting organisational needs to maximize the test plan coverage and to in- and quality assurance through balancing agile crease software reliability and quality in use,” and formal usability testing results,” in Soft- Computer Standards & Interfaces, Vol. 33, No. 2, ware Engineering Techniques. Springer, 2011, pp. 2011, pp. 152–158. 275–289. [21] R. Löffler, B. Güldali, and S. Geisen, “Towards [29] Y. Dubinsky, D. Talby, O. Hazzan, and A. Keren, model-based acceptance testing for Scrum,” “Agile metrics at the Israeli air force,” in Agile Softwaretechnik-Trends, Vol. 30, No. 3, 2010. Conference, Proceedings. IEEE, 2005, pp. 12–19.