Master Thesis Software Engineering Thesis no: MSE-2020-NN 01 2021 Acceptance Testing in Agile Software Development Perspectives from Research and Practice Nayla Nasir Dept. Software Engineering Blekinge Institute of Technology SE–371 79 Karlskrona, Sweden This thesis is submitted to the Department of Software Engineering at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 20 weeks of full-time studies. Contact Information: Author(s): Nayla Nasir E-mail: [email protected] University advisor: Davide Fucci Dept. Software Engineering Dept. Software Engineering Internet : www.bth.se/dipt Blekinge Institute of Technology Phone : +46 455 38 50 00 SE–371 79 Karlskrona, Sweden Fax : +46 455 38 50 57 Abstract Context: Acceptance testing is an important activity that verifies the confor- mance of a system to its acceptance criteria. It aims to provide a detailed com- munication of domain knowledge and is used to evaluate whether the customer requirements are met. Existing literature lacks the empirical evidence for accep- tance testing. Especially in context of industry practice, it is not in the authors’ consideration, except for a few studies, where the authors have investigated the state of practice in a specific domain. Objective: This study aims to recognize the state of research and practice of acceptance testing in Agile Software Development and investigate the similari- ties and differences in both perspectives. The study contributes to identify the industry-academia gap in the context of acceptance testing. Research Method: To identify the acceptance testing practices and challenges from research, I have conducted a literature review. For the industry perspective on acceptance testing practices and challenges, I have conducted an interview- based survey of the practitioners working in Agile Software Development envi- ronment. I followed the snowball search strategy to search the primary studies, whereas to select the respondents, I used the convenience and snowball sampling method. For data analysis, I followed the approach of thematic synthesis. Results: The results of this thesis are the outcome of a literature review of 20 selected studies and an interview-based survey with 12 practitioners representing 10 companies. I identified acceptance testing practices and challenges from re- search and industry. In the research, the most recommended form of acceptance testing is acceptance test-driven development (ATDD), and the majority of the studies are referring to the use of FIT for the acceptance testing. Customer in- volvement in different phases of acceptance testing is recommended in research. From the interviews, I come across that acceptance testing is manual at large in the industry, and the most challenging aspect is the customer’s involvement. Conclusions: From the findings of this thesis, it is concluded that there is a gap between the research and industry perspective of acceptance testing prac- tices. Currently, acceptance testing in industry is mostly manual, the research is not focusing on this aspect of acceptance testing. Despite the differences, there are some commonalities as well. Specially, most challenges of acceptance testing are similar in both perspectives. Researchers have to consider the commonalities, and they have to look at how they can minimize the acceptance testing challenges from the perspective of industry. Keywords: Acceptance testing, ATDD, Agile Software Development, practition- ers’ perspective, industry-academia gap. Acknowledgments First of all, I would like to thank my supervisor, Davide Fucci, for guiding me in this study. Whenever I asked for any guidance, he ensured his availability. Secondly, I would like to thank all the experts who took active and volun- teer participation in the interviews. Without their volunteer participation, I would not be able to finish this study. Lastly, I would like to extend my heartiest thanks to my family and friends for their unconditional support during this work. I owe a lot of gratitude to my husband, Nasir Mehmood Minhas, who has always been a source of guidance and motivation for me. His confidence in me, makes me set higher goals for myself. Nayla Nasir ii List of Figures 3.1 Example color coding to extract AT practices and challenges . 12 4.1 Collaboration challenges identified by the practitioners. ...... 32 4.2 AT practices – similarities and differences in two perspectives. 35 4.3 AT practices – extent of commonality in two perspectives. .... 36 4.4 AT challenges—similarities and differences in two perspectives. 37 4.5 AT challenges – extent of commonality in two perspectives. .... 38 iii List of Tables 3.1 Snowball iterations. .......................... 11 3.2 Data extraction form. ......................... 12 3.3 Interview Questions. ......................... 14 3.4 Interview participants. ........................ 16 3.5 Participants’ Organizational Context. ................ 16 3.6 Analysis procedure adopted for step2 and step 3. ......... 17 4.1 Acceptance testing practices in the literature. ........... 21 4.2 Acceptance testing challenges in the literature. ........... 22 4.3 State of acceptance testing practice/organizational policy. 24 4.4 Acceptance test automation. ..................... 25 4.5 Practitioners’ perspective of acceptance testing. .......... 27 4.6 Participants’ opinion on ATDD. ................... 28 4.7 Practitioners’ perspective of AT practices. ............. 29 4.8 Practitioners perspective of AT Challenges. ............. 31 iv Contents Abstract i 1 Introduction 1 1.1 Background .............................. 1 1.2 What is acceptance testing? ..................... 2 1.3Whatisthegap?........................... 2 1.4Focusofthiswork........................... 3 1.4.1 Objectives . ......................... 3 1.4.2 Contributions ......................... 4 1.5 Thesis arrangement .......................... 4 2 Related Work 5 3 Method 8 3.1 Research Questions .......................... 8 3.2 Selection of research method ..................... 9 3.3 Detailed study design ......................... 10 3.3.1 Literature review ....................... 10 3.3.2 Survey ............................. 12 3.3.3 Data analysis ......................... 15 3.4 Threats to validity .......................... 18 4 Results 20 4.1 Acceptance testing in research perspective – RQ-1 ......... 20 4.1.1 Acceptance testing practices in literature .......... 20 4.1.2 Acceptance testing challenges in literature ......... 22 4.2 Acceptance testing in industry perspective– RQ-2 ......... 23 4.2.1 Organizational policy of acceptance testing ......... 23 4.2.2 Acceptance test automation ................. 25 4.2.3 AT definition ......................... 26 4.2.4 Practitioners’ Perception on ATDD ............. 27 4.2.5 Acceptance testing practices in industry .......... 29 4.2.6 Acceptance testing challenges in industry .......... 31 4.3 Similarities and differences in the two perspectives – RQ-3 .... 34 v 4.3.1 Similarities and differences in AT practices ......... 34 4.3.2 Similarities and differences in AT challenges ........ 36 5 Analysis and Discussion 40 5.1 Who is involved in acceptance testing ................ 40 5.2 When acceptance tests are written ................. 41 5.3 How acceptance tests are written .................. 42 6 Conclusions and Future Work 44 6.1 Conclusions .............................. 44 6.2FutureWork.............................. 45 vi Chapter 1 Introduction 1.1 Background Testing is an essential activity of the software development life cycle. It is a complicated and costly activity and can consume up to 50% of the total cost [1]. The goal of testing is to ensure the quality of the system under test, and it is performed at various levels, including unit, integration, and system level. It verifies the correctness of the system’s functional and non-functional aspects, and this fact classifies the testing into two categories: functional testing and non- functional testing. Some of the common testing types are unit testing, integration testing, system testing, and acceptance testing [2]. The success of any software product lies with the fact that the customers/users accept it. Acceptance testing has a central role in this regard. It advocates the involvement of the domain experts (customers) in the verification/validation process. It gives them a chance to check if everything matches their expectations and if the requirements have been communicated and implemented correctly [3]. Despite its significance, empirical evidence on acceptance testing practices in Agile is very low [4], many authors [4, 5, 6, 7] suggested more research in this area. This fact motivated the author to carry out an empirical investigation on ac- ceptance testing in Agile Software Development. Agile Software Development is a controversial topic since the outset. Its proponents and opponents are in both research and practice. Despite many criticisms, there is no doubt that Agile methods are widely used in the software industry. Customer involvement, early and frequent feedback, and quick releases are some of the benefits of Agile de- velopment methodologies. Unlike classical software development methods, Agile methods advocate for early and frequent acceptance testing [8]. The subsequent sections define acceptance testing, describe the research gap, and put a light on the focus of current research. 1 Chapter 1. Introduction 2 1.2 What is acceptance testing?