Test processes for a Scrum team A case study with improvement suggestions ERIK KARLSSON, FREDRIK MÅRTENSSON Master’s Thesis at the department of Computer Science, LTH Supervisor: Emelie Engström Examiner: Per Runeson Lunds University, Faculty of Engineering Abstract Many software engineering projects have adapted Agile management methods like Scrum during the last years. For software testing in Scrum environments information is limited and guidelines and practices are less well established. How do testing work in combination with Agile development, and how do the tester role fit in Agile teams? Our goal was to examine the testing process in a Scrum team and provide potential process changes to improve their testing. This included eliciting existing requirements on the testing and researching how their testing should be performed, in terms of methodology, processes and routines. Finally, we investigated how competence and responsibility should be distributed within a team that is using Agile developing methods. Background information on the existing process was collected and discussed with the team members through group interviews. This information along with relevant books or scientific papers was used to create theoretical change proposals and the team’s opinions where collected through a group interview session. The results from this interview was then used for synthesizing an implementation suggestion adapted for the conditions at the section. Traditionally a separation between the testers and developers has been highly valued. It is argued that it would be hard for the developers to conceptualize potential faults with their own code. Such a separation does not tie in well with the Agile principles of face-to-face conversation and emphasis on working software over comprehensive documentation. Studies also indicate that the benefits of having a close relationship between the two roles outweighs the downsides, as it raises the cross competence in the team. Our proposed changes to the current testing process is to introduce Acceptance tests, which we mean should be automated. These practises should be combined with well structured, session based, exploratory testing. For existing applications we propose that Unit tests and automated Acceptance tests are retrofitted when changes are made. Preface This Master Thesis was carried out at Sony Ericsson Mobile Communications in collaboration with the Department of Computer Science at the Faculty of Engineering, LTH, at Lund University. It is the final examination of our Master of Science in Engineering degree in Information- and Communications technology. It corresponds to 30 ECTS, which equals one semester of studies. The thesis work has provided us with valuable insight and knowledge which will undoubtably aid us in our future lives. We are proud of the report we have produced and hope that it can be an asset to others as well. We would like to express our deepest gratitude towards the following people: Emelie Engström for being an excellent supervisor at LTH, guiding and supporting us throughout the thesis and constantly helping us to improve the quality of our work. Anders Nyberg for giving us the opportunity to carry out this thesis, and for helping out as the supervisor at Sony Ericsson MC. Per Runeson for raising our interest in software engineering processes, and for spending valuable time being the examiner of our thesis. All the members of the team at Sony Ericsson for letting us use their valuable time, participating in interviews and providing us with all background information we needed. Henrik Ljungdahl and Daniel Appelgren for meticulously reading our report and providing valuable feedback and improvement suggestions for our opposition. The time at Sony Ericsson has been inspirational and we are grateful to everyone who helped us realize this project. Thank you! Lund, June 2009 Erik Karlsson, Fredrik Mårtensson Contents I Introduction and background 5 1 Introduction 7 1.1 Thesis background ............................ 7 1.2 Purpose and goals ............................ 8 1.3 Report composition ............................ 8 2 Theoretical background and related work 11 2.1 Agile software development ....................... 11 2.1.1 The Agile manifesto ....................... 12 2.1.2 Principles behind the Agile Manifesto . 12 2.2 Scrum ................................... 14 2.2.1 The three roles in Scrum .................... 15 2.2.2 The Scrum flow .......................... 15 2.2.3 The artifacts in Scrum ...................... 17 2.3 Extreme Programming (XP) ...................... 18 2.3.1 Kanban .............................. 21 2.4 Software testing .............................. 21 2.4.1 Defining software testing ..................... 21 2.4.2 Principles of testing ....................... 22 2.4.3 Levels of testing ......................... 24 2.5 Software testing in Agile environments . 25 2.5.1 Quality Assurance in general . 25 2.5.2 Guided exploratory testing ................... 26 2.5.3 Test automation ......................... 27 2.5.4 Professional testers in a development team . 28 II Exploratory case study 31 3 Method 33 3.1 Observations ............................... 35 3.2 Interviews ................................. 35 3.3 Likert questionnaires ........................... 36 1 CONTENTS 3.4 Validity of research ............................ 36 4 Result 39 4.1 Organization ............................... 39 4.1.1 Scrum Team Q .......................... 40 4.1.2 The customer role ........................ 41 4.2 Current process .............................. 42 4.2.1 Pre Sprint ............................. 42 4.2.2 Sprint ............................... 44 4.2.3 Post Sprint ............................ 47 4.3 Potential areas of improvements ..................... 49 4.3.1 Customer involvement in validation of deliveries . 49 4.3.2 Testing goals - external expectations, internal commitment . 52 4.3.3 Automation ............................ 53 4.3.4 Planning and structuring the testing . 55 4.3.5 The tester role .......................... 56 IIIProcess change proposals 59 5 Test process change proposals 61 5.1 Have an experienced expert tester in the team . 62 5.2 Move towards increased automation . 64 5.3 Structure the testing ........................... 66 6 Evaluation 69 6.1 Method .................................. 69 6.1.1 Group interview ......................... 69 6.1.2 Validity .............................. 70 6.2 Team opinions .............................. 71 6.2.1 Have an experienced expert tester in the team . 71 6.2.2 Move towards increased automation . 71 6.2.3 Structure the testing ....................... 72 7 Implementation suggestion 75 7.1 Acceptance tests ............................. 75 7.1.1 Created by the Team ....................... 75 7.1.2 Automated ............................ 76 7.1.3 Automated on Legacy applications . 77 7.2 Session based exploratory testing .................... 77 7.2.1 Session rules ........................... 77 2 IVDiscussion and conclusions 79 8 Discussion 81 9 Conclusions 83 Bibliography 85 V Appendices 87 A Use Case example 89 B Problem validation Likert questionaire 91 C Likert validation results 95 3 Part I Introduction and background 5 Chapter 1 Introduction 1.1 Thesis background Sony Ericsson Mobile Communications is joint venture between Sony and Ericsson, who each own 50 percent of the venture. On a global scale the main focus of the company lies within the development of mobile multimedia devices; this meaning mobile handsets and accessories. Within the company there are several supporting branches, providing technical solutions and tools for other parts of the organization. This thesis report studies one of the supporting sections positioned at Sony Ericsson’s Lund site. The section is responsible for various software tools used in verification (testing) activities at other sections of the company. The work at the section includes customization, support and administration of off-the-shelf products as well as development of add-ons and supporting standalone software. Within the section software development is done with the help of Agile methods, mainly Scrum and Test-Driven Development. While there is a vast amount on information regarding Agile software development available in the form of books and reports, the amount of data on how to test in Agile environments such as the one chosen for this thesis is much more limited. While Agile solutions like Scrum and XP provide good guidelines and practices on how to manage a software project, such guidelines and practices are hard to find in regard to testing. How do testing work in combination with Agile development, and how do the tester role fit in Agile teams? This is the main area of interest for this report. 7 CHAPTER 1. INTRODUCTION 1.2 Purpose and goals The general purpose of this thesis is to examine the testing process in a Scrum team at the section. The aim is to provide a list of potential future process changes in order to improve their way of working with testing. There also exist an expressed wish from the section manager for us to examine the role of the tester vis-à-vis the team. Should there be a dedicated tester, a dedicated test team or should the developers incorporate all testing activities along with their own? The main goal of this thesis is: • To suggest a list of changes to routines and processes, that will improve the testing activities at the section. The main questions that this report will try to find an answer to is: • What requirements