Test Processes for a Scrum Team

Test processes for a Scrum team

A case study with improvement suggestions

ERIK KARLSSON, FREDRIK MÅRTENSSON

Master’s Thesis at the department of Computer Science, LTH Supervisor: Emelie Engström Examiner: Per Runeson

Lunds University, Faculty of Engineering

Abstract

Many software engineering projects have adapted Agile management methods like Scrum during the last years. For software testing in Scrum environments information is limited and guidelines and practices are less well established. How do testing work in combination with Agile development, and how do the tester role fit in Agile teams? Our goal was to examine the testing process in a Scrum team and provide potential process changes to improve their testing. This included eliciting existing requirements on the testing and researching how their testing should be performed, in terms of methodology, processes and routines. Finally, we investigated how competence and responsibility should be distributed within a team that is using Agile developing methods. Background information on the existing process was collected and discussed with the team members through group interviews. This information along with relevant books or scientific papers was used to create theoretical change proposals and the team’s opinions where collected through a group interview session. The results from this interview was then used for synthesizing an implementation suggestion adapted for the conditions at the section. Traditionally a separation between the testers and developers has been highly valued. It is argued that it would be hard for the developers to conceptualize potential faults with their own code. Such a separation does not tie in well with the Agile principles of face-to-face conversation and emphasis on working software over comprehensive documentation. Studies also indicate that the benefits of having a close relationship between the two roles outweighs the downsides, as it raises the cross competence in the team. Our proposed changes to the current testing process is to introduce Acceptance tests, which we mean should be automated. These practises should be combined with well structured, session based, exploratory testing. For existing applications we propose that Unit tests and automated Acceptance tests are retrofitted when changes are made.

Preface

This Master Thesis was carried out at Sony Ericsson Mobile Communications in collaboration with the Department of Computer Science at the Faculty of Engineering, LTH, at Lund University. It is the ﬁnal examination of our Master of Science in Engineering degree in Information- and Communications technology. It corresponds to 30 ECTS, which equals one semester of studies. The thesis work has provided us with valuable insight and knowledge which will undoubtably aid us in our future lives. We are proud of the report we have produced and hope that it can be an asset to others as well. We would like to express our deepest gratitude towards the following people: Emelie Engström for being an excellent supervisor at LTH, guiding and supporting us throughout the thesis and constantly helping us to improve the quality of our work. Anders Nyberg for giving us the opportunity to carry out this thesis, and for helping out as the supervisor at Sony Ericsson MC. Per Runeson for raising our interest in software engineering processes, and for spending valuable time being the examiner of our thesis. All the members of the team at Sony Ericsson for letting us use their valuable time, participating in interviews and providing us with all background information we needed. Henrik Ljungdahl and Daniel Appelgren for meticulously reading our report and providing valuable feedback and improvement suggestions for our opposition. The time at Sony Ericsson has been inspirational and we are grateful to everyone who helped us realize this project. Thank you!

Lund, June 2009 Erik Karlsson, Fredrik Mårtensson

Contents

I Introduction and background 5

1 Introduction 7 1.1 Thesis background ...... 7 1.2 Purpose and goals ...... 8 1.3 Report composition ...... 8

2 Theoretical background and related work 11 2.1 Agile software development ...... 11 2.1.1 The Agile manifesto ...... 12 2.1.2 Principles behind the Agile Manifesto ...... 12 2.2 Scrum ...... 14 2.2.1 The three roles in Scrum ...... 15 2.2.2 The Scrum ﬂow ...... 15 2.2.3 The artifacts in Scrum ...... 17 2.3 Extreme Programming (XP) ...... 18 2.3.1 Kanban ...... 21 2.4 Software testing ...... 21 2.4.1 Deﬁning software testing ...... 21 2.4.2 Principles of testing ...... 22 2.4.3 Levels of testing ...... 24 2.5 Software testing in Agile environments ...... 25 2.5.1 Quality Assurance in general ...... 25 2.5.2 Guided exploratory testing ...... 26 2.5.3 Test automation ...... 27 2.5.4 Professional testers in a development team ...... 28

II Exploratory case study 31

3 Method 33 3.1 Observations ...... 35 3.2 Interviews ...... 35 3.3 Likert questionnaires ...... 36

1 CONTENTS

3.4 Validity of research ...... 36

4 Result 39 4.1 Organization ...... 39 4.1.1 Scrum Team Q ...... 40 4.1.2 The customer role ...... 41 4.2 Current process ...... 42 4.2.1 Pre Sprint ...... 42 4.2.2 Sprint ...... 44 4.2.3 Post Sprint ...... 47 4.3 Potential areas of improvements ...... 49 4.3.1 Customer involvement in validation of deliveries ...... 49 4.3.2 Testing goals - external expectations, internal commitment . 52 4.3.3 Automation ...... 53 4.3.4 Planning and structuring the testing ...... 55 4.3.5 The tester role ...... 56

IIIProcess change proposals 59

5 Test process change proposals 61 5.1 Have an experienced expert tester in the team ...... 62 5.2 Move towards increased automation ...... 64 5.3 Structure the testing ...... 66

6 Evaluation 69 6.1 Method ...... 69 6.1.1 Group interview ...... 69 6.1.2 Validity ...... 70 6.2 Team opinions ...... 71 6.2.1 Have an experienced expert tester in the team ...... 71 6.2.2 Move towards increased automation ...... 71 6.2.3 Structure the testing ...... 72

7 Implementation suggestion 75 7.1 Acceptance tests ...... 75 7.1.1 Created by the Team ...... 75 7.1.2 Automated ...... 76 7.1.3 Automated on Legacy applications ...... 77 7.2 Session based exploratory testing ...... 77 7.2.1 Session rules ...... 77

2 IVDiscussion and conclusions 79

8 Discussion 81

9 Conclusions 83

Bibliography 85

V Appendices 87

A Use Case example 89

B Problem validation Likert questionaire 91

C Likert validation results 95

Part I

Introduction and background

Chapter 1

Introduction

1.1 Thesis background

Sony Ericsson Mobile Communications is joint venture between Sony and Ericsson, who each own 50 percent of the venture. On a global scale the main focus of the company lies within the development of mobile multimedia devices; this meaning mobile handsets and accessories. Within the company there are several supporting branches, providing technical solutions and tools for other parts of the organization.

This thesis report studies one of the supporting sections positioned at Sony Ericsson’s Lund site. The section is responsible for various software tools used in veriﬁcation (testing) activities at other sections of the company. The work at the section includes customization, support and administration of oﬀ-the-shelf products as well as development of add-ons and supporting standalone software. Within the section software development is done with the help of Agile methods, mainly Scrum and Test-Driven Development.

While there is a vast amount on information regarding Agile software development available in the form of books and reports, the amount of data on how to test in Agile environments such as the one chosen for this thesis is much more limited. While Agile solutions like Scrum and XP provide good guidelines and practices on how to manage a software project, such guidelines and practices are hard to ﬁnd in regard to testing. How do testing work in combination with Agile development, and how do the tester role ﬁt in Agile teams? This is the main area of interest for this report.

7 CHAPTER 1. INTRODUCTION

1.2 Purpose and goals

The general purpose of this thesis is to examine the testing process in a Scrum team at the section. The aim is to provide a list of potential future process changes in order to improve their way of working with testing. There also exist an expressed wish from the section manager for us to examine the role of the tester vis-à-vis the team. Should there be a dedicated tester, a dedicated test team or should the developers incorporate all testing activities along with their own?

The main goal of this thesis is:

• To suggest a list of changes to routines and processes, that will improve the testing activities at the section.

The main questions that this report will try to ﬁnd an answer to is:

• What requirements are there on the test process at the section, and how should they be prioritized?

• How should the test process be constructed, in terms of methodology, processes and routines, in their Agile environment?

• How should competence and responsibility be distributed within a team that is working with Agile developing methods?

1.3 Report composition

The report is divided into the following main sections:

• Chapter 1 - Introduction: The thesis background, goals and structure description

• Chapter 2 - Theoretical background and related work: An overview of the areas relevant to our thesis, as well as relevant scientiﬁc work previously published.

• Chapter 3 - Method: The method description for the ﬁrst phase of the project

• Chapter 4 - Result: Description of the organization and current process, as well as details on areas where improvements might be possible

• Chapter 5 - Test process change proposals: Our process change proposals

• Chapter 6 - Evaluation: Contains the method description and results of the evaluation of the change proposals

8 1.3. REPORT COMPOSITION

• Chapter 7 - Implementation suggestion: An concrete adapted implementation suggestion, based on the change proposals and the evaluation thereof • Chapter 8 - Discussion • Chapter 9 - Conclusions: A short summary answering questions stated in Purpose and goals (see section 1.2)

Chapter 2

Theoretical background and related work

In this section of the report we provide a brief overview of some of the areas relevant to our thesis. This includes providing definitions to some key terms and concepts that we will be referencing throughout the report. In section 2.1 we describe the Agile software development methods Scrum and Extreme Programming, and briefly mention Kanban. Section 2.4 covers basic software testing concepts in order to establish key terms. Lastly results from related work performed by others is discussed in section 2.5. In this last section different approaches to performing testing in Agile environments is discussed, along with results available through previous work.

2.1 Agile software development

The term Agile in software development is used to refer to certain methodologies, processes and practices. In this section we will describe the meaning and provide a brief history of Agile software development. Agile as the term is used in this report, was settled upon in early 2001. This happened at a meeting between 17 software engineering experts in Snowbird, Utah. These expert were all involved in diﬀerent camps, with the commonality that they were “sympathetic to the need for an alternative to documentation driven, heavyweight software development processes” (Highsmith [1, p. 1]). Prominent members of the Scrum, Extreme Programming, Crystal Clear, Adaptive Software Development, Feature Driven Development and Dynamic Systems Development Method movements, amongst others, were participating. When referred to as a collective, these methodologies had up until this time been called light or lightweight.[1, p. 1] The participants agreed to use the term Agile in the

11 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK future. The aforementioned methodologies are all examples of Agile methodologies, of which we describe Scrum in Section 2.2 and Extreme Programming in Section 2.3. The rest of the mentioned methodologies are considered outside the scope of this report.

At the same meeting the participants made an effort to define a set of values shared by their respective methodologies. This resulted in a Manifesto for Agile Software Development, most often called the Agile Manifesto, shown in its entirety in Figure 2.1. The participants later formed the Agile Alliance, an organization to promote these values. The values reflected in the manifesto is further explained in a set of 12 principles. The Manifesto and the principles are described in the following subsections. For a discussion of testing in Agile development, see section 2.5.

2.1.1 The Agile manifesto

The Agile Manifesto, seen in Figure 2.1, comprises four values, expressed as comparisons.

The manifesto states that the signees value the concepts on the left more than the concepts on the right. It is important to notice that the intention of the manifest is not to say that the concepts on the right is of no value. The signees in fact make a point of including that “[...]there is value in the items on the right[...]”. The manifesto is a statement saying, if given the choice, we prefer the items on the left.

To further explain the implication of following these values, 12 principles has been developed. These are described in the next subsection.

2.1.2 Principles behind the Agile Manifesto

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. Agile practices should enable the team to deliver early and often. Martin [2, p.7] mean that Agile teams should strive to deliver a rudimentary system in the ﬁrst few weeks of the project. This should then be followed by a release of increasing functionality every two weeks.

Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage. Martin means that the attitude of the Agile team should be one of fearlessness towards change. He describes it thusly: “They view changes to the requirements as good things, because those changes that the team has learned more about what it will take to satisfy the market”. In practical terms this means that the Agile team should strive to build structure of the system in such a way that the impact of a change is minimal.

12 2.1. AGILE SOFTWARE DEVELOPMENT

Manifesto for Agile Software Development

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value:

Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Kent Beck James Grenning Robert C. Martin Mike Beedle Jim Highsmith Steve Mellor Arie van Bennekum Andrew Hunt Ken Schwaber Alistair Cockburn Ron Jeffries Jeff Sutherland Ward Cunningham Jon Kern Dave Thomas Martin Fowler Brian Marick

Figure 2.1. The manifesto for Agile Software Development, or Agile Manifesto

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. The focus of an Agile team is to deliver valuable software. This means software that both satisﬁes the users needs (is functional) and is working. Plans or documents should not be counted as deliveries.

Business people and developers must work together daily throughout the project. There must be frequent interaction between all stakeholders in the project. Martin says “In order for a project to be agile, there must be frequent and signiﬁcant interaction between the customers, the developers, and the stakeholders”.

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. This principle follows the value “Individuals (and interactions) over process and tools” (our parenthesis).

13 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK

People are the most important contributer to the success of the project, all other things are of secondary importance. Martin means that if something is having an adverse effect on people, be it the process, environment or management, it is subject to change. The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. In an Agile team, the primary mode of spreading and obtaining information is conversation. Written documents, such as specifications, plans or designs, should only be created if there is a immediate and significant need. Working software is the primary measure of progress. An Agile team measure there progress in terms of customer needs met. Secondary systems like documentation or infrastructure code should not be part of this measurement. Martin expresses it “They [the Agile team] are 30% done when 30% of the functionality is working”. Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. The Agile team perform their work at steady pace. They should not use up tomorrows energy to sprint ahead today. Continuous attention to technical excellence and good design enhances agility. Agile team members commit themselves to producing the highest quality they can. There should not be situations were the team produce inferior work and promise to “fix it later”. Simplicity–the art of maximizing the amount of work not done–is essential. Agile teams choses the easiest route to the goal. They should not put a lot of effort into trying to anticipate all future problems. The best architectures, requirements, and designs emerge from self-organizing teams. The Agile team is self-organizing. No one can order work from individual team members. All responsibilities from the outside must be communicated to the team as a whole. The team is then responsible for carrying out the work in the best way possible. At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. The team must adjust its processes, relationships, rules and organisation when the situation calls for it.

2.2 Scrum

The Scrum process was formalised by Ken Schwaber and Jeﬀ Sutherland and ﬁrst presented at the OOPSLA (Object Oriented Programming, Systems, Languages and Applications) conference in 1995. It provides a framework for project management. This framework, or process skeleton, is often combined with other

14 2.2. SCRUM methodologies in order to form a complete work process. In software engineering Scrum is often used in conjunction with for example Extreme Programming (XP). In such an environment Scrum is responsible for managing the project and XP practices and principles are used for the software development. The Scrum process is iterative. Every iteration starts with the team reviewing the list of what needs to be done. They then chose some items to implement during the iteration. The choice must be such that, at the end of the iteration, they have turned the items into what Schwaber calls an increment of potentially shippable product functionality [3, p 12]. The team does its best during the iteration to fulﬁll this commitment. When the iteration is completed the ”increment” is shown to the stakeholders, and the list of what needs to be done is updated. The next iteration can then begin. In the next sections we will describe the roles, ﬂow and artifacts of Scrum.

2.2.1 The three roles in Scrum

Scrum deﬁnes three roles for the project, the ScrumMaster, the Product Owner and the Team. The responsibility of the Product Owner (also known as PO) is to represent all the stakeholders in the project and the system that is being developed. The PO is responsible for making sure the customer gets the most value out of the project. He/She sets up a list of requirements, called the Product Backlog. The PO is responsible for keeping the Product Backlog prioritized in such a way that the item that oﬀers the most value to the customer is at the top. The Team is responsible for developing functionality. The Team does this in a self-organizing and self-managed fashion. This means that the Team uses the Product Owner’s prioritized Product Backlog to select a number of items. The Team commits itself to complete these items during the iteration, also known as the Sprint. The success rests in the hands of the Team, they are left alone during the Sprint to realize their commitment. The ScrumMaster’s responsibility is the Scrum process. The ScrumMaster makes sure that everyone involved in the project understands Scrum, and follows the rules and practices.

2.2.2 The Scrum ﬂow

At the heart of each iteration is the Sprint. The Sprint is a period of time, usually 14 to 30 days. The goal of the Sprint is to end up with an increment of potentially shippable product functionality. The Sprint starts with a Sprint planning meeting. The objective of the meeting is to decide what shall be worked on during the

15 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK

Sprint planning Daily scrum Sprint review Sprint retrospective

day

Sprint

Project

Figure 2.2. The Scrum process

Sprint. The Product Owner describes the highest prioritized items in the Product Backlog, and the Team decides what it thinks it can turn into functionality during the Sprint. The Team then makes a rough plan for the Sprint, breaking the requirements on the Product Backlog into tasks, and placing the tasks on the Sprint Backlog. The team then starts doing the work needed to produce the functionality. Every day the team holds short meeting called a Daily Scrum, usually 15 minutes in maximum duration. Schwaber [3, p.8] describes the objective of the Daily Scrum as having each Team member answer the following questions: What have you done on this project since the last Daily Scrum meeting? What do you plan on doing on this project between now and the next Daily Scrum meeting? What impediments stand in the way of you meeting your commitments to this Sprint and this project? No other topic can be discussed during the Daily Scrum. At the end of the Sprint a Sprint review meeting is held. At this meeting the Team presents the developed functionality to the Product Owner and any attending stakeholders. The Team should present the Product Backlog it committed to, and the Product Backlog that was completed during the Sprint. The stakeholders are free to make observations about the functionality and request changes. They can also request new functionality. These suggestions are added to the Product Backlog for prioritization. The last meeting before a new Sprint can be started, is the Sprint retrospective. This meeting is conducted by the ScrumMaster, and the objective is to make the

16 2.2. SCRUM

Team revise their process. Schwaber [3, p.138] gives two questions that each Team member should answer: What went well during the last Sprint? What could be improved in the next Sprint? Schwaber also says that any decision about process improvements that can be acted on, should be added to the Product Backlog, noting that “Retrospectives that don’t result in change are sterile and frustrating” [3, p.139].

2.2.3 The artifacts in Scrum

Scrum defines a Product backlog. This is a prioritized list, often (simplistically) described as a To do-list. The Product backlog contains the requirements for the whole project. The Product owner is responsible for rating the items in the Product backlog in terms of business value. The items should also contain a rough estimate of time (or complexity) of the implementation of them. [3, p.10] The estimation for the items in the Product backlog is done in cooperation with the members of the development team [3, p.70f]. The Product backlog is constantly updated by the Product owner, and exists for the entire duration of a Scrum managed product. [3, p.10] Similar to the Product backlog, is the Sprint Backlog. The Sprint backlog contains the items (tasks) the development team defines for turning the Product backlog into a increment of potentially shippable product functionality. Each of the items put into the Sprint backlog is time estimated; if a task is estimated at more than 14 hours, it is considered a placeholder until broken down into smaller tasks. [3, p.12] Only the team is allowed to alter the Sprint backlog, as it contains the definition of the delivery the team members collectively have commited to [3, p.48]. Increment of potentially shippable product functionality The deliverable produced in each Sprint must be a (partly) usable piece of software, and should offer an improvement from the software produced in the previous Sprint. This can be achieved by for example adding features, fixing defects or by improving performance. In Scrum, all deliverables must be considered potentially shippable. They should be thoroughly tested, well-written and well structured. All user operations of the functionality of the delivered product are documented, either through a help function or through other user documentation. [3, p.12f]

17 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK

2.3 Extreme Programming (XP)

Extreme Programming is an Agile development methodology created by Kent Beck along with Ward Cunningham and Ron Jeffries. The following section contains essential information needed in this report in regard to XP practices. Beck’s original intention for XP was that it should be used in its entirety, including all practices. As the team in focus of this report use Extreme Programming mainly as an inspirational source, only parts of the theory behind XP is covered. The parts that will be covered in the following sections are those related to specifications/requirements, testing and the Rights and Responsibilities of the different stakeholders in XP. Other parts of this particular Agile development method is considered out of scope for this report.

Rights in XP

The Manager and Customer Rights (figure 2.3) and the Programmer Rights (figure 2.4), as listed by Jeffries [4, p. 7], is the foundation on which Extreme Programming relies. They are considered important not only to the members of a XP team, but also for the successful outcome of a project using XP. We have chosen to include these in the report since they clearly elicit important parts of Agile development.

18 2.3. EXTREME PROGRAMMING (XP)

Manager and Customer Rights

• You have the right to an overall plan, to know what can be accomplished, when, and at what cost.

• You have the right to get the most value out of every programming week.

• You have the right to see progress in a running system, proven to work by passing repeatable tests that you specify.

• You have the right to change your mind, to substitute functionality, and to change priorities without paying exorbitant costs.

• You have the right to be informed of schedule changes, in time to choose how to reduce scope to restore the original date. You can cancel at any time and be left with a useful working system reﬂecting investment to date.

Figure 2.3. Manager and Customer Rights

Programmer Rights

• You have the right to know what is needed, with clear deﬁnitions of priority.

• You have the right to produce quality work at all times.

• You have the right to ask for and receive help from peers, superiors and customers.

• You have the right to make and update your own estimates.

• You have the right to accept your responsibilities instead of having them assigned to you.

Figure 2.4. Programmer Rights

19 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK

Speciﬁcations and requirements

In XP, system specifications are made up by a number of Users Stories. Each User Story contains a short description of how a certain part or functionality of the software application is expected to work. User stories are created by the customers directly, with assistance by the developers. As the customers and developers discuss the functionality of the application the customer writes down, throws away or improves User stories. Each story is kept on a story card and should reflect a few days’ work (maximum). Jeffries states that it is preferred the developers don’t write the stories. The sole intent of User stories is to reflect the wishes and expectations of the customer, and as such should belong to them. [4, p. 23ff]

Testing overview

All testing in XP should be automated. Beck states that “Any program feature without an automated test simply doesn´t exist” [5, p. 59]. Since XP development is in its essence a Test-Driven development method, automated tests are written before functionality is actually developed. These tests are then run automatically and repeatedly. Each of these tests are also isolated from the others; which has to eﬀect that one failed test does not cause a vast number of other tests to fail. [5, p. 115f]

Acceptance testing

Rights and responsibilities go hand in hand, and this is clearly deﬁned when looking at Acceptance testing in XP. One responsibility of the customer is stated directly in conjunction with the right: You have the right to see progress in a running system, proven to work by passing repeatable tests that you specify. In order to be able to provide the customers with what they have a right to get, the customers are required to provide accurate information regarding this. In other words: it is the customers’ responsibility to provide the Acceptance tests for each iteration of the development. In this responsibility lies the decision on what is to be tested, what input and output is to be tested, and what functionality that is to be tested. What the responsibility does not include is how this testing should be performed; that responsibility lies on the developers themselves. In addition, as XP is built on the foundation that all testing should be automated, these Acceptance tests should be automated as well. [4, p. 32]

20 2.4. SOFTWARE TESTING

2.3.1 Kanban

The word Kanban is Japanese, meaning card or visible record. The use of it in industrial systems stems from from multiproduct manufacturing systems, where it is used to control the available resources. Each slot on the manufacturing line has a corresponding card which goes on a scheduling board; when all cards have been used, there is no more slots on the line. Whenever a unit is finished, its card is returned to available status.[6, p. 4] In software development, or support, each Kanban card represents a task, ticket or some other suitable unit. The workflow is divided into several states, each for which there is a set limit of available slots. When all the slots are filled in that state, no new cards can be put in that state until units already in that state is cleared.

2.4 Software testing

2.4.1 Deﬁning software testing

This section is an attempt to briefly outline what is included in the term testing. Looking at testing from a general perspective, Burnstein [7] lists two definitions of Software testing: 1. Testing is generally described as a group of procedures carried out to evaluate some aspect of a piece of software. 2. Testing can be described as a process used for revealing defects in software, and for establishing that the software has attained a specified degree of quality with respect to selected attributes. [7, p. 7] The Testing Standards Working Party’s Living Glossary defines software testing as The process of exercising software to verify that it satisfies specified requirements and to detect errors. [8] All of these definitions provide a general description of software testing as a whole. Firstly it states that testing is used to evaluate qualitative attributes of the software, such as if the software is secure, reliable and correct; secondly they state that testing is used in order to find defects in the software. Testing is an essential part of the process of assuring the quality of the software. The definition of software quality specified in the IEEE Standard Glossary of Software Engineering Terminology defines that quality relates to the degree to which a system, system component, or process meets 1. specified requirements. 2. customer or user needs, or expectations. [9, p. 60]

21 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK

Both the IEEE definition of Quality and the Living Glossary definition of software testing implies the need that requirements are clear in order to verify that said requirements have been fulfilled. There is also a notable difference between these definitions and that of debugging. Debugging is defined by Burnstein and Testing Standards Working Party’s Living Glossary respectively as

• Debugging, or fault localization is the process of (1) locating the fault or defect, (2) repairing the code, and (3) retesting the code. [7, p. 8]

• Debugging: The process of ﬁnding and removing the causes of failures in software. [8]

Debugging is thus a working process that is initiated after the completion of testing, whereas it has already been established that the software isn’t working as intended. The mere purpose of debugging is locating and ﬁxing the error, whereas the goal of testing is elicitation of the existence of the error itself.

2.4.2 Principles of testing

Burnstein lists 11 principles of testing [7, p. 27ﬀ], to a large extent derived from Myers’ book The Art of Software Testing. Many of these principles describe a testing process focused on a meticulous and rigourous approach, which might be hard to achieve in an Agile context. We include them as a good overview of values and practices that are valued in traditional testing.

1. Testing is the process of exercising a software component using a selected set of test cases, with the intent of (i) revealing defects, and (ii) evaluating quality. All software contains defects. This statement is important to embrace since it builds the very foundation of testing. By performing software testing these defects can be elicited in due time before the launch in order to evaluate and measure the quality of the software.

2. When the test objective is to detect defects, then a good test case is one that has a high probability of revealing a yet undetected defect(s). With defects being the foundation of testing, the focus of testing should be on areas of possible weaknesses and areas where the likelihood of ﬁnding defects is elevated.

3. Test results should be inspected meticulously. It is of great importance for the work following the presentation of test results that named results contain little or no room for misinterpretation. Ambiguous prerequisites or specifications for deciding whether a test result generates a pass or fail can cause tests to miss actual defects, or flag non-existing defects as failed. The first principle (all software contains defects) also applies to test cases; careful interpretation of test results is therefore required.

22 2.4. SOFTWARE TESTING

4. A test case must contain the expected output or result. While it is obvious that a speciﬁc input is needed in order to execute a test case correctly, it is equally important that the expected outcome or result exists; without this information, any conclusions on whether the test case passed or failed is impossible to draw.

5. Test cases should be developed for both valid and invalid input conditions. Just as all software contains defects, no input is ever guaranteed to be free of errors. Errors can be made due to external factors such as external data outputs or incorrect usage. Therefore all test cases must take into consideration the case of erroneous or invalid input conditions.

6. The probability of the existence of additional defects in a software component is proportional to the number of defects already detected in that component. Defects often occur in clusters. This is due to the fact that high complexity code, or badly designed code, is more likely to fail than easier and better designed code. This implies that if a large number of defects have been located in a component it is a high probability that a large number of still undetected defects exist; in this case a refactoring of the code would be a better solution than to ﬁx the defects found.

7. Testing should be carried out by a group that is independent of the development group. Since a developer writing code for a component of software has a set understanding of how that component works he may be biased. It may be hard for that developer to imagine where defects can occur, as well as imagining defect inputs. Throughout this report this type of bias will be referred to as knowledge contamination. An additional reason for the need of separate testers is that there may be a misconception in regard to the speciﬁcations, which may not be elicited else-wise.

8. Tests must be repeatable and reusable. For defects to be classed as defects they must be reproducible. If they are not, it cannot with certainty be said that a failed test result wasn’t a coincidence. Test cases that are clear and unambiguous will clearly point at the defect and will help developers to reproduce the error in order to ﬁnd the root cause of the defect.

9. Testing should be planned. There should exist plans for testing at all levels of testing and the objectives of the testing should be incorporated into these plans. This is necessary in order to ensure that suﬃcient time and resources is spent on testing activities, and to ensure that the testing is managed and monitored.

10. Testing activities should be integrated into the software life cycle. As software development as a discipline evolves, it is becoming increasingly important to include testing from the earliest time possible; preferably already in the requirements analysis phase. Certain test tasks, such as

23 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK

usability testing, can be performed using prototypes before a working interface has been implemented. 11. Testing is a creative and challenging task. A tester need to have cross competence and have understanding in both how software is developed and what problems that exists in the domain the software is to be deployed into.

2.4.3 Levels of testing

Acceptance System as a whole test - Customer requirements

System System as test a whole

Integration Component test group

Unit Test Component

Figure 2.5. Outline of the diﬀerent levels of testing

The process of software development occurs in several levels and as such testing also needs to be done in levels. Each level of testing has a specific area of focus and has an independent set of goals. An outline of the different levels is illustrated in figure 2.5. At the deepest level components are tested individually using Unit tests. Units, or components, are defined by The Testing Standards Working Party’s Living Glossary and Burnstein respectively as • a minimal software item for which a separate specification is available [8]. • the smallest possible testable software component [7, p. 137]. As units become ready and pass their respective Unit tests they can be integrated into larger groups where the individual units interact. Several units form a group of components, and these groups are tested using integration tests. Burnstein defines the goals for integration testing as 1. to detect defects that occur on the interfaces of units, 2. to assemble the individual units into working subsystems and finally a complete system that is ready for system test [7, p. 152]. What this implies is that while Unit testing focuses on the functionality of small independent components, and in some aspects the validation of the

24 2.5. SOFTWARE TESTING IN AGILE ENVIRONMENTS communication to and forth that component, integration testing focuses on the entire interaction between the group of units. Instead of working against stubs or drivers, the entire interface of the component is utilized as intended and can be tested in a more accurate environment. This level of testing should be performed continuously throughout the development as units become ready and can be integrated, and before the development of the entire system is complete. As development progresses and more subsystems pass their integration tests these subsystems can be integrated as parts of the whole system. System testing is applied on the entirety of the system as a whole, with all components integrated. Since this form of testing is performed on a complete system, testing can be complicated, costly and require a lot of time and resources. These factors need to be taken into consideration when planning the system testing. Such planning suitably takes place in conjunction with the requirement speciﬁcation phase. Constructing a proper test plan for the system test and executing it in a good manner also serves as a good preparation for Acceptance tests. System tests evaluate not only functional requirements, but quality requirements as well. Quality requirements incorporate aspects such as reliability, usability, security and performance. Which aspects of the software that are given focus during system test is dependent on the domain in which it is to be deployed and what time and resources that are available during testing. [7, p. 163ﬀ] Acceptance testing is when customers will determine whether or not the software developed meets the requirements set forth by the customer. This form of testing is performed by the customer, or representatives thereof, and is usually a planned lightweight version of the system test.

2.5 Software testing in Agile environments

In this section of the report we aim to cover related work as well as results pertaining to testing in Agile environments.

2.5.1 Quality Assurance in general

The Scrum process is iterative, consisting of several Sprints. Each Sprint consist of a commitment to turning backlog items into an increment of potentially shippable product functionality. This means that all deliveries should have a high expectancy of quality, a statement well in line with the Agile Sprint (see page 13). The report Mature Agile with a twist of CMMI written by Ruseng Jakobsen et al. is based on experiences from a company working with Scrum and Story Based Development. They state that “Explicit quality plans improves helps [sic] the team to build the right quality in” [10, p. 215]. Their experience is that such a plan

25 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK should be in the form of a QAS, a Quality Assurance Schedule. This document speciﬁes

• What stories are subject to inspection,

• What code is subject to review,

• What documents are subject to what types of review,

• What Unit test and automatic test is produced,

• What is included in the Acceptance test. The main purpose of a QAS is not documentary; it is intended as a tool to help the Scrum team to a clear definition on when a feature is done. This is also a necessity for proper planning. As the base for the report is a company using Story based development, they advocate the use of Story Completion Checklists. The intention of these checklists is to ensure the quality of the stories produced by structuring activities as well as defining when inspections occur [10, p. 215f]. An important detail to note is that this way of working requires someone to take the role of an inspector, although Ruseng Jakobsen et al does not define who this person is.

The subject of Quality Assurance is also covered by Lisa Crispin in her article Driving Software Quality (How Test-Driven Development Impacts Software Quality). Crispin covers several cases she has encountered in terms of Agile testing [11]. Crispin states that development dedicatedly using Test Driven Development as a test solution is a suboptimal solution. In one of the cases covered by the report the customer was reasonably happy with the software produced as it was fairly in line with the expectations; still there was a conflict between what the customer wanted and what was delivered. By changing the process and producing Acceptance tests before (or during the first days of) each iteration, a clearer picture of the customer expectations was perceived. As a consequence, the customer involvement in the product development was increased, and developers could more easily test what they were going to deliver. This also enabled the team to automate customer facing regression tests, as well as perform efficient exploratory testing; all to the result of a much more satisfied customer.

2.5.2 Guided exploratory testing

Adam Geras published a paper in 2008 regarding how to lead manual test eﬀorts in an Agile environment [12]. This paper was written from the test coordinator’s point of view and focused solely on the manual testing performed, including manual Acceptance testing. The way of working was inspired by Scrum and used Agile principles in an attempt to make the test process as lean as possible while still passing both internal and external audits.

26 2.5. SOFTWARE TESTING IN AGILE ENVIRONMENTS

Writing test cases and collecting test data was considered a waste due to the amount of time required to perform these tasks. The purpose was instead to take advantage of the testers’ experience and knowledge in the domain [12, p. 249], as well as customer involvement in regard to prioritization of what to test. Prioritization was made with a backlog, much like in Scrum. Instead of test cases, test targets were used. Test targets in this study were lists of items that were to be tested. Geras states that focusing on targets rather than on test cases gets quicker feedback due to improved communication. Discussing test targets also served as background for test eﬀort estimations, which were used for prioritizing what to test [12, p. 246]. Since the test team was using Scrum like principles, they also employed the use of a Burndown chart for visibility to the customer. A Burndown chart is graphical representation of the amount of work remaining over a ﬁxed amount of time (such as a Sprint) [3, p.11]. vs. time Scripted tests were also replaced with test guides for the highest prioritized test targets.

Geras concluded that the way they worked accomplished the goals they set out. Using test targets form the basis for managing an Agile test eﬀort [12, p. 250].

2.5.3 Test automation

Automation is at the heart of Agile software development, and the key to Agile testing. Puleio states in his report that Manual testing slows down regression testing and the feedback loop [13, p. 5], a statement that well coincides with the XP viewpoint (see section 2.3). One of the main conclusions that are drawn by Puleio is that all investments in automation, even at the cost of only a single feature being developed during a Sprint, will pay oﬀ in the long run. Functional tests and regression testing, if set up correctly, can be run either in a number of minutes time or overnight. [13, p. 6]

Experiences published by Ruseng Jakobsen et al. also support the need for test automation. Their conclusions support that when doing one month Scrum Sprints, all tests must be automated to the extent possible. This is important as automated testing is an integrated part of developing a Story, not a separate task. It is also of importance that the entire production line is standardized. This means that all environments (development, testing, staging and production) must be identical. This standardization allows for faster project startups, as well as faster working deliveries. In terms of automation, Ruseng Jakobsen et al. state that the use of Continuous Integration is a necessity, but it must be supported with discipline for checking in code in the repository. All code checked into the main repository must be working at the developers’ workstations before being allowed to check in. [10, p. 216]

27 CHAPTER 2. THEORETICAL BACKGROUND AND RELATED WORK

Automated functional testing

Puleio found that automated function testing was a great improvement in regards to regression testing after performing feature development [13, p. 3]. Before this was introduced, development progressed with regression testing being performed manually. Constantly prioritizing new feature development over regression testing caused a buildup of a ’test debt’. For each new development there was a certain amount of testing that was disregarded, eventually becoming a problem. Partial automation eﬀectively solved that problem and greatly shifted the time spent on testing towards actual development. Haugset and Hanssen list several advantages with automated functional testing in their report Automated Acceptance Testing: a Literature Review and an Industrial Case Study in the form of Automated Acceptance Testing [14]. According to their report several advantages of having access to Acceptance tests early, as well as automating them, existed. They found that

• Writing Acceptance tests before developing improved the developers’ understanding of the domain and the system that was being developed,

• Writing Acceptance tests gave developers a better overview of what the system was supposed to do in contrast to how it did it,

• The use of Acceptance tests contributed to the visibility and process control of Scrum [14, p. 34]. Haugset also elicits downsides, or costs, with this type of automation. One cost is writing the automated tests, a task that might take some time. Another cost is maintenance; maintaining automated tests costs time and eﬀort in the same way as maintaining an ordinary code base. Any time the system is updated, the test code base must be updated as well. Haugset therefore states the importance of evaluating and considering the potential beneﬁts of such automation against the cost in advance. It is also important to understand the implications of a possible false passed result of an automated test; for the test to be accurate and trustworthy, it must truly implement the intention of the customer or the requirement. If it doesn’t, it will create a false sense of control. [14, p. 37]

2.5.4 Professional testers in a development team

Traditionally testers and developers have been separate units in order to avoid knowledge contamination. In Agile environments this separation is no longer clearly required; it can even be seen as creating a bottleneck. The Agile Manifesto [15] states: Individuals and interactions over processes and tools.

28 2.5. SOFTWARE TESTING IN AGILE ENVIRONMENTS

This statement in the Agile Manifesto has proven to be of importance not only when referring to ordinary development teams, but also when integrating professional testers in Agile development teams. Talby et al., whose study focuses an organization based on XP practices [16], found that testers working independently found mostly minor mismatches between the developed application and the written requirements; much more seldom they found actual defects. These findings indicated that this was an inefficient way of working and that a more integrated approach between developers and testers is preferred. [16, p. 34] Besides the mere efficiency goals of increasing productivity, there are also other benefits of adding testers to development teams. Puleio [13] found in their study that including professional testers in a Scrum team increased cross competence in the team. Their study showed that in order to perform accurate task estimation deeper knowledge in the actual principles of the activities was needed; thus integrating a new role with different experience forced a wider knowledge, a common terminology and improved the group dynamics [13, p. 3]. These gains are important, since as Puleio notes: most developers are not good at estimations and generally underestimate the amount of time needed for an assignment, leaving little time to testing [13, p. 3]. Integrating testers into development focused teams also requires a change of mindset in the way of working. In order to efficiently be able to test features, tests need to be written before the actual development of the feature; this requires efficient communication between the different disciplines [13, p. 6]. This is also confirmed by Sumrell [17], who found that a change in roles and responsibilities where needed. Traditionally a QA team owns the responsibility of a product. Instead Sumrell found that all involved parties (Product Owner, Developers and Testers) need to work together on defining Acceptance tests [17, p. 3]. All of these reports confirm that the Agile Manifesto statement also applies to the integration of individuals from traditionally non-developer disciplines into developer teams.

Part II

Exploratory case study

Chapter 3

Method

Start Phase 1 - Exploratory case study

Research of related work and Observations relevant theoretical background

Group Interview

Likert Questionnaire

End Phase 1 - Exploratory case study

Figure 3.1. Phase 1 workﬂow

Our thesis work was divided into two phases. In this part of the report we will cover the ﬁrst phase, which was a case study conducted at the subject company section. The second part, the development and evaluation of potential process changes, is documented in Part III. The four main characteristics of a case study is that it is particularistic, descriptive, heuristic and inductive. Being particularistic is deﬁned as focusing on

33 CHAPTER 3. METHOD a certain event, occurrence or person. Descriptive means that the subject is thoroughly described. The heuristic trait means that it can deepen the reader’s understanding of the subject. Lastly, being inductive means the foundation of the study is based on the information at hand, and generalizations, terms and hypotheses stem from that information [18, p. 25ff]. The main purpose of a case study is that it is exploratory [19, p. 43]. The results obtained in this case study was used in the second phase, and are available in chapter 4. We made use of several methods in conducting our case study. The methods include observations, interviews (including group interviews) and Likert questionnaires. The flow of our activities in this phase is illustrated in figure 3.1. In the rest of this section we will describe these methods, and in the last subsection we discuss the validity of the research. In the following sections of this part of the report our aim is to produce an accurate description about the current development process as a whole, and the testing activities in particular. These sections are the results which we obtained from the methods used. Based on the results provided in this part of the report, a number of areas of particular interest are deducted. These are areas where we feel there is room for improvement.

34 3.1. OBSERVATIONS

3.1 Observations

We conducted research through observations in different phases. Observations can, according to Höst et al., be conducted with various degrees of interactivity with the observed subject. There is also a factor of the subjects’ awareness in regard to being observed. [19, p. 93] These two factors combined generate four different types of observations. The type of observation used by us differed throughout the project depending on the task at hand. In the initial phase of the project observations where made with little participation. Meetings were attended and occasional questions were asked in order to retrieve missing information. The type of observations made in this phase is best classified as observing participators; the group was aware of us being observers, but our primary objective was to gather information [18, p. 106]. Observations can be used as a tool to identify factors that might not be visible to those participating in it [18, p. 102], which was part of the motivation for using observations. The intention of our observations was to gather background information and to objectively identify factors in the working process that could be used as a foundation for discussions with the team members. In addition to our observations we also had unstructured, informal individual discussions with team members. In these we addressed either a single topic, or several smaller topics; these discussions occurred when a certain gap in our information needed to be filled or when a question needing a fast answer was raised.

3.2 Interviews

The interviews we conducted were semi-structured interviews. We had prepared guidelines and areas of interest for inquiries, but these were neither to be followed strictly nor did they specify everything in detail. The reason was that we were interested in the interviewees opinions and problem deﬁnitions. By doing semi-structured interviews we did not force them to adhere to our views and categorizations [18, p. 88]. These interviews were made in group and not individually, and were more in the form of focus group meetings. The motivation of this choice of interview method was made due to practical reasons. Firstly, to schedule individual interviews with all team members would be diﬃcult in regard to time. Secondly we wanted the team members to make use of each other and help each other in eliciting thoughts; since they are working as a group, we wanted their opinions as a group as well. The interviews were recorded on a digital medium and analyzed after each interview session. Notes were also kept during the interviews. During the analysis of the interview material the collected data was organized in groups by keyword or subject (for example ’Acceptance testing’ or ’Customer feedback’).

35 CHAPTER 3. METHOD

The interviews conducted resulted partly in complementary background information regarding the existing process and organization, partly in opinions regarding the current testing process. This information was then used to elicit certain areas of interest, while at the same time serving as a basis for the Likert questionnaire.

3.3 Likert questionnaires

One of our goals with this thesis work was to identify potential areas of improvements. We were also interested in identifying whether any part of these areas was perceived as a problem. To validate the findings of our observations and gather information on whether any area was perceived as being a problem we used a questionnaire of Likert items. A Likert item is a statement which the respondents are asked to evaluate from some perspective. In our questionnaire the level of agreement with a number of statements was measured. The response was given on a scale of five discrete steps, ranging from “Strongly disagree” to “Strongly agree”. The survey was conducted using a website, to which a link was emailed to the participants. All answers were submitted anonymously, a condition pointed out to the participants in the invitation email. The entire Likert questionnaire, as answered by the team members, can be found in Appendix B.A summary of all results can be found in Appendix C. The result from this survey was used when forming our suggestions for process improvements. As the Likert items are evaluated on a scale, we think that the answers might offer some information, however not conclusively, about the perceived severity any problems in the areas covered.

3.4 Validity of research

One of the risks of the case study methodology is that it can exaggerate or simplify certain factors in situations observed, which can lead to erroneous conclusions. They are also limited by the capabilities of the researcher. Inexperience in information gathering and interview technique greatly aﬀects the information that is collected [18, p. 47]. This risk is especially apparent when doing interviews. The use of leading questions can induce suggestion in the respondents answers, rendering the investigatory results invalid. At the same time leading questions can give access to areas the interviewer suspects the subject is withholding. [20, p. 45f] Since the quality of an interview is dependant on the interviewers experience as an interviewer, the quality of the answers might be aﬀected [20, p. 134]. This risk is applicable in our situation, as neither of the authors of this report has a high level of experience in conducting interviews. In

36 3.4. VALIDITY OF RESEARCH the interview form we chose for our investigations, there is also a risk of certain members not getting enough time or space to voice their opinions. This is an inherent problem in the group interview technique, and should be handled by the interviewer or moderator [21, p. 34]. The risk of ambiguity and leading questions also applies to the wording of the Likert items used. We do not believe that these risks affect the scientific validity of this report. As we use several different methods for gathering information, the risk of getting invalid data is reduced [19, p.118]. As with most research, the risk of bias is also present. As one of the authors of the report have previously worked with the subjects this was considered a risk in objectivity. This risk have been handled through a more critical viewpoint of that authors work by the second author. The validity can also be discussed in terms of generalization. As this report is based upon the specific preconditions that exist at the section where the thesis was conducted, certain parts of the conclusions drawn will not be entirely generalizable. Other aspects, such as the general question how should competence and responsibility be distributed within a team that is working with Agile developing methods will have a higher level of generalizability. The accuracy description of the Exploratory testing (see section 4.2.2) can be discussed. The description of this method is based on the information we obtained from the team members. However, since this method is not (yet) an established practice in the team we found that there were conflicting views on how it was supposed to work. We addressed this risk by following up on any uncertainties. However, it can not be guaranteed that the description given is completely accurate.

Chapter 4

Result

4.1 Organization

The section is responsible for building, customizing and maintaining a number of (software) tools used by the testing organizations at the same company. These test organisations issue requests for new tools, request changes in existing tools and submit defect reports for the existing tools. We will refer to the testing organisations as the customer throughout this report. The testing organisations are also the principle users of the products the section delivers, and in this capacity we will refer to them as the user.

Section Manager

Product Scrum Product Support Team Owner Master Owner

Scrum Team B Scrum Team Q

Figure 4.1. Outline of the organization

39 CHAPTER 4. RESULT

There are two main applications or systems amongst the section’s responsibilities. The ﬁrst is an application for automated testing of mobile phones; this application has been developed in-house. The second is a commercial oﬀ-the-shelf test management tool to which the section has added functionality and made customizations. In addition to this there are also a number of other smaller applications, most of them connected to one of these two larger systems. The section has therefore been partitioned into two groups, each responsible for one of the two systems. We will call these groups Team B (test automation tool) and Team Q (testing management tool). There is also a third group, responsible for user support, which we will call the Support Team. We have chosen to concentrate solely on Team Q in our case study.

The Support Team uses a similar Kanban system for current support issues, and Scrum for managing recurring tasks. Team B uses Scrum, and Team Q use either Scrum or a Kanban methodology for managing their tasks, depending on the situation and the tasks at hand. The Kanban methodology is only brieﬂy mentioned in this report (see section 2.3.1), and is used mainly as a reference for the reader; we will not further discuss this technique as we consider it to be outside the scope of this thesis. Both teams use several development techniques and practices, such as Test-Driven Development, pair programming and continuous integration. The domain in which our work will be conducted can thus be regarded mainly as a Scrum-managed Agile environment.

Each of the two teams have a Product Owner responsible for their respective projects. The ScrumMaster role is ﬁlled by one person, who shares her time between the two teams. The section’s organization is described by the diagram in Figure 4.1.

4.1.1 Scrum Team Q

At the time we arrived at the section, the team had been up and running for about 2 months. At the start of our investigation Team Q consisted of ﬁve developers who had previously worked together, either at the same section or on mutual projects. After Sprint 2, one developer left the section as well as the company. It was not clear at the time of writing if this team member would be replaced. This group of developers as a group will henceforth be referred to as the team. As mentioned in Section 4.1, the team shares a ScrumMaster with Team B.

In the current organization testing activities are all performed by the team, including integration testing. There are no dedicated testing role in the team, the developers share the responsibility of testing. There is also no separate test organization, such as a test team.

40 4.1. ORGANIZATION

Sprint types

The team mainly has two types of Sprints. Since the working process differs between the different types of Sprints, we consider there is a need to express the difference between the types of Sprints. The first type of Sprints contains new development. New development means that a large majority of the tasks in the Sprint backlog does not already exist in terms of code. Tasks can be new features, refactoring or completely new applications. The second type of Sprint is what we call maintenance Sprints, or defect fixing Sprints. The tasks in these Sprints are mainly focusing on finding and repairing the root cause of defects or deviations found in software put into production. Due to the fact that older applications lack support for the automated test frameworks, and may have been developed before Scrum or other Agile methodologies were introduced at the section, the development process in these Sprints is currently handled differently. Any differences will be described in the following sections, where applicable.

4.1.2 The customer role

The customer role in regard to Team Q varies depending on the task. The team is responsible not only for the development and customization of tools and support application, but also for the maintenance and operation of these after they have been put into production. This operation responsibility is a commitment in addition to the Scrum backlog items included in their Sprints. There are a number of diﬀerent situations where the customer role is diﬀerent from the other:

• Defect reports: These are mostly submitted by individual end-users of the system in question. In this case the Product Owner communicates with the team as well as the submitting user in order to establish both how critical the defect is and how complex a fix would be. In this situation the role of the customer is somewhat vague; the Product Owner has the voice of the customer, but cannot act as the customer. The team becomes responsible for validating that it’s fixed, but it doesn’t necessarily result in a delivery that can be Acceptance tested as per definition.

• Feature Requests: These are usually submitted by select individuals who have been appointed responsible persons in the organization. Per deﬁnition they are the customer, albeit the requests may be of such small magnitude that submitter isn’t involved in validating the completed work.

• Application Requests: These requests are for entirely new applications, for which detailed speciﬁcations and documentation is needed in order to start development. This is the situation where the role of the customer is most similar to the ’traditional’ deﬁnition.

41 CHAPTER 4. RESULT

This overall creates a situation where a clear deﬁnition of the customer is missing. In addition there can be several customers in each Sprint, customers who in the next Sprint are replaced by new customers.

4.2 Current process

The following sections contain descriptions on how the team works today. It is important to note that the following process descriptions are generalized and mainly explains how the team is intended to work; according to our observations these intentions and the actual work process used do not always coincide. Where these diﬀerences occur, it will be noted in the text.

Pre sprint Product Backlog

Sprint Product Owner prioritizes Use case based requirements Development

Corrections Sprint Planning Review poker Test cases based System on use cases Coding Debugging test Code review Sprint planning

Sprint Backlog

Figure 4.2. Outline of the current process in Sprints focusing on new development

4.2.1 Pre Sprint

Before a new Sprint is initiated, a series of events take place. These are intended as preparation for the upcoming Sprint. The intention is to produce an overview of the workload of the upcoming Sprint and to create awareness in the team about potential impediments. Firstly Planning Poker is used as an instrument to estimate upcoming tasks. Secondly Sprint Planning is conducted in order to plan all the available time in the Sprint. During both of these events, the only people allowed to attend are the ScrumMaster, the team and the Product Owner. This is to not skew or inﬂuence

42 4.2. CURRENT PROCESS the opinions of the team members. The team members are considered to be the experts on the subjects to be discussed, and the presence of external stakeholders could aﬀect their judgements. Instead it is the role of the Product Owner to speak for these stakeholders, as well as to ensure that their requirements are prepared and speciﬁed as items in the backlog.

Planning Poker

Team Q uses a procedure called Planning Poker to estimate the complexity required to achieve individual tasks in the project backlog. The Planning Poker takes the form of a meeting, about 2 hours in duration. All the team members, as well as the ScrumMaster attend. The ScrumMaster acts as a moderator, and tries not to take part in the planning. During observation sessions the Product Owner was present as an observer, but not necessarily for the entire session.

The term Planning Poker comes from the use of a deck of cards. Each developer is given a series of cards, depicting the values 0, 1, 2, 3, 5, 8, 13, 20, 40, and 100 respectively. A task is then chosen from the list of tasks to estimate. The task is presented by the team member who is most familiar with it. There can be a discussion about various aspects of the task that will inﬂuence its complexity, but it should be kept fairly short. Each developers then privately picks the card from his/hers set of cards that most accurately estimates the complexity of the task. To do this the estimator must rely on previous experience of similar task, domain knowledge and past performance of the team. When all estimators are done the cards are revealed. The developers with the highest and the lowest estimates must then explain their reasoning. After some bargaining and discussion a consensus is reached.

In planning poker for maintenance Sprints estimates are not only based on domain knowledge and past experience; in non-trivial cases an investigation has been undertaken during the previous Sprint in order to gain more knowledge on the problem at hand. By entering this information into the knowledge domain, the complexity estimation is improved. Nevertheless, these estimates tend to get a very high complexity rating.

Sprint planning

Before the start of every Sprint (usually the day before the new Sprint, the closer the better) a Sprint Planning Meeting takes place. The meeting is attended by the Product Owner, Scrum Master and the entire Scrum Team. In the current process, Sprint planning meetings usually start with a short session of Planning Poker to estimate newly submitted items for the backlog.

43 CHAPTER 4. RESULT

In this meeting the details of the items gone through in the poker planning is time estimated. Each item is broken down into several subtasks. Usually a task is divided into three subtasks: 1. Implementation, 2. test and code review, 3. corrections. The team internally agrees on the time needed for the various tasks in the backlog, and they are added to the Sprint. This continues until the Sprint is full. A Sprint is considered full when all pre-allocated hours in the Sprint have been assigned to tasks. The prioritization of the items in the Product backlog is done by the Product Owner, but it is the team who decides on what they are willing to commit to completing during the Sprint. This in turn may lead to negotiations with the Product Owner, as well as re-prioritization, but it is always the team members who decide on what they are committing to.

4.2.2 Sprint

Requirements speciﬁcation

Requirements are based on either change requests or vague ideas on new application. These requirements are usually vague at the start of the Sprint, and are not based on agreed use cases between the Product Owner and the customer. Requirements are usually kept in the form of use cases (for an example, see appendix A). Each of these use cases consist of prerequisites, a main success scenario and the expected outcome of the successful execution of the main success scenario. The prerequisites are a list of conditions that are required to be met before the main success scenario could be performed. An example might be ”User is logged in.” The main success scenario is a series of steps that are performed by the user, for example “User clicks Log out button.” The expected outcome describes what is expected to happen e.g. “Logout screen is shown.” Each one of these use cases also contains a number of extensions. An extension is an identiﬁed variant of the main success scenario. For example, it might be possible to log out using not only the “Log out button”, but also by using the menu option “Connection -> Log out”. The intended behaviour of the application is described for each extension. It is worth noting that during course of our observations, the way of writing requirements was not permanent. During one development Sprint the requirements were written in the form of User stories instead. Also, in parallel with our work a diﬀerent thesis project was conducted. As the purpose of that

44 4.2. CURRENT PROCESS thesis was to evaluate the current way of working with requirements, it is worth noting that the conditions on which we base our work may have changed as well.

Development

During the development process the team members are helped by a number of tools and subprocesses. Different tools and plugins are used at each developer’s workstation in order to assist in following processes and conventions such as coding and commenting guidelines. The development process has influences from Extreme Programming (see section 2.3) and preferably adhere to Test-Driven Development; the latter is mostly used in Sprints containing development of new applications and features and not maintenance Sprints. Since older code was developed without the Unit test framework, those applications are deemed not suitable for test automation without refactoring. If not using the Unit test framework (for example in defect fixing Sprints where support for the test automation framework is lacking), each programmer tests his own code while programming in order to see if it works. This means checking that variables get the right assigned values, and that the code follows the intended path. In other cases Unit testing is used for module testing. Each developer writes the Unit tests for the module he is working on. Once a coder in the team finishes his assigned task, the code is reviewed by another member of the team. The code is audited for obvious faults, logical errors, and for errors in relation to the coding guidelines. The developer who has written the code also tries to justify his choice of solution during the review. Errors found during code review are fixed and the changes are once again code reviewed. When no further flaws are found during code review the programmer can continue with the next task. All completed tasks are marked as done in the Sprint backlog. Implementation, code review, test and corrections are all planned as separate tasks. When the application is deemed suitable, some integration testing is also executed. This integration testing can be either automated or completely manual. In one case the entire backend of the application was tested to confirm that the correct data was entered in the database after passing through all non-GUI layers of the application. This validation was done manually and no records of the test results were kept.

System test

At the end of the Sprint all produced items are integrated and tested as a whole. Testing of applicable test cases related to the implemented tasks is performed on the integrated product. For defect ﬁxing Sprints, the team themselves chooses what regression testing is needed after ﬁxing a defect.

45 CHAPTER 4. RESULT

Previously scripted manual step-by-step test cases were written for the applications the team was responsible for. Over time the developers noticed that this method of testing was ineﬀective. Some of the test cases took a lot of time to update when the application changed, resulting in the developers not using them as intended. Instead they performed tests made up on the ﬂy. This method of testing was scrapped in favour of an exploratory test method.

Exploratory testing The exploratory test method is intended to be used during System testing and Integration testing. Since this method is not (yet) an established practice in the team, the following is a description of how it is supposed to work. The team starts the testing with a quick meeting, deciding on what areas of the application to test. When this has been decided a time-boxed testing session starts. The length of each box should be not be more than 2 hours; 90 minutes is the preferred length. All testing sessions should have a person in the role of a moderator, who overlooks the testing. Within these sessions the developers both test the main success scenario and utilize their own experience to test other aspects of the application. The intention is to localize areas where the application may have defects and to take notes of this. It is also the intention that notes should be taken during the testing in order to be able to reproduce the test path taken later. After each time-box, the team has a meeting to discuss any defects found. These defects are then fixed, and another iteration of testing is performed. This process is repeated until the System test is over. Since the change of test methodology, the team has only performed System testing in this manner once. During this system test several team members where absent, which had to effect that the entire System testing was performed by only 2 developers with assistance of the Product Owner. This caused several problems, primarily a lack of time and resources. This in turn lead to not all rules for the testing being followed. Most notably the test cases were not updated at all. Notes on failed test results where kept on paper only, and were discarded after the defects where corrected. If performed the way the team described the process, all testing is to be documented during the test in order for the tests to be reproducible. Each time box is supposed to be prepared beforehand, with a clear goal on the area of testing. Due to lack of time, these preparations were somewhat lacking. The testing is also meant to be performed with a moderator, a role that was filled in the last minute by the Product Owner. Also worth noting was that during this instance of system testing no regression testing was performed after correcting defects. The developer fixing the defect merely tested that the intended functionality was working. All of these setbacks were perceived to be caused by having too few resources and insufficient time.

46 4.2. CURRENT PROCESS

Test case design The exploratory testing method caused changes in the test case design. The old scripted test cases were written based on the different Use Cases. For each Use case, test cases were written in the form of test manuscripts for manual execution. One test case was written for the main success scenario and at least one for each extension. These test cases were written in a form closely resembling the Use Case format, with the exception that each step in the Use case got a specification of the expected outcome of that specific step.

The new test case strategy intends to utilize previous testing results in the testing process. Initially, the test case consists of 2 step:

1. Execute main scenario.

2. Make useful notes on results, and comments on testing.

This way of working has several goals. Firstly, it should eliminate duplication of important information. In the old way of working, the prerequisites of the main success scenario often had to be speciﬁed both in the requirements and in the test case. If a prerequisite changed, this had to be changed in the use case as well. This not only meant a doubled work eﬀort, but also added the risk of discrepancies between test case and requirement.

The second step is considered to ensure that the test case matures and evolves. The philosophy stated for this is that the ones with the best knowledge of the application are the people who designed it. They are also considered to be experts in knowing where things usually go wrong, thereby the most capable people updating the test case. These comments and notes do not make complete test cases in the traditional form; the base of this testing is more exploratory and leaves freedom to the tester.

During defect ﬁxing Sprints it is the intention that when working with an application where the test cases are in the old step-by-step form, that these test cases be rewritten in the new form for the exploratory testing. None of these test cases are automated; all tests cases produced based on the Use case requirements are manual tests.

4.2.3 Post Sprint

For the team followed there are two activities after each Sprint: the Sprint Review and the Sprint Retrospective. A description of these can be found in section 2.2.2 of this report.

47 CHAPTER 4. RESULT

Sprint Review

For the team, no real Acceptance testing occurs. Due to the unclear role of the customer and not having any clear requirements on the delivered product, this level of testing is instead replaced with a demonstration of the product. All ordering parties are invited to this demonstration, which is performed as a planned meeting in a meeting room. This meeting constitutes the Sprint Review at the end of each Sprint.

48 4.3. POTENTIAL AREAS OF IMPROVEMENTS

4.3 Potential areas of improvements

One of the purposes with our thesis project was to identify areas in the testing process where improvements were possible. This is something we kept in mind during the work of describing the team’s development process. In this section we present our ﬁndings, which served as a basis for our process change suggestions in Chapter 5 in Part III of the report.

Some items in this section were reported as problems directly by the developers, others were uncovered by comparing the circumstances at the team to situations in relevant literature. We were interested to know if the latter were perceived as problems by the team. We investigated this using the Likert questionnaire described in the Method section of this chapter (see section 3.3). Four of the ﬁve developers answered our questionnaire. The entire Likert questionnaire, as answered by the team members, can be found in Appendix B. A summary of all results can be found in Appendix C.

4.3.1 Customer involvement in validation of deliveries

Finding: A lack of proper Acceptance testing makes it hard to validate if the products delivered is what the customer wanted. During the group interviews the team members described the customer involvement in validation of functionality as very low. The functionality is demonstrated to the customer at the Sprint Review, but during the Sprints the developers are sometimes not sure if they are building the right functionality.

Although the team is inﬂuenced by the principles in XP, this is one of the areas where they are not. In XP validation of the product is handled through the process of Acceptance tests. The customer is responsible for verifying the correctness of the test cases. The test cases themselves are based on User Stories, which are written by the customer themselves, or with help from the developers. Acceptance tests in XP are always automated. We asked the team what the thought about the missing Acceptance tests, during the group discussion. One developer said:

“But the Unit test have nothing to do with the customer, it should be a set of Acceptance tests, that should be written by the customer. But we have no Acceptance tests at all. [...] it would be great if we had Acceptance tests from the customers. That we could actually run through them and say “This is actually working the way they want it to work.””

During our time at the section, this problem became clear when it was discovered on the morning of the same day as the Sprint review that there existed additional

49 CHAPTER 4. RESULT customer requirements. This requirement was discovered by the team members themselves. “Last time at the demo [the Sprint review] when we supposed to present the first version of the application [...] we figured out the same morning that ’If we’re going to demonstrate this, we must figure out what to demonstrate’[...] And there was actually a [new] customer requirement that we found out by ourselves, so we had to fix it. They [the customer] had no Acceptance test for that meeting, but to set up a demonstration we had to create a requirement ourselves, that they should have given us [...] or the Product Owner should have told us.” This was corroborated by the answers to the Likert item Not having access to Acceptance tests at an early phase of a Sprint makes it harder to validate that the product meets the user’s needs and expectations, shown in Figure 4.3.

Figure 4.3.

An interesting result was the answers to the statement There is no need for the customer to write Acceptance tests as their requirements are sufficient to decide when a feature is done. As shown in Figure 4.4, the opinions are somewhat divided. We think that this might be due to the fact that the level of detail of the requirements differs depending on the customer and application. One factor that is causing the low level of customer involvement is no doubt the in-house situation, the fact that the customer is in fact a different section in the same company. But the role of Customer in an Agile project comes with more commitment, than does the role of ’submitter of a feature request to the Support department’. We feel that the customer in this case sometimes is somewhere in between these roles. All respondents to the Likert questionnaire answered ’Agree’ with the statement The customer is satisfied with getting a demonstration of the product instead of an Acceptance test. This indicates that this is foremost a problem for the team, and not the customer.

50 4.3. POTENTIAL AREAS OF IMPROVEMENTS

Figure 4.4.

Figure 4.5.

Two of of the four statements in the Agile Manifesto (see ﬁgure 2.1 on page 13) are particularly violated due to the problem mentioned. Firstly Customer collaboration over contract negotiation. It is clear that the customer collaboration is lacking, while at the same time there is no contract negotiation; the latter is due to all development being performed towards internal customers within the organization. The second aﬀected statement is Working software over comprehensive documentation. The lack of Acceptance testing and customer feedback makes it hard to validate that the delivered software is meeting the customers expectations both in regard to quality and how it works.

Several of the Agile Principles (see 2.1.2) are also violated due to this situation. Deliver working software frequently means delivering software that satisfies the users needs (is functional) and is working. In order to be able to achieve that goal, and run an Agile project, there must be frequent and significant interaction between the customers, the developers and the other stakeholders. The most clearly violated principle, which applies to this problem is Working software is the primary measure of progress. As this principle states that an Agile team should measure their progress in terms of customer needs met, it is impossible to act in accordance with this principle without customer involvement and clear definitions

51 CHAPTER 4. RESULT and requirements of the products requested. If projects depend on customer feedback on any level, and this involvement isn’t suﬃciently fulﬁlled, Sutherland et. al. state that it is necessary to “communicate to the appropriate level, individual, or group in the organization to allow for corrective action, as corrective action may be beyond the scope of the project team” [22, p. 3].

4.3.2 Testing goals - external expectations, internal commitment

Finding: There are neither requirements nor expectations on what testing should be performed on products delivered. During the group interviews the team members discussed that no requirements on the testing existed on either the section or within the team. Likewise, no team member could recall such requirements ever been supplied. This lack of requirements covered all aspects: how to test, what to test and the amount of testing that should be performed. That this was considered to be a problem for the team was also corroborated through the Likert statement It’s a problem that we don’t have a clear picture of what’s required of our testing where the answers indicated a quite high level of agreement (see ﬁgure 4.6)

Figure 4.6.

Finding: The only customer expectations in regard to product quality known to the team are that the products should work. Finding: Very little feedback is received from the customer, unless critical errors occur. The topic of quality expectations was covered during the group interviews. There seemed to be a consensus within the team that they didn’t know of any real customer expectations on the products. One of the team members answered the following:

“I guess the only expectation from the customer is that it should work.”

52 4.3. POTENTIAL AREAS OF IMPROVEMENTS

This indicated that there were no known customer expectations on how the products worked, as long as they got the job done. This could also be seen in the Likert questionnaire, where the statement We seldom get feedback on our products unless critical errors occur answered with agreement (see ﬁgure 4.7).

Figure 4.7.

There were also further indications during the group discussions that there were no known quality expectations from the customer: Developer 1: “It seems like the acceptance level quite high here. I mean, compared to if we were working in, like, medicine. Then the acceptance level would be of more importance” Developer 2: “Or if we were to sell our products to external companies.” These results were similar to those regarding customer involvement in Acceptance testing (See section 4.3.1). Just as one developer stated this was likely due to the fact that the development was performed for in-house customers. The answer to the Likert question The testing we’re doing is sufficient to the need (see figure 4.8) should therefore be interpreted with caution. If the customer shows no interest in neither the product quality nor the testing thereof, the testing may be considered sufficient regardless of the relation between the amount of testing and the quality of the product.

4.3.3 Automation

Finding: Automated functional testing is perceived as something that could be useful. However, it is uncertain whether the cost of using it in all applications is worth it. Automated testing is an important part of Agile software development. Extreme Programming, from which the team is inﬂuenced, states that all testing should be automated (see subsection 2.3), or as Beck states [5, p. 59]:

53 CHAPTER 4. RESULT

Figure 4.8.

“Any program feature without an automated test simply doesn´t exist” In the current process a certain degree of automated testing is performed (see 4.2.2), mainly in the form of Unit testing when developing new products. During the group interviews it was indicated that no refactoring was done to introduce test automation for older applications. The reason stated for this was that it was deemed too expensive by the product owner. This stand is partly supported by Beck who states (in the context of retroﬁtting XP) that you shouldn’t go back and write automated tests for existing code. Instead, tests should be written on demand when needed, such as in the following situations [5, p. 126]:

• When adding functionality to untested code, • When fixing a defect, • When refactoring is needed. As a response to a question regarding adapting more automation, the team also expressed that automated Acceptance testing would be beneficial. “I would actually hope for automated testing for Acceptance test [...] then you would know that the customer is going to be happy.” Even though this automation would be beneficial, it would not be a an obvious choice. “It depends on the size of the application, if a system test takes too long to do, it might be worth doing.” As automated Acceptance testing would eliminate a large part of the current system testing the later statement regarding system tests at first seem partly contradictory. Therefore it is worth emphasizing that parts of the applications developed and maintained by the team are very small in size. In that context, the statement instead hints at automation being like bringing a gun to a knife-fight.

54 4.3. POTENTIAL AREAS OF IMPROVEMENTS

Finding: There is no coherent strategy in regard to test automation.

As the team was formed only a couple of months ago, and not all of its members have been using Test-Driven Development practices earlier, the degree of automation used varied depending on the developer. When asked if there was a coherent strategy on how to use Unit tests, the answer given was that they had not had any discussions regarding that subject. Instead they used a best-eﬀort strategy, where each developer developed Unit tests as they saw ﬁt.

4.3.4 Planning and structuring the testing

Finding: There is no deﬁned process to choose the scope of the testing.

Finding: The ’done’ criterion for the testing is often “We have no more time” or “We haven’t found any defects in a while”.

The team has decided to perform Integration and System test with the exploratory testing method described in the previous section. It seems, however, that most frequently both Integration and System tests are performed more like ’happy testing’. The developers run the application, making the test cases up as they go along. If more than one developer is performing the test, the developers might split the application and test some parts each. What to test and how is decided by ’common sense’ by the developers.

The defects that are found are ﬁxed and a new testing session is performed. In one of the group session a developer expressed that they stop testing when they run out of time or if they ﬁnd no more defects.

According to the Likert questionnaire two of the team members strongly agreed to to the statement We often cut back on testing when we are pressed for time. As seen in Figure 4.9, one developer did not agree.

At the same time three of the four developers disagreed with the statement We generally estimate too little time for testing. (Figure 4.10)

Currently there are indications that the team members’ views on how well their products are tested differs. The Likert question When we are done with a System test, I can make confident statements on how well each feature has been tested generated mixed results, and lacked a clear consensus (see figure 4.11). On at least one occasion the System test was performed by two of the (at that time) five developers. If not all members are present, this will of course lead to differing knowledge within the team on the quality of the application.

55 CHAPTER 4. RESULT

Figure 4.9.

Figure 4.10.

4.3.5 The tester role

Finding: There is no clearly perceived need of a test expert in the team, but it is considered a problem that no one is clearly responsible for the testing. Customers neither can’t nor shall write functional tests by themselves. Customers should provide test data which can be translated into tests. Beck suggests that this should be done with the help of a tester. This tester should also over time create tools that let customers write, maintain and run their own tests. According to Beck, this is one of the reasons XP teams always have at least one dedicated tester [5, p. 118]. Our results from the Likert questionnaire did not clearly conﬁrm there was a perceived lack of dedicated testers in the team. The question It’s a problem that we dont have any test experts on our team did however have a maximum spread

56 4.3. POTENTIAL AREAS OF IMPROVEMENTS

Figure 4.11.

Figure 4.12. on the answers (see ﬁgure 4.12), indicated mixed opinions in regard to the subject. The reason for this could either be due to various reasons, depending on what the participants include in the deﬁnition of a test expert.

The developers generally agreed with the statement It’s a problem that there is no one that is clearly responsible for our testing, indicating a perceived problem within the team (see ﬁgure 4.13).

Finding: There is an expressed wish for external testers of products delivered. Finding: There is a concern for knowledge contamination both in regard to current team members and a potential dedicated tester in the team.

57 CHAPTER 4. RESULT

Figure 4.13.

As Burnstein cites in her principles of testing (see 2.4.2), “Testing should be carried out by a group that is independent of the development group”. The team today consists solely of developers influenced by Test First philosophies, and does at this date have neither dedicated testers (see 4.1.1) nor customer or end-user influence on the testing (see 4.3.1). As it is today, another person besides the developer writing a specific piece of code reviews that developers work. Answers during the group interviews indicated that this was a potential problem, or an area in which there was room for improvement. “It would be better to have others testing, outside the team. It would bring new eyes on the application [...] on how to use it. If you know how it’s supposed to work, maybe it is hard to click on the wrong button.” Continuing on that track, discussing knowledge contamination raised the following statements: Developer 1: “To use the application in the wrong way, I think, that’s hard.” Developer 2: “Yes, it’s hard to misunderstand on purpose.” These answers indicated a need for someone with more test expertise. This should preferably be external testers, as there was a perceived risk of having testers on the team: “Moderator: Does it have to be someone outside the group?” Developer 1: “Maybe an end user”. Developer 2: “Even if the person is not a developer, if they’re in the team they would still know how the application works. It would be better to have someone from outside the team, I think. [...] since we set up the [application] we know how it is supposed to be used, there is a big chance that we going to use it that way.”

58 Part III

Process change proposals

Chapter 5

Test process change proposals

Based on the results from the exploratory case study (see chapter 4) and books and publications regarding related work (see chapter 2) we propose a number of change suggestions, in total affecting several aspects of the testing process as a whole. Clear definitions and requirements of what is required of the software produced as well as the quality thereof is a necessity. As there currently is a lack of customer interest, this does not always exist. In parallel with our research a second thesis work was carried out, investigating possible changes in the way of working with requirements. Depending on the results of that thesis, the problem with nonexistent or unclear definitions and specifications might be eliminated. If so, a number of other improvements become possible, all of which improve different parts of the entire development process. The following sections contain our suggestions directly relating to the testing in the Agile development process at hand. Each of these suggestions are designed so that it can be used and evaluated individually. While each of them can be implemented individually, they are all in various degree depending on the customer being more involved in the development process. The first proposal is to have an experienced expert tester in the team, as related studies have found that this is beneficial (see section 2.5.4). This proposal is presented in section 5.1. The second proposal is to move towards increased automation, as automation is one of the foundations of Agile development. This proposal is presented in section 5.2. Our last proposal is to Structure the testing, to have a clear plan for what should be tested and how. This proposal is presented in section 5.3. These results where all evaluated using a group interview. The results of the evaluation can be found in chapter 6.

61 CHAPTER 5. TEST PROCESS CHANGE PROPOSALS

5.1 Have an experienced expert tester in the team

While there is no perceived lack of professional testers in the team, there is some consensus regarding the risk for knowledge contamination (see 4.3.5). Currently, the people testing the application are the same individuals that is responsible for the design, the architecture and the implementation of the application. They are also responsible for taking decisions on what and how to test the application. While this is not perceived as a problem in itself, it affects the testing in terms of contamination. The developers themselves become responsible for validating the product from the end-user perspective, which presents a risk for a conflict between what the customer wanted and what was delivered. This situation is similar to what Crispin describes [11], and which is covered in section 2.5.1. Various reports have shown that there are significant advantages of including professional testers in an Agile development team. These advantages outweigh the negative impact of knowledge contamination (see 2.5.4).

• An increase of productivity • An increase of cross competence in the team • It establishes a wider knowledge, a common terminology and improves the group dynamics

• Estimations on test complexity and time estimations are improved • Forces a change in the process to involve stakeholders (developers, tester and product owner) in producing Acceptance tests The last item listed is one of great importance for the team in subject in this report, as there currently is a lack of Acceptance testing and customer feedback. Previous studies performed on the area of Acceptance testing has shown that writing Acceptance tests before developing improves the developers’ understanding of the domain and the system being developed. It also provides a better overview of what the system is supposed to do in contrast to how it does it.

Change proposal

We propose the use of a professional tester by the team. It should be a skilled tester with much experience. This tester should be a full time member of the team, and not a resource shared with other teams. Initially the tester should have the role of test expert in the team. The goal of this is to promote a deeper understanding and nurture the cross competence in the team. It should be the responsibility of the professional tester to assist the team in writing and preparing test cases. In the case of Guided Exploratory Testing the professional tester should assist in constructing test guides. The scope of the testing, for example

62 5.1. HAVE AN EXPERIENCED EXPERT TESTER IN THE TEAM what is included in a System test, should be determined in conjunction with the Product Owner. These choices should be based both on risk, but also by using a different perspective (user centric) than that of the developers. It is important to note that the person filling this role should not be responsible for performing the actual testing; this responsibility is shared by the team collectively. The main purpose of the introduction of this role is to force the gains possible, stated in the parent section to this. Over time knowledge transfer occurs, a common terminology is established, and the level of professional test competence in the team increases. During this time it is also the intention that the person in the professional tester role should move towards filling a developer role. Just as the existing developers get a deeper understanding in the discipline of testing, the professional tester also get a deeper knowledge in development.

63 CHAPTER 5. TEST PROCESS CHANGE PROPOSALS

5.2 Move towards increased automation

In the current process some test automation is being performed on new applications that are being developed. The level of automation as well as the choice on where to use it is decided by each individual developer; neither the team nor the section has a policy in regard to the level of automation. The automation is also isolated to Unit tests only, but there is an expressed wish for performing automated Acceptance testing where applicable (based on the cost of introducing it in time versus the size of the applications being developed). For existing applications which haven’t been developed with test automation in mind, no automation is being introduced. For details regarding the current situation, see section 4.3.3.

Change suggestions

We propose an increase in test automation. For all new applications we suggest:

• Create an automated testing framework from the very beginning,

• Automate all Acceptance tests.

For existing applications, that currently lack the support for the test automation framework, we suggest following the XP retroﬁtting practices as stated by Beck [5, p. 126]:

• When you need to add functionality to untested code, write tests for its current functionality ﬁrst.

• When you need to ﬁx a defect, write a test ﬁrst.

• When you need to refactor, write a test ﬁrst.

This will initially make development slow, as a lot of time will be put into developing tests. Over time this will change, and the parts of the code visited often will look and feel as code that was developed from the ground up with test automation in mind. This will also mean that the parts of the code most frequent to change will be thoroughly tested as well [5, p. 126f]. This choice is of course under motivation of cost, but we highly recommend it. As Test-Driven development favors constant refactoring and writing what XP calls the simplest code possible that passes the test, an introduction of automation also serves as documentation or readability of the code. This would be particularly true for automation of Acceptance tests, since they can be seen as the ultimate documentation for a feature. Martin states that:

64 5.2. MOVE TOWARDS INCREASED AUTOMATION

“So, just as Unit tests serve as compileable and executable documentation for the internals of the system, Acceptance tests serve as compileable and executable documentation of the features of the system.” [2, p.27] Or to paraphrase Beck: “Anyone who can’t ﬁnd the rest of what they need to know from the code and the tests [besides a general description of the main objects in the system] have no business touching the code.” [5, p.156]

65 CHAPTER 5. TEST PROCESS CHANGE PROPOSALS

5.3 Structure the testing

This proposal is very much designed to be easily integrated into the team’s current way of testing. It is intended as a complement to the exploratory testing the team intend to use. The proposal is an eﬀort to structure the process, with the aim of enabling the team to make more informed decisions about the testing. This proposal is intended to be used in both System and Integration testing.

Change suggestion

We suspect that the team might benefit from structuring their testing effort in a number of areas. We describe these areas here. But we feel that the most important aspect of this proposal is to introduce a more clearly defined testing process. This process is far from complete or as detailed as one would want, but the important part is to create something to work with. We want the team to start to consider testing as an important part of software engineering, and most of all take the time to reflect over and improve their testing.

Responsibility: On several occasions the team expressed a wish for someone to be responsible for the testing, a ’test leader’ or ’someone to monitor/administrate the testing’. We suggest that the team shares this responsibility amongst themselves. This means that decisions concerning the testing is either taken by the team together, or if taken by an individual developer, reviewed by someone else in the team. Just as code design and implementation is subject to Design and Code reviews, testing decisions and execution is subject to reviews.

Planning: Before every testing session an informal meeting should take place. This should take place as close to the testing as possible. It’s important that as many team members as possible takes an active part in this meeting. The objective of the meeting is to determine the scope of the testing. Every testing effort, in every organization, is limited by time and resources. It is impossible to test everything. Therefore a very important part of testing is the process of choosing what to test. We propose that the team tries to prioritize what to test, and test the most important items first. They should consider the following factors in their decision. Value, that which brings the most value for the user. The features that are used the most. Risk, the most likely features to contain defects, typically the areas of the application that has been changed recently. Also areas that has not been covered by Code reviews or Unit tests. This includes analyzing which part of the application a change is affecting. Changes in a lower layer of the application might affect several features in the application. Coverage, areas which have not been tested for a long time, or even ever. The result of the planning meeting, must not necessarily be a written plan, but should result in all team members knowing what to do for the upcoming test session.

66 5.3. STRUCTURE THE TESTING

Done criteria: We propose that the developers take a joint decision about when a feature is suﬃciently tested. Each developer summarizes his/her testing, and the other team members suggest scenarios that can be tested. Each scenario is discussed, and can be considered important enough to test or not. The testing should not be considered ’done’ until there are no more important test cases left untested. The process makes the whole team aware of what has been tested, but equally important, what has not been tested. This is an important factor in estimating time for future testing eﬀorts and forming an opinion about the quality of the application.

Chapter 6

Evaluation

The following sections contain the results of the evaluation of the change suggestions presented in chapter 5. The ﬁrst section contains the description of the method we used to evaluate the suggestions. The following section, section 6.2 contains a summary of the team’s opinions on the suggestions.

6.1 Method

The change suggestions were evaluated by the members of the team through a group interview in order to elicit their opinions. Based on our suggestions and the results of the evaluation thereof an implementation suggestion was written. The goal of the resulting implementation suggestion is to be more speciﬁc to the circumstances at hand, and to provide a more concrete set of improvements that can be implemented. This suggestion has not been evaluated.

6.1.1 Group interview

To evaluate our change suggestions we conducted a semi-structured group interview with the members of the team. Just as during the group interviews conducted during the exploratory case study we chose to do these interviews in group (see section 3.2). We wanted the group’s opinions as a whole, and that the team members might help each other elicit opinions through discussion. During the interview we presented each suggestion orally. During these presentations the team members had the possibility to ask questions, and to get clariﬁcations if needed. The questions we were seeking to get answers to were:

69 CHAPTER 6. EVALUATION

• What good and bad aspects do the team think exist regarding this suggestion? • What diﬃculties do they see in introducing it? • Would they be interested in trying this suggestion? Why/why not?

6.1.2 Validity

The same validity issues as mentioned regarding interviews in the first phase of this project (see section 3.4) applied to this interview session as well. In addition to this, one team member chose not to participate during evaluation interview; it is our opinion that this did not affect the result. Another validity issue regarding this interview session is that we were pressed for time. As we only had access to the team for a single one-hour interview, any validity aspects concerning the interview may have had a bigger impact than wished for in terms of clarity. Our inexperience as interviewers could make it hard to elicit clear and honest opinions. Judging from the results from the evaluation, it is not our belief that this is the case. The answers we got were thorough, and the interview subjects made sure to ask for clarifications during the discussions. It is clear that the opinions and evaluations we obtained regarding our proposed change suggestions are affected by external factors. Factors such as understaffing or having a high workload have an effect on the attitudes towards changes; in some cases it also makes changes impossible to carry through. One example of this can be seen in the group evaluation regarding an increase in automated testing. While there was an agreement that a high level of automation would provide good test coverage and produce well tested code, there was reluctancy in regard to the time needed to produce such automation. It would also possibly require more equipment, such as a secondary build server. Both of these factors were resource related, factors out of the team’s control. The same resource problem was also visible in the discussions regarding the proposal of having a professional tester on the time. Having lost one developer resource already, the need of filling that gap was considered of higher priority. Problems mentioned earlier as well as time constraints connected with thesis projects pose a limitation to the results we present.

70 6.2. TEAM OPINIONS

6.2 Team opinions

The following sections contain the opinions of the team in regard to the suggestions presented in chapter 5.

6.2.1 Have an experienced expert tester in the team

Summary: The team was positive to the idea of having access to professional testers as a resource, albeit seeing a lack of personnel resources as a problem. In regard to resources it was the opinion of the team that they saw the most beneﬁt of having dedicated testers as a resource outside of the team, possibly as a shared resource with the other team. The main beneﬁt of a professional tester was considered the testers ability of ensuring that what was being developed matched what the customer wanted.

Initially in the discussions the team was hesitant to the suggestion of having a professional expert tester in the team. The reason stated for this was not not due to a reluctancy in regard to the tester itself, but due to a lack of resources. If they were to choose between having another developer on the team or having a professional tester based on the current situation, there was a consensus that another developer would be the obvious choice. When asked to disregard the current resource situation and instead focus on the theoretical beneﬁts, other aspects came up. One of the positive aspects that was elicited by the team members was that a tester would be capable of bridging the requirements and the actual work produced. As testers are good at translating user data into tests, the developers would know that they were on track with what they were doing.

When discussing the role a tester would ﬁll the members of the team showed more interest in having a dedicated tester. As there are no dedicated testers as of today, this was considered to be a big potential beneﬁt. This tester role was not discussed as a member of the team; instead it would be a shared resource between all teams, in order to not to turn into a wasteful resource.

6.2.2 Move towards increased automation

Summary: In both the cases of new and older software the team were positive to the principle of having a high level of automated testing. What they saw as a downside was the cost of the automation, as it tended to take long to implement as well as slow down their use of Continuous Integration.

71 CHAPTER 6. EVALUATION

Currently all of the applications developed by the team have an architecture where the UI is separated from the business logic. This has enabled the team to run certain automated integration tests, from the top of the business layers all the way down. What they experienced as problematic with this approach was that even if they got good test coverage as well as validation that their code was working, the tests took a long time to run. This was not ideal for the current Continuous Integration setup. If there was the possibility of getting a second Continuous Integration setup which would be running these tests nightly, then it might be worth looking into.

When discussing Unit tests and coverage they had previously tried to maintain an 80% coverage rate. What they experienced as problematic was that whenever they needed to refactor the code, or do a split of a function, the total time needed for the task increased drastically. If the code had a good level of coverage, updating all test cases consumed a lot of time. The general opinion was that it was better to decrease the coverage and let the coverage of the integration tests be suﬃcient.

Retroﬁtting automation for the legacy products was considered interesting, and something the team were interested in looking into and adapting more. To some degree they where doing this in their current way of working, but not as a coherent strategy. While being more positive to the suggestion in regard to legacy products, there was still reluctancy in regard to the cost. As several of the teams tasks involve updating the existing software and ﬁxing defects, there were several situations where the time needed to write and/or update automated tests would be too costly.

6.2.3 Structure the testing

Summary: It is seen as problematic that no clear Done criteria exist. It is also considered hard to prioritize what to test based on value, as very little is known regarding what is valuable for the customer.

During the discussions regarding this suggestion, the team agreed that this is similar to what they have been trying to do in the past. They had planned their testing on a whiteboard.

The team does consider it problematic that they don’t have a clear done criteria. Currently the criteria is ’We’re out of time and it’s time to release’. As long as the software is semi stable, no more time is required by the team. With the guided exploratory testing the done criteria the team would wish for is instead ’Can everyone agree that we’ve tested all we can, and that nothing has been found during the last time boxed session’. This last criteria was used when ﬁxing defects in one application; during this iteration the team worked using the Kanban

72 6.2. TEAM OPINIONS methodology and found that they were less stressed than when performing testing in a usual Scrum Sprint. The team saw a problem with prioritizing the testing by value. They did not feel they knew enough about the customers or users prioritization to use this factor. They reported that this have had the eﬀect that a lot of time have been put into the development of features, that later turned out not being used. When asked they said that the prioritization of the backlog did not give them enough information to do this type of evaluation.

Chapter 7

Implementation suggestion

In chapter 6 we presented the team’s opinions on the different suggestions in chapter 5. Using what we learned from this evaluation, and our overall study of the section, we have put together a new process. In the new process we keep the session based exploratory testing, but add some structure to this testing effort. We add Acceptance tests to the process. These tests will be written for every feature that the team commits to. The tests will be created by the Team and must be approved by the Product owner. We further suggest that the Acceptance tests should be automated. This will lead to a process in which the most important functionality is covered by continuously executed automated tests, and the rest of the testing effort is spent on a focused exploratory testing. The following subsections outline the process.

7.1 Acceptance tests

Since there currently isn’t any real Acceptance testing being performed, it is our opinion that this should be introduced. This will create a more clearly deﬁned done criteria, and will require an increase in customer involvement.

7.1.1 Created by the Team

The Team writes Acceptance tests for every feature before committing to deliver the feature. The Product Owner is responsible for approving the test cases.

The team will not commit to any backlog item before translating the item into a number of Acceptance tests. The tests can be speciﬁed in an informal way, as long as both the team and the Product Owner agree that they are unambiguous and

75 CHAPTER 7. IMPLEMENTATION SUGGESTION comprehendible. Once the Product Owner has approved the test cases, the feature can be considered ’done’, when and only when all the test cases are passing.

Acceptance tests are a crucial part of the XP methodology, where they are created by the customer or someone close to the customer. These tests will be created by the Team, and it is up to the Product Owner to ensure that they meet the customer’s speciﬁcation. We think, however, that there are several advantages to this way of working. One advantage is that the team is forced to examine if enough information about the feature is provided to implement it, at an early state. Another is that there will be an unambiguous way to tell when a feature is ’done’, as a clearer picture of what the customer wants can be achieved [11]. A third advantage is that “...Acceptance tests serve as compileable and executable documentation of the features of the system.” [2, p.27] This is important when new developers join the team.

The Acceptance tests are meant to improve the problems relating to the customer relationship noted in section 4.3.1 and to improve the ability to plan and structure the testing, an area where there is room for improvement (see sections 4.3.4 and 6.2.3 for details). It also acts to create clear goals for the testing (see section 4.3.2 for details on this area).

7.1.2 Automated

The Acceptance tests should be automated and run continuously

This means that the business layers must be decoupled from the UI so that it is possible to gain access to those layers from the test case. As stated by the team during the solution proposal evaluation this was already the case in most of the applications (see section 6.2.2). This means that the developers will know when a feature is done. Not knowing was a problem mentioned by the developers. Furthermore the test cases cover the most important aspects of the applications, and gives a basic testing coverage of the application.

Tests should be run in developer sandbox environments whenever a feature functionality is deemed suﬃciently done for passing those tests. These tests should then be enabled on the Continuous Integration server to run continuously in order to verify that new functionality does not break older code. This provides an eﬃcient way to drastically reduce the time needed for regression testing. If test execution time is considered a problem (as mentioned during the solution proposal evaluation, see section 6.2.2) we suggest a second Continuos Integration environment dedicated for this purpose. Alternatively, divide the Acceptance tests into subsets and run in parallel on several machines.

76 7.2. SESSION BASED EXPLORATORY TESTING

7.1.3 Automated on Legacy applications

When working on existing code that currently lack the support for the test automation framework, the XP retroﬁtting practices as stated by Beck [5, p. 126] should be followed:

• When you need to add functionality to untested code, write tests for its current functionality ﬁrst.

• When you need to fix a defect, write a test first. • When you need to refactor, write a test first. As uncovered during the evaluation of the proposal regarding increased automation, it was perceived as a problem that when splitting functions and refactoring code it took a lot of time to update the tests (see section 6.2.2). We do not consider this to be a problem. As testing it manually each time the application is altered takes time also, the automated solution is more beneficial in the long term. Using the practices stated by Beck, the parts of the system visited often and the parts that attract attention and new features will be well tested automatically [5, p. 127].

7.2 Session based exploratory testing

Exploratory testing is the technique of choice for System and Integration testing at the section. It is our opinion that aspects of the test process should be more clearly deﬁned in regard to that technique.

7.2.1 Session rules

The team should strive to perform test sessions as early in the Sprint as possible. This means trying to integrate two or more features as early as possible, and performing a test session. This will lessen the stress in the latter part of the Sprint. Before the start of each instance of Exploratory testing in a Sprint, the Product Owner prioritizes the areas or features of the applications in terms of value and risk. This prioritization should then be used as a base for planning the focus of the testing. This is an implementation similar to the one used by Gerras [12] (see section 2.5.2). During a time-boxed session, we recommend doing pair testing. Beck states that one is more likely to overanalyze, make mistakes and disregard practices under pressure if performing the task solo [5, p. 67]. Despite that statement is stated in regards to developing code, we believe this applies to testing as well.

77 CHAPTER 7. IMPLEMENTATION SUGGESTION

After each time-boxed testing session, each pair justifies to the other members of the team what they have tested and why. The other team members then try to find something overlooked during that pair’s testing. This way of working complies with Burnstein’s basic principle that Test results should be inspected meticulously [7, p. 27ff]. Any additional testing needed is put into the next time-boxed session. During all testing results on how the testing is performed should be documented in order to eliminate redundant testing. All test strategies should be kept on a list for inspiration on how to test during other testing sessions. If a new strategy is discovered during a session, it should be added to the list. If any defects are found during the testing, they should be recorded along with information necessary to reproduce them. This complies with Burnstein’s principle that Tests must be repeatable and reusable [7, p. 27ff]. The defects recorded should then be prioritized in cooperation with the Product Owner in order of importance for fixing. Any defects not fixed (for instance due to a lack of time), should be noted and put in the product backlog. That way they are visible to the Product Owner and can be prioritized for future Sprints. As indicated during the exploratory case study, it was considered a problem that there was no one clearly responsible for the testing (see section 4.3.5). This intends to create control over the testing. It was also stated during the change proposal evaluation that the team found it hard to plan what to test according to the value for the customer (see section 6.2.3). Increased planning, structure and involvement of the Product Owner in the testing seeks to rectify this problem.

78 Part IV

Discussion and conclusions

Chapter 8

Discussion

During the cause of our project we evaluated the team’s opinions about their testing. We found that their impression is that the customers/users are satisﬁed with the quality of the applications. Likewise, the team feels that their testing process is functioning adequately, even if they are not satisﬁed with it. This is important as we feel that this decides the mandate we have for changing the test process. The general opinion that the process is working adequately makes us reluctant to change the process extensively. That served as the motivating factor for our implementation suggestion, which can be found in chapter 7. This suggestion is based on both feedback from the team on our change proposals (see chapter 5) and related work presented in chapter 2. It is not a radical departure from the process the team describes as their current goal. The exception was the testing aimed at ensuring that the functionality actually is what the user want. The team reported a need for improvement in this area, thus justifying a more radical change.

The depth of the improvement suggestions presented can be discussed. The focus of this thesis was on processes, roles and methods in regards to testing in an Agile environment. As this covers a wide area of subjects, the change proposals and implementation suggestion doesn’t provide any detailed solutions. Nor does it give any specific tools that can be used, or any details on how to perform the testing under certain specific conditions. During the course of the project we became aware that such questions existed. The questions how do you automate your test cases if the application is of design X and how should the Acceptance tests be written were some examples of this. Due to the nature of the focus of this thesis and the time constraints connected with the project, there was no possibility for us to go into any detail in these areas. Instead, we feel that this thesis could serve as a basis for future thesis works. We particularly got a lot of questions regarding Unit testing, on how to deal with some specific implementation situation, or how to minimize some problematic condition. This might be better adressed by setting

81 CHAPTER 8. DISCUSSION aside time within the team for training. As there are numerous books on the subject something like a study circle might be a good idea. Besides testing, the various tasks that lie within the team’s responsibilities along with the factors affecting their Agile process opens a number of questions. Is Scrum is the most suited way of working for this team? Currently buffer hours has to be added to their Sprint backlog in order to have room for support and maintenance issues. These are tasks that cannot be predicted before the Sprint starts. Not only does this not follow the rules on how the Sprint backlog is supposed to work, it generates an uncertainty in all estimations. This is also related to the operation responsibilities of the team. If any of the systems they are responsible for needs attention to be operational, this takes priority over the backlog items they have committed to. It is our opinion that this risks creating an unbearable situation in the long run, as the team can never truly commit to their work. This opens a window of opportunity in regards to future work. We would like to see more research in how the implementation of Kanban systems in software development environments are best used. Traditionally, Kanban systems stem from the manufacturing industri. Can these systems be adapted in a different way, and if so, how does an Agile testing process fit into that environment? How do you perform software testing in the most suited way in such an environment? We would also like to see more research in Agile test planning. The uprise and popularity in using guided exploratory testing methods opens the possibility for discussion regarding this subject. Is there a good way in maintaining a documented test plan while not sacrificing agility? Detailed test planning provides good documentation, but at the same time makes the process less lean. A possible research subject could be to formalize the control over test coverage. One suggestion might be to analyze and separate the software into components, features and capabilities. These could then serve as a basis for planning what testing to perform in each level, for each item.

82 Chapter 9

Conclusions

The main questions that this thesis was to try and ﬁnd an answer to were:

• What requirements are there on the test process at the section, and how should they be prioritized?

• How should the test process be constructed, in terms of methodology, processes and routines, in their Agile environment?

• How should competence and responsibility be distributed within a team that is working with Agile developing methods? Answers we obtained during the group interviews in the exploratory phase indicated that there existed almost no known requirements on the test process, neither from the customer nor within the section. The team stated that all new applications should have Unit tests. Besides that no expressed requirements existed. The only other requirements that existed were those each developer had individually. They wanted their own code to be suﬃciently tested in order to deliver working code. How much testing that was needed, as well as the thoroughness of the testing, was decided individually. Therefore no clear answer can be given in regard to how any requirements should be prioritized. In order to create customer involvement, as well as form a strategy on how to introduce proper Acceptance testing, we propose that Acceptance tests be written by the team based on the requirements in the Scrum Product backlog. These Acceptance tests should then be validated by the Product Owner, either directly or by communicating with the customer. It is also our opinion that all Acceptance tests should be automated. There was expressed concerns in regard to team resources as well as execution time for such automation. Given a situation where this wasn’t a problem, an increase in automation was seen as beneﬁcial. It is therefore our opinion that an automation of Acceptance tests should be performed. It would provide valuable information on the progress of the

83 CHAPTER 9. CONCLUSIONS development. Introducing this type of automation in applications currently lacking support for the test automation framework would also gradually reduce the regression test times. Currently manual testing is performed using exploratory testing. There is no reason to change this strategy. It should be used as a complement to the automated testing. Even if it is an Agile process, the testing must still have clear rules. It is important to structure all testing sessions. This includes planning, execution and evaluation of the result of the test. Planning involves setting goals for the testing session and deciding how to best achieve those goals. The different areas to test should be prioritized together with the Product Owner. This creates more customer involvement in the testing, and aims to get the most value out of the testing session. After each testing session, the most important task is to come to an informed and joint decision that the goal for that session has been met. This decision should take into account the defects found. It should also take into account what has been tested, and how it has been tested. While it is important to know what you have tested, it is equally so what you have not. The separation between the role of a tester and the role of a developer has traditionally been considered necessary to avoid knowledge contamination. Studies on the area have shown that even though this contamination exist, it is often more beneficial if the tester is a member of the development team. Agile development revolves around cross competence. Over time developers change the way they construct applications by developing with a tester mindset. This in turn lead to less errors being introduced into the code, which is to be compared to more errors found. Working in a more Agile way with testers on the team requires a change of mindset in order to not fall back into a Waterfall development flow. Both disciplines have to be involved from the start of each development iteration, and assist each other. Responsibility should be shared, although testers are better at translating customer data into tests. A successful integration takes advantage of the fact that developers are good at developing, and testers are good at testing. Still, make sure that developers test and that testers also develop in order to not fall into a Agile Waterfall trap.

84 Bibliography

[1] J. Highsmith. “History: The agile manifesto,” [online]. 2001. Available from: http://www.agilemanifesto.org/history.html [cited 2009-04-28].

[2] R. C. Martin, Agile Software Development, Principles, Patterns, and Practices (Alan Apt Series). Prentice Hall, 2002.

[3] K. Schwaber, Agile Project Management with Scrum. Microsoft Press, 2004.

[4] R. Jeﬀries, A. Anderson, and C. Hendrickson, Extreme programming installed. XP series, Boston, Mass. : Addison-Wesley, cop., 1st ed., 2001.

[5] K. Beck, Extreme programming explained. XP series, Boston, Mass. : Addison-Wesley, cop., 1st ed., 2000.

[6] G. N. Krieg, “Kanban-controlled manufacturing systems,” Lecture Notes in Economics and Mathematical Systems, vol. 549, 2005.

[7] I. Burnstein, Practical Software Testing. Springer Professional Computing - A process oriented approach, 2003.

[8] “Living glossary,” [online]. December 2006. Available from: http://www.testingstandards.co.uk/living_glossary.htm [cited 2009-03-02].

[9] “Ieee standard glossary of software engineering terminology,” vol. IEEE Std 610.12-199, September 1990. Available from: http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=00159342 [cited 2009-04-03, 15:05].

[10] C. Jakobsen and K. Johnson, “Mature agile with a twist of cmmi,” Agile 2008 Conference, pp. 212–217, 2008.

[11] C. Lisa, “Driving software quality: How test-driven development impacts software quality,” IEEE Software, vol. 23, no. 6, pp. 70–71, 2006.

85 BIBLIOGRAPHY

[12] A. Geras, “Leading manual test eﬀorts with agile methods,” Agile 2008 Conference, pp. 245–251, 2008.

[13] M. Puleio, “How not to do agile testing,” Agile Conference, 2006, pp. 305–314, 2006.

[14] B. Haugset and G. Hanssen, “Automated acceptance testing: A literature review and an industrial case study,” Agile 2008 Conference, pp. 27–38, 2008.

[15] “Manifesto for agile software development,” [online]. August 2008. Available from: http://agilemanifesto.org/ [cited 2009-04-01, 19:22].

[16] T. David, H. Orit, D. Yael, and K. Arie, “Agile software testing in a large-scale project,” IEEE Software, vol. 23, no. 4, p. 30, 2006.

[17] M. Sumrell, “From waterfall to agile - how does a qa team transition?,” AGILE 2007 (AGILE 2007), pp. 291–295, 2007.

[18] S. B. Merriam, Fallstudien som forskningsmetod. Studentlitteratur AB, 1st ed., 1994.

[19] M. Höst, B. Regnell, and P. Runeson, Att genomföra examensarbete. Studentlitteratur AB, 1st ed., 2006.

[20] S. Kvale, Den kvalitativa forskningsintervjun. Studentlitteratur AB, 1st ed., 1997.

[21] V. Wibeck, Fokusgrupper. Studentlitteratur AB, 1st ed., 2000.

[22] J. Sutherland, C. Ruseng Jakobsen, and K. Johnson, “Scrum and cmmi level 5: The magic potion for code warriors,” Proceedings of the 41st Annual Hawaii International Conference on System Sciences (HICSS 2008), pp. 466–466, 2008.

86 Part V

Appendices

Appendix A

Use Case example

Appendix B

Problem validation Likert questionaire

This questionnaire is intended to document the current process you’re using, and to validate the results of the group discussions. When answering the questions below it is important that you consider that your answers reﬂect the current situation you work in, and not take into account how it worked in the past or how it is intended to work. Answers are anonymous. Instructions: Please specify your level of agreement to the following statements.

91 APPENDIX B. PROBLEM VALIDATION LIKERT QUESTIONAIRE

The testing we’re doing is suﬃcient in relation to the need. Strongly disagree 2 2 2 2 2 Strongly agree

As a team, we are good at coordinating and structuring our testing. Strongly disagree 2 2 2 2 2 Strongly agree

We consider the question “Why did we not ﬁnd this in our testing” when a bug is found. Strongly disagree 2 2 2 2 2 Strongly agree

It’s a problem that we don’t have a clear picture of what’s required of our testing. Strongly disagree 2 2 2 2 2 Strongly agree

The customer is satisﬁed with getting a demonstration of the product instead of an Strongly disagree 2 2 2 2 2 Strongly agree

Acceptance test. Strongly disagree 2 2 2 2 2 Strongly agree

It’s a problem that tests are based on sample data rather than clear speciﬁcations. Strongly disagree 2 2 2 2 2 Strongly agree

Not having access to Acceptance tests at an early phase of a Sprint makes it harder to validate that the product meets the user’s needs and expectations. Strongly disagree 2 2 2 2 2 Strongly agree

Most of the times we can’t start testing our features until the end of the Sprint. Strongly disagree 2 2 2 2 2 Strongly agree

It’s a problem that besides Unit Testing, almost all of our testing is performed in the last days of the Sprint. Strongly disagree 2 2 2 2 2 Strongly agree

We often cut back on testing when we are pressed for time. Strongly disagree 2 2 2 2 2 Strongly agree

92 It’s a problem that there is no one that is clearly responsible for our testing. Strongly disagree 2 2 2 2 2 Strongly agree

We seldom get feedback on our products unless critical errors occur. Strongly disagree 2 2 2 2 2 Strongly agree

It’s a problem that we don’t consider risks when deciding how to test. Strongly disagree 2 2 2 2 2 Strongly agree

There is no need for the customer to write Acceptance tests since their requirements are suﬃcient to decide when a feature is done. Strongly disagree 2 2 2 2 2 Strongly agree

We’re not testing our products as much as we need. Strongly disagree 2 2 2 2 2 Strongly agree

We generally estimate too little time for testing. Strongly disagree 2 2 2 2 2 Strongly agree

When we are done with a System test, I can make conﬁdent statements on how well each feature has been tested. Strongly disagree 2 2 2 2 2 Strongly agree

It’s a problem that we dont have any test experts on our team. Strongly disagree 2 2 2 2 2 Strongly agree

Appendix C

Likert validation results

95 APPENDIX C. LIKERT VALIDATION RESULTS

96 97 APPENDIX C. LIKERT VALIDATION RESULTS

98 99 APPENDIX C. LIKERT VALIDATION RESULTS

100 101