Quality in Agile Teams - Testing in the Driver’S Seat an Insight Into Quality in Agile Teams

Quality in agile teams - testing in the driver’s seat An insight into quality in agile teams

October 2017

KPMG.com/in 01 Agile adoption – Introduction 1.1 Drivers for agile adoption 1.2 Agile methods and practices 1.3 Barriers to agile adoption 1.4 Fully agile or just fast waterfall 1.5 Why agile projects fail page 01-08

02 Agile – an antipattern to testing?

2.1 Agile an antipattern to testing 2.2 Deconstructing testing in agile 2.3 Testing maturity in agile teams page 09-11

03 Quality in agile teams

3.1 Improvising quality in agile teams 3.2 Measuring quality success in agile 3.3 Automation tooling 3.4 Management tools page13-16

04 Key takeaways

page17

05 Summary

page18

© 2017 KPMG, an Indian Registered Partnership and a member firm of the KPMG network of independent member firms affiliated with KPMG International Cooperative (“KPMG International”), a Swiss entity. All rights reserved Executive summary

As digital transformation continues to play a critical role in the IT landscape, the software development lifecycle too is transforming

While the agile adoption rates as cloud, analytics, the Internet of doubled over the past five years have been steadily climbing, there Things (IoT) and mobility, testing while an improvement in quality is a has been a paradigm shift in the has now evolved into business major driver for agile adoption. focus of agile teams. With digital assurance rather than an end-of- initiatives driving technologies such cycle activity. Agile adoption has

Automation leads the way for agility

Testing is now at the forefront of the development cycle adding the most business value to organisations. More agile teams are adopting test-driven development and acceptance test- driven approaches. As organisations are embracing agile, there has been an 85 per cent01 increase in test % % automation over a two-year period 2013 2017 across all industry domains.

01. KPMG in India’s research and analysis 2017 based on KPMG testing survery

1.1 Agile adoption – Introduction

24% 7% 2% 16% 51%

Hybrid Leaning towards Pure Pure agile Leaning towards waterfall waterfall agile

Source: Tech Beacon survey - Is agile the new norm? Accessed on the 14 September 2017 Agile adoption percentages

Software development landscape

Today’s software development Adopters of agile strongly believe Lately, adoption of agile lifecycle is more outcome oriented that this new development development methodologies when compared with traditional methodology is centred on is seen as a remedy for most development lifecycles such the customer and helps teams problems that plagues the as the waterfall development collaborate better to deliver development groups in an methodology. The pace of change higher business value.Though the organisation, but where does today is so unrelenting that agility adoption of agile has been fairly testing and quality fit into the has become completely critical in slow, it has gained momentum in agile conundrum? Will the puzzle order to respond to the customer’s the past five years. make a complete picture with the demands and rapidly changing current focus on development With the new values, practices market landscape. and engineering? There has been and benefits - agile is seen as a paradigm shift in the focus of The idea of an approach where the the unconventional alternative accepting and adopting agile across designs are made upfront, even to the traditional command-and- the organisation - that shift from before any code is written is seen control-style management. Agile engineering and development to as risky in today’s times as the methodologies are spreading testing and quality outcome. With feedback is slow and does not across a broad range of domains approaches such as test driven allow for changes quickly. Lately, and functions and even into the development and Acceptance Test adoption of agile development C-suite from broadcasting to Driven Development, organisations methodologies is seen as a remedy leading agriculture machines are now accepting that testing for most problems that besets developer to producing fighter jets should be in the driver’s seat rather the development groups in an to logistics providers to wineries than just the backseat. organisation. for their warehousing to senior management. Agile software development originated 15 years ago and has Adopting agile methodologies is not been seen a worthy opponent only seen as a means to accelerate to the ailing traditional software revenue growth but also help development cycles. The agile create a new breed of competent methodology was recognised in professionals by taking people 2001 at a summit of practitioners out of their functional silos and who found consensus around core putting them in self-organised and values captured in 12 principles customer-centric multidisciplinary called the ‘Agile Manifesto’. teams.

1.2 Drivers for agile adoption

Key reasons for adopting agile

Accelerate Ability to Increase Enhance Improve time to manage productivity software business/IT market changing quality alignment priorities

100% 90% Accelarate time to 80% market

70% Enhance ability to 60% manage changing priorities 50% Increase 40% productivity 30% Enhance software 20% quality 10% Improve business/ 0% IT alignment 2013 2014 2015 2016 2017

Source: State of agile survey, 2013,2014,2015,2016, 2017, Version One sccessed on 14 September 2017

Drivers for agile adoption Why have so many organisations corporate vision; prioritise strategic With the implementation of agile moved to agile? What were the initiatives; simplify and focus methodologies, organisations feel factors that pushed organisations work; assigning the right people that there is more freedom to to adopt agile methodologies? to tasks; all while enhancing team adapt to the changing requirements collaboration and helping teams as compared to the waterfall The top five reasons for raise the bar on quality consistently. development practices where the organisations to adopt agile user requirements were frozen five years in a row has been to Even as organisations are looking to early on and the customer may accelerate time to market, ability embrace agile for various reasons, not have access to quick feedback to manage shifting priorities, most still feel the learning curve loops. With the adoption of agile, increasing efficiency and to is steep and consider that their teams within the organisation also improve software quality. Most organisation was at or below a ’still feel empowered to deliver business responses were centred on better maturing’ level. As organisations value in relatively short amounts of customer focus, enhanced quality ramp up their game, responding time. and increasing predictability across to customer expectations and the delivery lifecycle. The other responding to increasing market major benefit with adopting agile is demand are the major drivers for widely seen as the ability to deliver change. higher value work by helping create

1.3 Agile 2.0

Ideation Engineering Development Quality Purpose Business Value

Ideation Engineering Development Quality Business Purpose Value

With the wide acceptance and able to respond to change. As agile matures, the focus on adoption of agile methodologies quality has been reinforcing the Factors such as faster time-to- across the broad spectrum of importance of the role of the tester market, better software quality industries, the focus of agile has and its benefical impact on the agile and customer satisfaction and also slowly evolved over the past team. not productivity gains in terms of five years. The reasons for adopting improved engineering discipline, agile initially were was faster were seen as the leading benefits time-to-market and to increase associated with the methodology. productivity amongst teams. The Over the past five years the focus reasons for adopting agile has has changed from engineering now pushed the focus back on practices to testing and quality. quality and customer satisfaction. The primary motivators for agile Initially, agile was focussed on unit adoption continue to be -faster testing and acceptance testing, time-to-market, increasing software while the functional testing did not quality and customer satisfaction. appear to play an important role The centre of the viable, profitable and was assumed that it could be cosmos is no longer the completed by the developer it self. organisation: it is the customer. There was to be a cultural shift for Agile, therefore, has a massive the traditional system tester to be effect has to how an organisation part of the agile team. is run and how the organisation is

1.4 Agile methods and practices

11% 9% 8% 6% 66%

Scrum/ Custom/Tailored Scrumban Kanban Scrum XP hybrid hybrid

Key agile methods and practices

70%

60%

50%

40%

30%

20%

10%

0% 2013 2014 2015 2016 2017

Scrum Scrum/XP hybrid Custom/Tailored hybrid Scrumban Kanban XP FDD Lean Others

Source: State of agile survey, 2013,2014,2015,2016, 2017, Version One sccessed on 14 September 2017

Scrum and Scrum/XP Hybrid (68 Though Scrum, Kanban and XP they want to engineer solutions, per cent) continue to be the most popular agile methodologies, all helps them engage productively common agile methodologies used three methodologies follow the with the customer through frequent by respondents’ organisations. principles of the Agile Manifesto communications. Kanban could Adopting scrum in combination that aims to provide maximum be appropriate for teams dealing with the desirable features of other business value to customers in with continuously fluctuating agile processes is seen as more shorter iterations. The differences backlog of items. Kanban helps beneficial among agile practitioners. between them are due to the with limiting the intake of backlog methodologies trying to uphold the items while ensuring efficeiency. While the usage of XP as an agile principles in fundamentally XP adds another level of intricacy, independent methodology dissimilar viewpoints. bringing a strong focus on quality continues to decrease (<1 per cent), by maintaining a set of core the practices associated with XP Scrum is seen to be apt for teams engineering practices which aims to are still prevalent. who can dedicate their shared keep the code clean and software time to a project or product. Scrum stable. allows teams to decide on how

1.4 Barriers to further agile adoption

Unable to change General Trying to fit agile Availability of Management organisation resistance to into non-agile skilled personnel support culture change framework

Key barriers to further agile adoption

70%

60% Ability to change organisation culture

50% General resistance to change 40% Trying to fit agile 30% elements into non- agile framework 20% Availibility of skilled personnel 10% Management 0% support 2013 2014 2015 2016 2017

Source: State of agile survey, 2013,2014,2015,2016, 2017, Version One sccessed on 14 September 2017

Barriers for agile adoption

One of the biggest factors that is Though the main priority for organised agile teams is also seen inhibiting organisations to adopt business leaders continues to be as a factor inhibiting agile adoption. agile fully is the inability of the agility, they are unclear on their Finally, the lack of management organisation to accept and adopt organisation’s goals and purpose. support to usher in new changes change - the organisation’s DNA Business leaders need to prioritise – in terms of structural changes, does not allow change and there is the most important business cultural changes, etc. leads to the a general resistance to change. goals rather than focus on multiple failure of agile adoption across the initiatives, as executives may organisation. Key barriers to adopting agile launch multiple initiatives with methods and practices are: urgent deadlines where nothing 1. Ability to change the actually gets done. Albeit with organisation’s culture their best of intentions, they erode 2. General resistance to change benefits that agile innovation can 3. Trying to fit agile elements into deliver, all while falling back into the non-agile framework. vortex of old practices. 4. Availability of skilled talent The unavailability of skilled 5. Management support. personnel to empower self-

1.5 Fully agile or just fast waterfall

79% 21%

Fast Agile Waterfall practices

Source: State of testing survey 2017, accessed on the 15 September 2017 Fully agile or Fast waterfall

Fully agile or fast waterfall?

The broad adoption of agile has not 1. Writing tests whenever code is cent) of organisations that use the solved the pressures for constantly checked in agile software have achieved this faster releases with better quality. 2. Frequency of releases micro-iterative approach. But for Organisations still want to deploy 3. Speed of defect fixing most agile teams (59 per cent) it even faster with better quality. is still the standard for individual 4. Automation percentage. The timeline for deployments and features to be completed before releases has only gotten smaller The most important best practice being passed on to the Quality from months to weeks to daily to for testing in agile is that code Assurance (QA) team, which hours. is tested as soon as it is written. essentially leads them back to the the traditional waterfall way of Though 79 per cent02 of the The goal for a fully agile team is testing. When the tests are not respondents are said to have to achieve small iterations, where written whenever code is checked adopted agile in some form features are broken down into small in, rather when it is written after but only 21 per cent consider testable requirements to speed completion and passed on to the themselves fully agile. The factors the identification and fixing of bugs QA team it is just fast waterfall and that encourage teams to consider while the code is still fresh in the not truly agile. themselves fully agile are: developer’s mind. Unfortunately, only a very small number (24 per

02. State of testing survey 2017, accessed on the 15 September 2017

1.6 Agile projects failure

Census fail

• The 10 million dollar03 the router, and there was a • An audit across its 400-strong contract was awarded by backup – but in configuring technology staff found it the Government to conduct the backup, vendor hadn’t needed to train more scrum the online census, but the conducted a full power-down- masters, product owners and combination of inadequate non- reload test. team members - the failure was functional testing (performance largely due to a lack of skilled • The vendor and its network and security) took the systems personnel and the lack of focus providers had to block traffic that offline for two days on non-functional testing 04. didn’t originate in the country - • The heavy traffic load crashed but that did not happen

Healthcare release

• Healthcare website is • While the front end employed • The project involved 60 different an example of software Scrum, it missed the second organisations, managed by development gone wrong. The principle of the Agile Manifesto: government employees instead original budget was USD93.7 ‘Working software’ of a lead contractor million, which grew to USD292 • Regulations pertaining to large • Lack of a clear test strategy and million prior to the launch of government contracts, are not a lack of a state-wise release the website. The final cost of a good match for agile software plan. building the website was close development to USD2 billion05.

Siren

• Another more recent example of ways of agile development level along with the the gap a U.K. government agile project along the governmental project in communication between failure for a crime, intelligence management and internal external and internal teams and and custody suite. The official controls a lack of transparency when report blamed the inexperience considering the timelines of the • This was seen as a failure of the in-house team in the project (eight years).06 of agile adoption at every

Welfare reform project

• High profile failures such as • In the case of the welfare reform • Secondly, a failure of the U.K. government’s welfare project, the National Audit procurement and contracting, reform project – were examples review of the project clearly reinforcing the hearsay and of high profile and costly indicated that the failure of the rumours. One of the major (estimated final cost GBP2.4 project was an agile adoption problems noticed was a lack of billion07) project failures failure with work culture failures transparency and a failure of mentioned performance reporting.08

03. Australian Bureau of Statistics, accessed on 15 05. Cio.com accessed on 19 September 2017 08. computerweekly accessed on 19 September 2017 September 2017 06. BBC accessed on 15 September 2017 04. ABC Australia, accessed on 15 September 2017 07. IT governance UK accessed on 15 September 2017

As we look at agile adoption, its a backseat. With coding and short end users, lack of programme approaches and the barriers to release cycles, quality suffered and management for large scale adopting agile; the recurring factor retaining the customer base was projects and lack of testing that was driving organisations the elephant in the room. Agile as demonstrated in the above towards being agile was responding teams now had to grapple with instances. While trying to solve to customer expectations to deliver developer testing and their take on one of the problems of unyielding business value, faster release quality rather than an independent requirements, these projects cycles and quality. voice on quality. Projects would created several other problems fail regularly due to lack of a long leading to complete project failure. Earlier, the focus was hugely on term strategy and plan, lack of development, engineering and requirements management across delivering quickly and testing took iterations, technically unaware 1.7 Why agile projects fail

Gartner’s research explores why agile projects, causes of project failure across three project sizes. The major takeaway for test professionals is for 72% 28% them to analyse the balances required when delivering projects in shorter iterations with greater quality and all the agreed functionality. Success Failure The leading cause for failure of agile projects across all project sizes is the Distribution of success and failure across issue around functionality – especially Source: Gartner-Why projects fail accessed on 15 September 2017 when the teams are unable to deliver the business value promised, leading to either expensive rework or Why agile projects fail? cancellation of the work orders. Lack of quality comes a close second 30% for the failure of agile projects across 25% the different project sizes. As seen in the above instances of project failures, 20% a lack of proper testing can lead to epic failures. Despite the teams 15% trying to deliver in shorter iterations, the teams did not spend enough 10% time planning for the testing types and phases. Due to a lack of proper 05% testing, the quality of the product application was affected and the 0% damage caused was irreparable. Small sized project Medium sized project Large sized project It is not entirely unexpected to see that the challenges delivering projects Functionality issues Sustainability late Poor quality on time, within budget and with the High cost variance Cancelled after launch Others agreed functionality with crunched testing efforts are mentioned by

Source: Gartner-Why projects fail accessed on 15 September 2017 two-thirds of the respondents as the causes of project failure, in line with our previous observations.

2.1 Agile - An antipattern to testing

Percentage of Testing

Size of test team

Business Value

Note : Indicative illustration Timeline

SDLC Incremental Models Spiral Models Agile Models

Lifecycle

Relevance of testing In agile methodologies, the lack constantly changing environment? and the amount of testing did not of well represented, detailed increase over time or add business Until recently, testing was seen requirements, short iterations value. The effort expended by as a factor that affected the agility with quick delivery bring to light the test teams did not yield the of teams while the focus was on significant questions on the business value when compared the development and engineering. traditional view of testing. How can with the effort invested. The role of a tester and relevance end-to-end testing be carried out was questioned as testing was With the advent of agile, the when the complete application or presumed to be a hindrance in the business value being delivered product is not yet available? What agile environment. has seen a more focussed levels of documentation would approach. With the adoption of suffice? Is there a possibility of consistently agile, development teams have raising the bar on quality while What phases and levels of testing shown that automation is the way, reaping the advantages of agile – does one eliminate and what does thereby, reducing the manual effort agile methodologies stress on one retain in the agile delivery and the number of testers or QA the need for speed, consistent lifecycle? How does one measure in the team. Though the number constant customer engagement the effectiveness and efficiency of of testers in agile teams are fewer and acceptance of change. automation efforts? In all of this, in number, the amount of testing the non-functional tests such as The emphasis has now shifted from increased sprint on sprint. This performance testing, load testing, development and engineering to is made possible by agile teams security testing, usability testing, testing. The evolution of testing adopting technology such as reliability testing, scalability, etc. has progressed from being tester- Robotic Process Automation (RPA), get pushed till the very end. With centric (manual testing) to effective Machine Learning (ML), Artificial developers donning the hats testing that adds business value Intelligence (AI) and cognitive of testers as well, the role of a (automation). With the traditional analysis. professional tester is also a bit of waterfall development lifecycles, a riddle. What value can the tester though the number of testers were add to this highly unstructured and more, the effectiveness of testing

2.2 Deconstructing testing in agile teams

There are many myths that shroud not only around agile practices, but the mindsets of organisations also around testing and the need for looking at adopting agile testing in agile teams. methodologies. These myths are

Testing in agile is unstructured, Agile does not have defined Testing in agile is ad-hoc Myths undocumented and quick strategies

Though different from planning Agile manifesto states “Working Testing synchronized with the in waterfall, testing planning in software over comprehensive dev cycles, agile aims for quick Reality agile is an inherent part of sprint documentation “and suggests feedback loops with continuous planning and test cycles are in- to only document that what is testing and frequent testing sync with the dev cycles needed and has clear added value to document

Only unit tests are sufficient- Unit tests eliminate the need for Test Driven Development(TDD) Agile practices can be “tailored” Myths manual testing is enough

Test Driven Development (TDD) Tailoring could be done to While Test Driven Development is good, but one has verify if get around an organizational may reduce the manual testing Reality the code written meets the dysfunction, while ensuring that efforts, it will not completely specified requirements and units the Agile Manifesto is preserved. remove the need for manual are insufficient to cover this Quasi-agile does not reap the testing benefits intended

Developers have adequate Unit tests form 100% of your testing skills, agile does not Myths design specification need testers

The focus for the developer Unlike the V-model, the and tester is different- An requirements may not be frozen Reality independent testing team serves in agile , but requirements are as a check point for feedback refined though collaboration for on quality, to validate the robust specification functionality

2.3 Testing maturity in agile teams

Self organising Quality goals Automation Governance Measure

• Development • Quality goals • Quality pyramid- • Actively practice • Test efficiency and test work embedded within based strategy acceptance as partners sprint goals test driven • Test Coverage • Continuous to deliver development • Refactoring to applications • Balance functional integration and and non-functional continuous • Strict definition of reduce technical • Development tests deployment done (DoD) debt and test teams communicate in • Bugs are fixed real-time. immediately.

To better understand why releases Self organising teams: be covered by automated user were slower than expected for Development and QA teams need interface tests. This can help teams organisations even though they had to work as partners to deliver the truly achieve agility. embraced agile, one needs to look promised business value to the Governance: Self organising at the testing and QA practices that customer. With self organising teams may be able to govern the organisations followed. While teams, the communication should themselves when the team sizes there are many approaches taken to be real time and impediments need are as prescribed, but as the teams agile development, as a general rule to be addressed immediately. scale up, governance becomes key there are certain key best practices Quality goals: Quality goals need in delivering large projects. Teams for testing in a fully agile process. to be locked down with the sprint may need to strictly adhere to the As agile adoption gains momentum, goals. The quality goals should definition of ‘ready’ and definition organisations could succeed if they not decrease despite changes to of ‘done’ as the teams scale. For understand the different levels the customer requirements. Agile large implementation, governance of agile maturity. Product and teams need to set the test strategy should be able to provide the development managers need to and continue the test planning as teams with the bigger picture and evaluate where their organisation part of the sprint planning activities. the roadmap to achieve maximum stands on the maturity scale, and Quick time-to-market can only be business value all the while being what would suit them best. achieved with quick release cycles nimble and adapting to changing An agile maturity framework is – quick release can be possible customer requirements. undoubtedly needed to help make only when the defects are fixed Measure and adapt: As the the case for acceptance, adoption, immediately and the defect ageing three pillars of Scrum (Inspection, process improvement and is controlled and monitored. Adaption and Transparency) benchmarking. Test automation: Test automation suggest, it is important to maintain As part of an agile readiness plays a crucial part in the agility of transparency, to consistently assessment, IT development teams and their ability to respond inspect and adapt to the needs of organisations should first assess quickly to changes. With the the customer. To achieve the quality their as-is development landscape automation scripts being written levels set, agile teams have to at the technical, programme and in tandem with the code, the monitor progress and quickly take project and organisational levels. effectiveness of automation tests corrective actions. Metrics play a Process and practices should be are higher. With the quality pyramid crucial part in the process achieving assessed for their effectiveness as reference, the automated unit the expected quality. Teams need and ease of adoption. tests percentage should be the to monitor the metrics such as highest approximately 70-75 per percentage of automation achieved, The five pillars of achieving test cent, while automated API tests, automation yield, test efficiency maturity in the agile development integration tests and component and test coverage, which help methodology are: tests should comprise of 20 per ensure that the quality goals are not cent, the rest of the tests should hampered.

3.1 Quality in agile teams

Effective agile adoption Plan With process coaches to help the teams align to the chosen agile methodology, agile adoption becomes easier and also reduces the risk of adoption failures Transparency in Maintain transparency reporting through out the lifecycle of the project with open communication and accurate and timely reporting of progress Objectives aligned to bigger picture With agile teams working in silos and no coherent plan for the project release, teams lose focus on the bigger picture. The project plan should cover the broader objectives Requirement For large scale projects, the decomposition agile teams should be mindful of refactoring and should chalk out the risks for the level of refactoring

Business assurance and common Work with the business ownership stakeholders to develop common strategy for agile testing to ensure time to market and quality. Promote exploratory Collaborative testing and business reviews early and frequently Collaboratively work with framework for development teams on development and business prioritisation, team- testing owned deliverables. Use Test Driven Development as tool and promote ‘defect prevention’ culture in team Continuous and early testing Execute unit, acceptance, regression and non-functional tests continuously to identify defects early. Manage interdependencies with other projects. Reusable easily Easy-to-use functional and maintainable non-functional automation automation framework depending the application and technology portfolio

3.2 Measuring quality success in agile

Measuring agile success As organisations have embraced agile, it’s natural for return on investment to 70% be foremost on the agenda. But how 60% can one measure whether the switch to agile is bringing about the desired 50% results? Quality metrics should be able to 40% give agile teams a competitive edge - quicker time-to-market, enhanced 30% customer experience, risk mitigation 20% and more rather than just numbers in terms of cost savings. 10% Most organisations have a set of goals that they want to achieve 0% through this transition – such as faster 2015 2016 2017 implementation, increased efficiency On-time delivery Product quality Customer/User satisfaction and mostly making customers happy by raising the quality of a project, Business value Product scope Source: State of agile report product, or service. Business value was cited as the second most popular Agile metrics for success measure (46 per cent) of an agile initiative’s success, rising from fourth 80% in the prior year. On the contrary, it ranked as the eleventh most popular 70% measure (23 per cent) on a day-to-day basis. This highlights an important 60% conflict between how more strategic areas of the enterprise are measuring 50% success and how success is being 40% measured at the team level. Business value as a measure of an 30% agile initiative’s success went from fourth in 2015 to second in 2016. 20% On-time delivery of projects and customer/user satisfaction remained 10% in the top three measures of agile 0% initiative’s success as they have in the past few years. 2015 2016 2017

Scrum Scrum/XP hybrid Custom/Tailored hybrid Scrumban 09. State of agile survey, 2013,2014,2015,2016, 2017, Version One sccessed on 14 September 2017 Kanban XP FDD Source: State of agile report

While business value was cited as number one measure of an agile delivering value and what factors the second most popular measure project’s success. need improvement. As with the (46 per cent) of an agile initiative’s pillars of Scrum, teams can get a While the success of the transition success, it ranked as the eleventh clear indication what areas need cannot be measured with a single most popular measure (23 per cent) to be further inspected, and adapt parameter, organisations will of an agile project’s success. Velocity quickly by introducing, removing or get clarity on whether they are (67 per cent) continues to be the amending practices.

3.3 Automation tooling

Leading automation tools Test automation in agile in agile Over time, agile test tools have • Static analysis tools: Static 80% become increasingly important analysis tools for code review 70% to the performance of agile against coding standards are 70% 65% teams. The use of the right used to find defects and are tools not only helps agile teams a more feasible option than 60% improve their performance but doing manual code review. 52% also increase their efficiency. 50% • Functional testing: Earlier, Automation is the need of the functional automation testing 40% hour and is used to eliminate was restricted to regression the mechanical, repetitive tasks testing which is aimed at 30% and bring in quicker feedback reducing the cycle time for loops and faster release cycles - regression. With agile teams, 20% essential in agile. With the fast testing starts early on in the 12% 13% pace of delivery in agile, test development lifecycle with the 10% data management and test test design being the priority 0% environment management needs while test execution should to be efficient with no room be automated. for time consuming manual CI is used for build, unit and With the adoption of test efforts. Organisations across functional tests automation, organisations could all industries are expected to reap the following benefits: Test cases automated via frameworks achieve more with less - less Open source tools like Selenium and time, less resources and 1. Reduced time-to-market Appium less budget. However, many – one of the biggest reasons organisations still continue to do for organisations to adopt Licenced/Propriety like HP UFT manual testing. agile methodologies Manual Testing only Activities that can be automated 2. Improved visibility – Test across the development lifecycle automation along with Source: State of agile survey, 2013,2014,2015,2016, 2017, Version One accessed on 14 September 2017 spans code reviews, build and analytics allows stakeholders release management, unit to fully understand the testing and functional testing. progress and status of testing

• Build and release: Nightly 3. Identifying defects earlier builds are the norm in agile – to ensure that defects are teams and there is a need identified and fixed early on, for zero manual intervention smart automation is effective in this process. This in identifying patterns requires good configuration 4. More testing – with test management and build tools. automation, organisations can • Unit testing: These are part test more in less time. More of the nightly build, thereby test cycles can be covered in creating a quick feedback less time. loop for the development team. The execution of these unit tests require manual interference.

3.4 Agile lifecycle management tools

Use of agile management tools

80%

70%

60%

50%

40%

30%

20%

10%

0% 2013 2014 2015 2016 2017

Excel Microsoft Project JIRA VersionOne HP ALM Microsoft TFS Bugzilla In-house Googles Docs Rational CA agile Vision/Central

Source: State of agile survey, 2013,2014,2015,2016, 2017, Version One accessed on 14 September 2017

Use of agile Agile management tool management tools landscape

$1.10 With fierce global competitive ALM tools are Microsoft Project, pressures and ongoing economic Atlassian JIRA and VersionOne. volatility, organisations are Agile ALM tools are now not just $800 reinvesting in IT projects. There is a ‘nice to have’ capabilities but are steady increase in agile Application core to business execution. Lifecycle Management (ALM) $572 Agile ALM is driven by the need for investment. Squick analysis into consistent IT- business alignment, $405 the worldwide agile ALM market. corporate governance, compliance The survey shows strong growth $291 and regulatory requirements. Agile patterns for agile ALM software vendors have been evolving their for the 2014–2019 time frame, with offerings across the three core Revenue in millions growth to around USD1.8 billion by functional markets that feed this 2019 and a high CAGR of 32 per competitive market — software 2013 2014 2015 2016 2017 cent (on small initial numbers)10. The change, configuration, and process complexity of deployment/DevOps Source: IDC Worldwide agile Application Life-Cycle management (SCCPM); project and Management Software Market Shares, May 2014 environs with mobile, social, accessed on 14 September 2017 portfolio management (PPM); and cloud, and big data and analytics automated software quality (ASQ) is a business inevitability. Complex with agile testing. sourcing needs of IT projects also stress on the need for agile ALM automation for successful agile 10. IDC Worldwide agile application life-cycle management software market shares, May 2014 adoption. Commonly used agile accessed on 14 September 2017

4.1 Key takeaways

Iteration Agile development plan Retrospective Agile testing

Strict Definition Test story Facilitate Automate test Plan for non- of Done (DoD) preparation developer for scripts functional testing early automation

Feature acceptance Monitor test Activities Story task break-down Consolidate and log feedback test cases and early progress and automation status

Testing within iteration

• Unit testing and Integration testing • Performance testing/Security esting • Functional and E2E testing • Regulatory/Compliance testing • Regression testing for previous iterations • Final UAT & Final Regression testing

Test Types Test • Feature acceptance testing • Exploratory testing

Agile testing activities are asynchronous in flow yet harmonious with development activities and business expectations.

The key takeaway here would be:

All testing activities within Teams should have a All test/QA activities User stories should be the iteration/sprint should robust definition of ready should be well planned broken down to tasks as be well chalked out and and definition of ‘done’ and chalked out. Changing small as possible. All tasks planned for. The iteration/ for all activities and tasks customer need should that do not have enough sprint plans should include across the iteration be tracked in the lifecycle clarity should be pushed the testing types and test management tool back for more refinement cycles as well for transparency and until the team is clear on 01 02 efficiency 03 the expectations 04

Early automation should be Working with cross- Non-functional testing Test progress and status facilitated in terms of unit functional teams including performance should be tracked tests early on. Automation helps ensure room for testing, security testing consistently with agile as early as possible should refinement and quick and cross browser ALM tools for transparency be the way forward. Teams adaptation. All Scrum/ compatibility tests should and to enable teams to can adopt a test driven or iteration activities should not be pushed to the far take quick corrective acceptance test driven be carried out to reap all end of the development actions. approach to achieve benefits. lifecycle, but should be maximum quality and considered and planned deliver business value to for early on. the customer consistent 05 06 07 08

A test-driven approach has Automated unit tests test-first teams find that documenting only what repeatedly helped achieve precisely determine the they get significantly is needed, unit tests can quality across the lifecycle. behaviour of the system, less defects throughout also be used as design Automated unit tests help With test-first teams, the the system life cycle and documentation that is in– weed out defects earlier percentage of defects debugging time is reduced. sync with the production in the lifecycle, leading found later in the lifecycle With the Agile Manifesto code. to quick feedback loops. is much less . Good emphasising the need for

5.1 Summary

Enable agile readiness • Agile coach and Process coach to mentor teams

• Structured approach to enable agile readiness from people, process & tools perspective • Standardized agile training to all team members Shift left • Quality Pyramid-based Strategy • Ensure test strategy and quality goals are set in line with business goals Test driven approach • Adoption of techniques such as Risk Based Testing\OA based test design for enhanced test coverage and optimized test cases • Ensure acceptance criteria and Definition of Done is aggressive followed Enhanced test automation • End to end lifecycle focused automation v/s focus on regression Automation • Functional Test Automation – Build ahead automation • Test design automation – Model Based Testing • Test Data Management

Project management • Agile Estimation models for predictable demand and capacity management • Platform/Application aligned Regression teams • Focused reporting on status and progress to maintain transparency through all phases of the project

Adopting agile has numerous Organisations looking at transforming helping ensure higher productivity benefits as compared to the their development methods first and efficiency. Though agilists may traditional development methods need to evaluate and assess their not completely agree with the need such as waterfall or the V-model. current landscape and the challenges for project management in agile Adopting agile brings about a visible they face in delivering business teams, there is a pressing need for change in the productivity, efficiency value. They also need to be mindful teams to be able to see the bigger and increases team happiness. of the organisation’s culture and its picture. While the agile development With time-boxed meetings, less acceptance of change. Consistent teams focus on delivering the best documentation, agile reduces the and meaningful engagement with possible solution to the customer, time wasted in redundant and teams through agile workshops, project management in terms of repetitive tasks and helps the team trainings and coaches would help capacity management, status and deliver high-value features. This organisations build the confidence progress reporting to maintain helps ensure faster time-to-market, and trust in the journey of successful transparency are much needed. increased predictability and reduced agile adoption. Readiness in terms of Today, for testers to be able to add risks. With self organising teams people, process/practices and tools value to the agile teams, they not and transparency within teams, could help achieve success. only have to be technically sound agile helps broaden organisational As seen before, agile adoption is but their focus needs to shift experience. With less time spent driven by the need for quality delivery. towards business value and meeting on micromanaging multiple projects, Organisations must be cognisant of customer expectations. Only finding agile helps the organisation prioritise the test approach and test strategy flaws may not suffice but the tester their strategic goals. that they implement. With the needs to understand if the output However, we do not believe that adoption of models such as risk- being delivered is complete and companies should adopt agile based testing, OA for optimisation, acceptable to the customer. For the methods in all places. In many agile teams can achieve greater test testing to be truly agile in today’s functional areas, such as plant coverage. Acceptance test-driven world, we need to understand the maintenance, purchasing, sales development can be key to achieving business context while helping calls, or accounting, more traditional greater business value and enhanced ensure that testing is always in the structures and processes are likely to customer satisfaction. End-to-end driver’s seat. deliver lower cost, more repeatable automation in the delivery lifecycle outcomes and more scalable could help agile team’s deliver better organisations. quality in short timelines while

This publication has been prepared by a team comprising: • Preethi Kudur • Nisha Fernandes • Viswanath Ramachandran • Darshini Shah

KPMG in India contacts:

Mritunjay Kapur Partner and National Head Strategy and Markets Leader - Technology, Media and Telecom T: +91 124 307 4797 E: [email protected]

Akhilesh Tuteja Partner and Head IT Advisory T: +91 124 307 4800 E: [email protected]

Prasanth Shanthakumaran Partner Business Excellence Tech T: +91 80 3980 6000 E: [email protected]

Rajesh Kannan Director Business Excellence Tech T: +91 80 3065 4677 E: [email protected]

KPMG.com/in

The information contained herein is of a general nature and is not intended to address the circumstances of any particular individual or entity. Although we endeavour to provide accurate and timely information, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future. No one should act on such information without appropriate professional advice after a thorough examination of the particular situation.

The KPMG name and logo are registered trademarks or trademarks of KPMG International.

This document is meant for e-communication only.