Continuous Testing: Transforming Testing for Agile and DevOps Table of Contents Introduction 3 Continuous Testing vs Test Automation 4 Traditional Testing Doesn’t Work 4 What Is Continuous Testing? 5 Comparing Continuous Testing with Test Automation 6 Risk 6 Breadth 7 Time 7 Continuous Testing > Test Automation 8 Continuous Testing and Agile + DevOps Success 9 Key Findings 9 Key Recommendations 10 Additional Research Findings 10 The Top Continuous Testing Roadblocks 11 Time and resources 11 Complexity 11 Results 12 The Path to Continuous Testing 13 Conclusion 14 www.tricentis.com © 2019 Tricentis GmbH. All Rights Reserved Transforming Testing for Agile and DevOps | Page 3 Let’s face it. Businesses don’t want—or need—perfect software. They want This guide explores what Continuous Testing really involves, to deliver new, business-differentiating software as soon as presents new research on why it’s so critical for Agile + DevOps, possible. To enable this, we need fast feedback on whether the and offers tips to help you make it a reality in your own team and latest innovations will work as expected or crash and burn in organization. production. We also need to know if these changes somehow broke the core functionality that the customer base—and thus the business—depends upon. This is where Continuous Testing comes in. There are four main sections Continuous Testing is the process of executing automated tests as part of the software delivery pipeline in order to ob- tain feedback on the business risks associated with a soft- ware release as rapidly as possiable. Continuous Testing does Continuous Testing vs Test Automation not require any specific type of testing approach (shift left, shift right…) or testing tools. However, it does require that: • Actionable feedback is delivered to the right stakeholder at the right time The Top Roadblocks to Continuous Testing • Testing occurs across all phases of the software delivery pipeline Test automation is essential for Continuous Testing, but it’s not sufficient. Test automation is designed to produce a set of Continuous Testing and Agile + DevOps Success pass/fail data points correlated to user stories or application requirements. Continuous Testing, on the other hand, focuses on business risk and providing insight on whether the software can be released. Beyond test automation, Continuous Testing also involves practices such as aligning testing with your busi- The Path to Continuous Testing ness risk, applying service virtualization and stateful test data management to stabilize testing for Continuous Integration, and performing exploratory testing to expose “big block” issues early in each iteration. It’s not simply a matter of more tools, or differ- ent tools. It requires a deeper transformation across people and processes as well as technologies. www.tricentis.com © 2019 Tricentis GmbH. All Rights Reserved Transforming Testing for Agile and DevOps | Page 4 Continuous Testing vs. Test Automation Like Lucy and Ethel struggling to keep pace at the chocolate fac- As expectations associated with test are changing, legacy test- tory, many software testers have been scrambling to keep pace ing platforms aren’t keeping up. Legacy testing platforms take with accelerated processes—then along comes the supervisor a “heavy” approach to testing. They rely on brittle scripts, de- proclaiming, “You’re doing splendidly…Speed it up!” liver slow end-to-end regression test execution, and produce an overwhelming level of false positives. As a result, they’ve achieved limited success with test automation. The overall test automation rate is 18%, on average—8% for enterprises. In a polling question asked at industry webinars and trade shows, respondents overwhelming reported that the results of test au- tomation to date have been “So-So.” Traditional Testing Doesn’t Work Recent changes across the industry are demanding more from testing while making test automation even more difficult to achieve. There are several reasons for this: • Application architectures are • Thanks to Agile, DevOps, and • Now that software is the primary increasingly more distributed and Continuous Delivery, many appli- interface to the business, an appli- complex, embracing cloud, APIs, cations are now released anywhere cation failure is a business failure. microservices, etc. and creating from every two weeks to thousands This is true even for a seemingly virtually endless combinations of of time a day. As a result, the time minor glitch that could have severe different protocols and technolo- available for test design, mainte- repercussions if it impacts the user gies within a single business trans- nance, and especially execution experience. As a result, applica- action. decreases dramatically. tion-related risks have become a primary concern for even non-tech- nical business leaders. Given that software testers are facing increasingly complex ap- to transform the testing process as deliberately and markedly plications, they are expected to deliver trustworthy go/no-go as we’ve transformed the development process. This requires decisions at the new speed of modern business. More of the more effective test automation…and more. This requires a dif- same traditional testing approaches won’t get us there. We need ferent approach called Continuous Testing. www.tricentis.com © 2019 Tricentis GmbH. All Rights Reserved Transforming Testing for Agile and DevOps | Page 5 What Is Continuous Testing? Continuous Testing is the process of executing automated tests Test automation is designed to produce a set of pass/fail data as part of the software delivery pipeline in order to obtain feed- points correlated to user stories or application requirements. back on the business risks associated with a software release Continuous Testing, on the other hand, focuses on business risk as rapidly as possible. Continuous Testing does not require any and providing insight on whether the software can be released. specific type of testing approach (shift left, shift right…) or testing To achieve this shift, we need to stop asking “are we done test- tools. However, it does require that: ing” and instead concentrate on “does the release candidate have an acceptable level of business risk?” • Actionable feedback is delivered to the right stakeholder at the right time • Testing occurs across all phases of the software delivery pipeline Here are 5 key attributes to Continuous Testing Assesses business risk coverage Requires a stable test environment is its primary goal to be available on demand Seamlessly integrates into the software Delivers actionable feedback appropriate delivery pipeline and DevOps toolchain for each stage of the delivery pipeline Establishes a safety net that helps the team protect the user experience www.tricentis.com © 2019 Tricentis GmbH. All Rights Reserved Transforming Testing for Agile and DevOps | Page 6 Comparing Continuous Testing with Test Automation The main differences between Continuous Testing and test automation can be grouped into three broad categories: Risk Breadth Time If your test cases weren’t built with business risk in mind, your Risk test results won’t provide the insight needed to assess risks. Most tests are designed to provide low-level details on whether user stories are correctly implementing the requirements—not Businesses today have not only exposed many of their inter- high-level assessments of whether a release candidate is too nal applications to the end user, they also have developed vast risky to release. Would you automatically stop a release from amounts of additional software that extends and complements taking place based on test results? If not, your tests aren’t prop- those applications. For example, airlines have gone far beyond erly aligned with business risks. exposing their once-internal booking systems. They now let customers plan and book complete vacations, including hotels, To be clear: We’re not suggesting that low-granularity tests aren’t rental cars and activities. Exposing more and more innovative valuable; we’re stating that more is needed to stop high-risk can- functionality to the user is now a competitive differentiator—but didates from going out into the wild. it also increases the number, variety and complexity of potential failure points. Here are some ways that testers can address risk: • Understand the risks associated with the complete applica- Large-scale “software fails” have such severe business reper- tion portfolio cussions that application-related risks are now prominent com- • Map risks to application components and requirements ponents of a business’ public financial filing. Given that notable (which are then mapped to tests) software failures resulted in an average -4.06 percent decline in • Use a test suite that achieves the highest possible risk cov- stock price (which equates to an average of negative $2.55 bil- erage with the least amount of test cases lion loss of market capitalization), it’s not surprising that business • Always report status that shows risk exposure from busi- leaders are taking note—and expecting IT leaders to take action. ness, technical, performance, and compliance perspectives www.tricentis.com © 2019 Tricentis GmbH. All Rights Reserved Transforming Testing for Agile and DevOps | Page 7 Breadth Even if a business manages to steer clear of large-scale software Here are some ways to address testing breadth: fails that make the headlines, even seemingly minor glitches can • Define and execute complete end-to-end tests that exer- still