Why You Need AI in Your Continuous Testing Strategy

Why you need AI in your continuous testing strategy

How are business leaders responding to AI? ...... 5

How can AI help your testing practice?...... 6

What can you do today?...... 7

Executive Summary

At the top of every CIO’s mind is how to drive for- AI is being implemented in an infinite number of ward innovation at speed, increase levels of pro- ways in testing: ductivity, deliver superior and highly scalable dig- • Creating automation based on an applica- ital products and services that increase customer tion’s mockup or simple design satisfaction, and save costs – all at the same time. Unfortunately, without modernizing your testing • Handling maintenance of rapidly changing with intelligent automation, and treating testing applications as a strategic component to your digital success, • Automating hard-to-reach interfaces like you won’t be as competitive and you can’t inno- remote applications vate as quickly. How do we get there? • Performing self-healing upon changes to Artificial intelligence (AI) and machine learning object recognition (ML) technologies are already being adopted by • Reducing the testing scope in large regres- businesses today in QA and test automation to sion suites through smart impact analysis deliver more resilience, agility, speed, and business growth. More than half of business leaders All these trends show us that smart technologies have witnessed how AI solutions have already will undoubtedly increase cost efficiency, reduce the need for manual testing, shorten time to increased productivity (IDC FutureScape, 2018). Many more executives (63%) say that reducing market, and, most important, help create and costs is a major factor in adopting AI solutions. sustain the circle of continuous testing and continuous quality. Gartner (2019)1 agrees: improved defect detec-

tion (48%), reduction in test maintenance costs This white paper explores why business leaders (42%), and improved test coverage (41%) are and CIOs urgently need to adopt AI in their contin- some of the top benefits that AI/ML brings to uous testing strategy, and also provides some software testing. key considerations on how AI/ML can be used to AI is a “must-have” when selecting new QA solu- accelerate business growth, innovation, and tions, products, or tools – and 86% of respon- competitive advantage. dents agree (World Quality Report, 2020-2021).

1 Gartner (2019). Critical Capabilities for Software Test Automation. (Based on the 2019 Software Quality Tools and Practices Survey) tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 2 Why do we need AI in software testing?

This section dives into some of the challenges and market trends of software testing in order to address why new approaches such as AI/ML are urgently needed to drive digital success.

A rapid rise in modern application Test maintenance is a huge cost and development resource expenditure

Today’s leading enterprises strive to deliver How do you go from releasing once per year to high-performance, highly scalable, and always-on twice every month? This question trickles through digital services. These services are built on the minds of most CIOs who want to increase custom “modern architectures” – an application speed of innovation and drive forward digital suc- stack that relies on new tiers of technology, such cess. Without modernizing testing with intelligent as microservices and containers, typically running automation, you’ll discover that the time and cost on cloud platforms. In fact, the development of of testing will grow disproportionately to your modern applications is surging at a dizzying rate. desired innovation speed. The point is that the IDC predicts that by 2023, over 500 million new maintenance of script-based testing tools make applications will be introduced. it impossible for you to accelerate your testing process and innovation. At the same time, customers rely on legacy systems and old technologies for various busi- With script-based testing approaches, you may ness-critical processes like processing customer initially get high automation rates. However, as data. Technology is often pulling us in multiple, your application evolves and new features get competing directions. We’ve been slow in coming added, and as your landscape of integrated up with better strategies for testing both modern applications and systems increases in scope, and legacy applications. Even the earliest adopt- the maintenance that’s required for you to keep ers of DevOps and agile practices cannot keep up up with those releases will simply become too with the fast-moving technological changes. difficult, resource-heavy, and cost-intensive. Your automation rates will plummet, and your manual The COVID-19 pandemic has also shifted much testing will skyrocket. of the world to remote work environments. Organizations who adopt modern applications We need alternative methods to script-based test- to maintain agility are already realizing benefits. ing. AI/ML approaches to test automation already According to the VMware global survey (2020), exist and are being implemented today to reduce cloud-native solutions are allowing organizations the time and cost of burdensome test mainte- to enable remote workforces, push quick up- nance. AI-powered test automation uses neural dates in response to a changing landscape, and networks to see and steer elements on a page, maintain reliable uptime. This will lead to an even without needing to look at the technical layer. greater demand for virtual and remote desktop interfaces like Citrix and VMware. UI testing is slow, and being slow is costly

To cope with these modernization initiatives, The sooner you find a bug, the cheaper it is to fix advanced AI test automation capabilities are it. This is why it is essential to test as early as pos- already being applied. Organizations can reduce sible in the development lifecycle. DZone Trend modernization costs and efforts, as well as build Report (2020) states that more and more orga- enhanced test automation for remote and virtual nizations are shifting their attitude to “test early apps, including legacy and modern apps. and test often.” One major challenge, however,

tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 3 is that the User Interface (UI) of an application In order to close the skills shortage gap as well as has to be built before any test automation can meet the demands of the proliferation of new ap- occur. As a result, UI automation continues to be plications, it has become more urgent than ever to a bottleneck in quicker testing, development, and embed business users or SMEs into the test auto- delivery. mation process. AI can augment the amount of test automation that needs to be achieved by business- According to the 2020-21 World Quality Report, es today and thereby help narrow the skills gap. testing still represents nearly 30% of IT budgets. With AI-powered test automation, we can finally Importance of low-code/no-code solutions create tests as early as possible in the development lifecycle – even before the UI is created. AI/ Pega (2020) states that code is a major obstacle ML technology provides you with the capability to for IT and business collaboration. In fact, create tests in the design phase – based on a sim- underutilizing low-code platforms is a major ple mockup – which enables an extreme Shift Left missed business opportunity. Adopting low-code approach to testing. Testing earlier means testing or no-code solutions will empower business faster, which can reduce the cost of testing and al- users to take ownership of their applications low you to ship more innovation to your customer. whilst being guided by IT on best practices and security. AI/ML is already being incorporated into Suitable for multi-skilled personas low-code/no-code platforms in order to enhance the automation of business workflows and help Enterprises today prefer a testing platform that is embed the business user even deeper into the more accessible to a broader range of skill levels test automation process. (Forrester Wave Q2, 2020). Script-based test automation leaves non-technical and business DevOps means more automation users out of the testing game, leading to an inefficient utilization of your organization’s resources. DevOps demands more automation in order to meet faster delivery without compromising on In testing, a skills shortage could translate to a quality. Continuous testing, or integrating test- lack of coding and scripting knowledge to create ing continuously throughout the CI/CD pipeline, tests, as well as a lack of domain knowledge of a is important so that testing doesn’t become a business application. Business users or subject bottleneck in the software delivery process. AI is matter experts (SMEs) working in industry-spe- already playing an important role in scaling the cific domains like banking, retail, healthcare, and DevOps process, particularly in achieving more telecom have the advantage of applying domain automation throughout the CI/CD pipeline: knowledge of an application for more efficient and comprehensive testing.

Continuous integration (CI) Continuous Delivery (CD) Continuous Deployment (CD)

Automated testing and artifact Automated deployment to test Automated deployment to creation and staging environments Production

Augmented automation is, of course, just one example. AI can also assist in “predicting” how the code will behave by analyzing the entirety of DevOps processes across various toolchains.

Business leaders’ and executives’ response to AI/ their understanding of the business resources ML approaches for test automation is optimistic. and the set-up needed to make new technology work well. Additionally, Pega (2020) outlines that According to Pega (2020), business leaders are 67% of organizations are also using AI to support eager to explore the opportunities that AI can decision-making and 64% are using AI to reach deliver, but they still have a lot to learn. 51% of decisions without human input. respondents think senior leaders need to improve their understanding of how AI changes pro- The IDC FutureScape: Worldwide IT Industry cesses and affects jobs. Additionally, 50% of re- 2018 summarizes that business and IT leaders’ spondents think senior leaders need to improve common response to AI is positive:

23% 83%

have incorporated AI say AI is a strategic into processes, product priority for their businesses & service offerings

31% 54%

plan to invest in AI in AI solutions implemented in the next 12 months their business have already increased productivity

63% 51%

say pressure to reduce think senior leaders need to costs will require them improve their understanding to use AI of how AI changes processes and affects jobs

57% 61%

said they are planning said AI/ML features would to adopt AI in QA be very valuable in software testing tools

Application modernization and digital transformation

AI has the capability to power resilient, reliable, and flexible test automation for both modern and legacy applications, and across different technologies. This will allow you to modernize your testing and drive forward digital transformation, without having to spend too much money on a complex toolset, customization, or manual testing that only slows down modernization efforts.

AI/ML solutions can also create reliable automation for remote or virtual desktop infrastruc- tures (VDI) with ease, as it detects objects on an interface by looking at the visual cues only, not at the underlying technical layer. Remote or virtual desktop interfaces such as Citrix or VMware will become even more relevant, given the shift to remote work.

Release faster, deliver more innovation, and cut costs

In-sprint automation is finally possible. With more robust and adaptable to the changes that AI-powered test automation, you can create test inevitably occur with each application update assets in the design phase of your application – and upgrade. Self-healing capabilities can also based on simple mockups and prototypes. These automatically adapt test cases without needing same tests can be reused again and again, even to waste resources on heavy refactoring, which when the application is fully developed. The time directly leads to lower costs, more ROI, and faster span of testing can be decreased from days to product releases. mere minutes, which translates directly into cost savings. Increase release predictability

AI-assisted test automation can also significantly Despite having DevOps tools in place, many reduce test maintenance and the time needed enterprises are still “stuck in the middle.” Some to fix costly tests that slow down innovation. have made some progress with agile delivery, but Agile teams can free up resources – rather than not consistently across all teams and projects. fixing tests, they can focus on planning activities Daily production deployments are still a dream to ensure more efficient and smarter testing. for many. In general, there is demand for better Through advanced machine learning technol- visibility when and where defects are likely to ogy, AI solutions can detect and steer objects occur. Utilizing data for predictive models like this like the human eye. Test automation becomes can be of huge benefit.

Research tools and vendors for your specific business needs

The question for your organization is no longer if creating tests based on simple mockups, and, you should adopt AI in your existing test auto- therefore, testing much earlier in the develop- mation activities and process, but how. If you’re ment cycle. AI-powered test automation is not evaluating AI tools, you need to carefully consider just used for visual testing, it can also create test your business needs, priorities, and initiatives – automation for modern and legacy enterprise will AI be implemented to de-risk your releases applications and systems in order to keep end-to- and improve quality? Or do you need AI to reduce end and integration tests running. the maintenance of your test automation, im- Modernize your continuous testing prove resilience, and increase speed? capabilities – holistically There is an overwhelming variety of testing tools in the market today who claim to use ML and Organizations across industries and company AI. Therefore, it is crucial to carefully assess how sizes have realized the upside of testing early these tools work, what testing challenges they and continuously throughout the development claim to tackle, how the technologies will learn lifecycle (DZone, 2020). Ensuring that you have a and evolve over time, how they differentiate holistic approach to continuous testing is ab- themselves from other AI-based solutions, and solutely critical to innovation at speed. Forrester what their technologies can’t do. Not having this (2020) outlines 12 critical must-do’s for achieving insight can lead to poor strategic and procure- continuous testing and highlights that the most ment decisions. effective practices are those that encompass people, process, and technology. For instance, carefully consider the difference between visual testing tools and AI-assisted test automation. At the simplest level, visual testing tools compare visual differences between a base- line image and a checkpoint image of an application. These tools focus on design elements, such as validating the size, position, and color scheme of visual elements. They are intended for regres- sion testing and focus on finding cosmetic bugs. Most visual testing tools on the market today do not have automation capability and depend on test automation frameworks to drive them.

AI-powered test automation, on the other hand, uses neural network technology to steer elements on a user interface. As such, this technology is better suited for reducing test maintenance, creating stable and resilient test automation,

tricentis.com © 2021 Tricentis GmbH. All Rights Reserved | 7 Even when looking for AI-powered solutions, it AI will enhance test automation, not fully replace is important to honestly analyze your challenges it. If anything, AI will provide testers with more and continuous testing maturity. Without proper time to focus on what is important. And focusing metrics and insight, it becomes difficult to know on what is important actually makes testers more which AI solutions you should consider. The valuable. There are an infinite number of areas in Tricentis Continuous Testing Maturity Assess- testing that will always require human collabora- ment can help you identify your gaps, quickly tion, interaction, and involvement. For instance, and reliably, in order to enhance your short-term ensuring that business requirements are properly and long-term strategy. For instance, if you are mapped out to testing activities, understand- struggling with inefficient test execution, and lim- ing the integration of business applications and ited time to run thousands of test cases, you may interconnected enterprise technologies. Freed-up consider adopting smart impact analysis. This time and resources can also allow teams to focus approach uses AI to identify most-at-risk objects on developing a business-driven mindset that in your application updates and upgrades. Alter- requires soft skills for efficient collaboration within natively, if you want to lower your maintenance and across agile teams. These comprise multiple costs and utilize your resources more efficiently, personas, from developers, software development AI/ML technology can be used to deliver more engineers in test (SDETs), business users or sub- resilient test automation, cut costs, and propel ject matter experts, manual testers, and more. your business efficiency forward. Secondly, AI needs a lot training. We need to feed Combining brains – AI and embracing change machine learning systems with a lot of data in order to ensure they deliver the resilience and Every new introduction of technology brings reliability needed to deal with frequently chang- about an equal dose of skepticism and excite- ing applications. It has become more important ment. Rightfully so – it is a natural human tenden- than ever to monitor AI as we feed it with both cy to question, be curious, and to experiment. good and bad data, or as we implement different Unfortunately, the common attitude is that most ML techniques as systems go from supervised to testing and QA jobs are at stake and that people unsupervised learning. will be replaced by AI automation and robots. This thinking is understandable but faulty. In fact, All in all, utilizing AI for software testing has the potential just the opposite is more likely. for monumental impact. It is important to embrace the positive business advantages, yet also be wary of the limitations of AI/ML technologies and develop ways to mature them in our testing practices for the future. Final Thoughts Tricentis customers are already leveraging AI to increase the speed and resilience of testing, and reduce time to market. Want to learn how you can too? Reach out to us here.

Tricentis embeds AI capabilities across the portfolio in order to deliver a comprehensive strategy that will allow agile and DevOps teams to keep up with application changes, and take the pain out of test design, execution and maintenance. Learn more.

Tricentis is the global leader in enterprise continuous testing, widely credited for reinventing software testing and delivery for DevOps and agile environments. The Tricentis AI-based, continuous testing platform provides automated testing and real-time business risk insight across your DevOps pipeline. This enables enterprises to accelerate their digital transformation by dramatically increasing software release speed, reducing costs, and improving software quality. Tricentis has been widely recognized as the leader by all major industry analysts, including being named the leader in Gartner’s Magic Quadrant five years in a row. Tricentis has more than 1,800 customers, including the largest brands in the world, such as Accenture, Coca-Cola, Nationwide Insurance, Allianz, Telstra, Dolby, RBS, and Zappos.

To learn more, visit www.tricentis.com or follow us on LinkedIn, Twitter, and Facebook.

AMERICAS EMEA APAC 2570 W El Camino Real, Leonard-Bernstein-Straße 10 2-12 Foveaux Street Suite 540 1220 Vienna Surry Hills NSW 2010, Mountain View, CA 94040 Austria Australia Unites States of America [email protected] [email protected] [email protected] +43 1 263 24 09 – 0 +61 2 8458 0766 +1-650-383-8329

v. 0521