The Rise of Continuous Testing Accelerating Application Delivery and Quality An Unlikely Ally

As the application economy drives companies to roll out applications more quickly, companies are seeing testing in a new light. Once considered a speed bump on the DevOps fast track, new tools and testing methodologies are emerging to bring testing up to speed. In this ebook, we’ll explore some of the challenges on the road to continuous testing, along with new approaches that will help you adopt next-gen testing practices that offer the ability to test early, often and automatically.

A recent poll showed that more than 75% said they wanted to “shift left” and move testing from downstream back into development.1

1 Freeform Dynamics, “Continuous Testing: Lessons from Leading Practitioners”. 2 Want to Find the Source of Many Significant Challenges?Look to the Left.

It’s clear that traditional testing models just don’t work in a This disconnect between requirements DevOps setting—your world moves too fast, and there’s too and the final output can have costly much at stake to rely on the slow, mostly manual status quo. consequences. If you can’t get your The initial problems start all the way to the left of your software requirements right—and do it development lifecycle in the requirements phase. In fact, 64 quickly—then you risk: percent of total defect cost can be traced to poorly defined or incomplete requirements.2 Costly rework Defective applications Don’t send your teams User expectations are Ambiguous requirements mean poor outcomes and “back to the drawing skyrocketing—they’ll close board” late in the and delete your app in big risks software development seconds if they sense When your requirements aren’t clear, you usher in an array of lifecycle. performance issues. problems—defective software, delays and clashes between the business and IT, to name just a few. Today, most testing is still done manually, but these methods are a remnant of the past and don’t provide enough coverage, covering just 10 to 20 percent of functionality.

2 Hyderabad Business School, GITAM University, “Quality Flaws: Issues and Challenges in Software Development,” November 2012

Damage to your brand If users detect a pattern of poor design or execution, negative reviews and high abandonment rates could harm your company’s position in the marketplace.

3 Use Modeling and Automation for Clear, Unambiguous Requirements

Everyone wants lower testing costs, better quality and faster Improve Quality application delivery—the question is “How?” To bring these benefits to your own organization, you need 80% increase in test coverage to adopt a new approach that: • Allows you to design requirements with extreme clarity so potential defects can be eliminated or addressed in fewer hours instead of Boost Productivity days or weeks • Helps you test more efficiently, so you can run the smallest number 30% of tests possible to cover 100 percent of your testing criteria reduction in test cycles • Fosters automation in test case design, test generation and change implementation Save Time Right now, you probably have too many manual steps that slow you down. Defining requirements, designing test cases and other manual testing activities are keeping you from moving faster and ramping up your 99%+ reduction in time to reflect applications’ quality. changes in tests Implementing advanced requirements engineering and test design automation capabilities Learn more about CA Agile At CA, this is the approach that guides the continual refinement of our Requirements Designer CA Agile Requirements Designer solution.

With CA Agile Requirements Designer, you’ll automatically generate the optimal set of tests right from your requirements modeled as complete and unambiguous flowcharts. Then when requirements change, these tests will update automatically so you can deliver software that reflects changing user needs.

4 How Much Time Do Testers Spend Looking for Test Data?

If you work in testing, you already know that time wasted looking for the right test data is a pervasive problem. But looking for test data is only one part of the issue—finding the right data or creating it manually consumes about 50 percent of a tester’s time.3 And a staggering 70 percent of all test data is still being developed manually.4

In addition to wasted time, there are Don’t let defects remain unresolved Improper handling and formatting of other consequences that stem from until late in the development cycle, test data gathered from production these challenges. where they require far more time and databases can expose you to breaches, expense to fix. fines and reputational damage. Decline in software quality Increased exposure to compliance Delayed agile implementation and data privacy issues

Lengthy searches for test data and manual provisioning not only hinders the testing process, it also slows down the entire supposedly agile organization.

3 CA Technologies implementation field experience with actual customers 4 Bloor Research, “A Bloor Market Update Report: Test Data Management,” March 2017

5 Accelerate the Rate of Test Data Creation to the Speed of Agile

You’ve seen your teams wait days or weeks for the right test data, or spend long hours manually developing it themselves. How can you get away from this tired process that relies only on production data and start testing at the speed of agile? By exploring new technologies that: • Support more data sources, including unstructured data, along with information in NoSQL environments and mainframes. • Mass produce test data automatically and quickly, using sophisticated synthetic data generation that simulates the variability and formats of production data. • Mask data the right way so you can get the most out of production data when needed (without sacrificing security or violating compliance protocols). Mitigating problems and achieving real cost savings • Extensive and flexible synthetic data creation • Access to their own copy of test data via virtual test data creation • Ability to ramp up their data creation speed to match the pace of your agile environment Finding, improving, protecting and delivering test data is made easier, and subsetting and masking is more efficient, giving your testing teams greater ROI from your tools during this phase of your software development lifecycle.

Learn more about CA Test Manager

6 Have You Hit the “QA Wall”?

To compete in the application economy, software developers like you are looking for ways to ramp up speed, boost quality and deliver a How can you help your testers quality experience to your customers. And you’re not alone—nearly contribute to the overall speed of the 75 percent of companies have adopted DevOps processes to help DevOps team? them accelerate the software delivery lifecycle. But regardless of the By adopting a way to simulate benefits by DevOps, you’re still running into the “QA wall.” In one unavailable systems. survey, 63 percent of respondents cited test/QA as the main hold-up in the software production process.5 It’s easy to see why this is a headache for your testers, and the testing phase also slows down and undermines your overall approach to rapid application development. Failing to break through this bottleneck can result in:

Longer cycles Higher costs Diminished quality

29% of DevOps Testing a wide variety Inefficient testing practitioners say of devices and practices can increase that quality was environments can mean the pressure on your being sacrificed due allocating funds to hire entire DevOps team. to QA challenges.6 additional testers.

5 Computing Research, “DevOps Review 2017,” March 2017 6 Computing Research, “DevOps Review 2017,” March 2017 7 Get Exactly What You Need (When You Need It)

Continually releasing software requires collaboration across each phase of the software development lifecycle. Testers are typically CA Service Virtualization at a disadvantage because they can’t access the incomplete or customers have seen: unavailable services required to complete their tasks. However, by creating virtualized services across the entire software supply chain, testers can contribute more value to the Up to overall development process. With this approach, you can: 50% • Capture and record traffic or messages flowing between any two systems. • Model captured data and correlate it into a virtiual service, which is a faster time “conversation” of appropriate requests and responses that is plausible to market enough for use in development and testing. • Simulate to respond to requests with appropriate data just as the real thing would, except with more predictable behaviors and much lower setup and teardown cost. Up to Creating virtual services for the test environment 90% Developers, integration testers and performance testers can all move more quickly with CA Service Virtualization. You can now simulate systems that aren’t available, regardless of where they are across the software more defects found development lifecycle. (at least one stage Teams can now collaborate more efficiently, allowing you to accelerate earlier in the software release cycles and reduce testing environment infrastructure costs. delivery lifecycle.)7 Learn more about CA Service Virtualization

7 Forrester Consulting, “The Total Economic Impact™ of CA Technologies Service Virtualization,” December 2015 8 Do You Have Difficulties Scaling Test Coverage for Your Agile Teams?

To bring apps to market quickly in the age of digital For the Test Center of Excellence (CoE) in the transformation, companies have adopted the use of DevOps practice, this holds specific challenges: agile development and DevOps delivery. However, testing as a discipline has fallen behind, relying on legacy tools and methods that are poorly suited for Tool incompatibility resulting in rapid, quality application development. management complexity Agile teams forced to switch out of their preferred Infrequent testing leads to bad applications. If your open-source tools in their local IDEs today results testing practices haven’t “shifted left,” where we test in QA as the bottleneck with larger operating for application performance early and often in the pains and longer development cycles. software development lifecycle, you’re preventing your developers from contributing to the continuous delivery culture needed for success in the application Inability to scale up and out efficiently economy. for performance The reality is that performance testing of applications Load testing to over 2 million virtual users at is often an afterthought given today’s heavily manual enterprise scale (for projects across multiple test processes and limited availability of test resources distributed teams in the same instance) is a challenge for Test CoEs today. for development teams.

Under delivering on test runs for continuous testing Overwhelmed by proliferation and complexity in application delivery, the VP of Applications are limited in their ability to accelerate test automation for test runs across the (CI/CD) pipeline.

9 Democratize Your Performance Testing

“Test early, test often” should be the motto of any organization seeking to stay competitive in the application economy. In the modern software factory, testing should be democratized, offering teams the ability to run tests to ensure application quality, regardless of their place in the development cycle. As you bridge and broker enterprise coverage at scale from the Test Center of Excellence, performance engineer your applications in the cloud for continuous quality. By cultivating this approach, you can: Run open-source-based load-performance and API tests continuously anytime, anywhere at enterprise • Offer self-service capabilities in the cloud so scale from your DevOps practice that developers can easily run open-source-based performance tests that matters from their local IDE With a next-generation, browser-based SaaS testing of choice. This lets them test early and often solution like CA BlazeMeter® for the modern SDLC, regardless where they are around the world and at align your current testing platform in the Test Center any time of the day. of Excellence to open-source-based tools and best • Scale load testing to over 2 million virtual users practices that agile development teams use. Stay ahead hitting the application simultaneously across multiple of the market by democratizing performance and application projects. Bridge and broker for enterprise function testing of Web/mobile/microservices/API over test coverage at scale in the Test Center of Excellence, a single integrated toolchain. to performance and function test over a single Bridge and broker enterprise test coverage at scale platform. across the tens/hundreds of application projects in • Seamlessly test and integrate continuously with a the business. wide variety of tools, such as Jenkins, Bamboo and TeamCity, for continuous delivery. DevOps practices Remember to leverage the extensiveness of DevOps can performance engineer applications in the cloud integrations out of the box for the benefits of a CI/CD continuously for quality, as they leverage on OOTB pipeline, as you accelerate test automation continuously DevOps integrations that help accelerate test for quality applications in the cloud. automation across Dev/QA and ops frictionlessly. Request your CA BlazeMeter demo now. www.blazemeter.com/shiftleft

Learn more about CA BlazeMeter

10 How Risky Are Your Releases?

Only 16% Unfortunately, much of your testing still relies on manual methods with only 16 percent of of test test activities automated on average.8 What’s more, testing often is siloed across the activities organization—disconnected people and tools—leading to inconsistent visibility into testing automated progress an testing results. on average8 The lack of collaboration, visibility and feedback across the release process can create unnecessary release delays and risk with business-impacting consequences:

Missed deadlines Lost time Disappointing quality For every day a release is late, that’s Too much manual work, rework, Consumers abandon loyalty fast with unrecoverable revenue lost. Your unplanned work, wait time and technical all the choice in the market. If your competitors are out in market before debt creates inefficiency in your applications don’t perform as expected you. Your customers are missing key pipeline—hours that could be spent on or have too many issues, consumers find functionality enhancements. innovation and your competitive edge. another solution. Focusing on producing the highest quality apps is critical.

In the modern software factory, testing can’t be an isolated workflow disconnected from your application release process. Quality must become everyone’s concern—a shared responsibility across development, testing, release and operations teams. And these teams need to be able to work together effectively and efficiently to deliver high quality work, on time.

8 Capgemini and Sogeti, “World Quality Report 2017-18” 11 Orchestrate the Delivery of Quality

Testing is still stuck in legacy mode, relying on manual processes. But it doesn’t have to be this way—with the right approach, you can “shift left,” bring in more automation, and start intelligently testing apps.

Building more efficient continuous testing frameworks for quality delivery CA Continuous Delivery Director helps your testing strategy come to life by orchestrating the delivery of quality. It provides testing transparency, automation and intelligence for the modern software factory by managing testing stages, change validation and delivery risk Reduce release across your organization. risk and You can now connect the testing toolchain from planning into production, supporting improve pipeline intelligent, automated testing cycles, fast testing feedback, complete visibility of all efficiency 9 testing activities, and testing insights that inform release health. You’ll be in complete by 30%. control of your testing workflows ensuring the right tests are run in the right environments at the right time.

With this new way of thinking, you can:

Go faster by implementing a Adapt quickly to the code change Improve customer experience mission control that reduces committed to the pipeline, which by gaining more insight into cycle time. means you find and fix issues levels of quality and release earlier and more efficiently. readiness.

Learn more about CA Continuous Delivery Director

9 CA Continuous Delivery Director Business Value Estimation 12 Broadcom Inc. is a global infrastructure technology leader built on 50 years of innovation, collaboration and engineering excellence. Broadcom Inc. (NASDAQ: AVGO) is a global technology leader that designs, develops and supplies a broad range of semiconductor and infrastructure software solutions. Broadcom’s category-leading product portfolio serves critical markets including data center, networking, enterprise software, broadband, wireless, storage and industrial. Our solutions include data center networking and storage, enterprise, mainframe and cyber security software focused on automation, monitoring and security, smartphone components, telecoms and factory automation.. For more information, go to www.broadcom.com.

For product information please visit our website at: broadcom.com Copyright © 2020 Broadcom. All Rights Reserved. Broadcom, the pulse logo, Connecting everything, CA Technologies, and the CA technologies logo are among the trademarks of Broadcom. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. BC-0572EN January 06, 2020

13