Continuous Quality and Testing to Accelerate Application Development

How to assess your current testing maturity level and practice continuous testing for DevOps

Continuous Quality and Testing to Accelerate Application Development // 1 Table of Contents

03 Introduction

04 Why Is Continuous Quality and Testing Maturity Important to DevOps?

05 Continuous Testing Engineers Quality into DevOps

07 Best Practices for Well- Engineered Continuous Testing

08 Continuous Testing Maturity Levels

Level 1: Chaos

Level 2: Continuous Integration

Level 3: Continuous Flow

Level 4: Continuous Feedback

Level 5: Continuous Improvement

12 Continuous Testing Maturity Assessment

13 How to Get Started with DevOps Testing?

14 Continuous Testing in the Cloud

Choosing the right tools for Continuous Testing

On-demand Development and Testing Environments with Infrastructure as Code

The Right Tests at the Right Time

20 Get Started

20 Conclusion

21 About AWS Marketplace and DevOps Institute

21 Contributors Introduction

A successful DevOps implementation reduces the bottlenecks related to testing. These bottlenecks include finding and setting up test environments, test configurations, and test results implementation.

These issues are not industry specific. They can be experienced in manufacturing, service businesses, and governments alike. They can be reduced by having a thorough understanding and a disciplined, mature implementation of Continuous Testing and related recommended engineering practices.

The best place to start addressing these challenges is having a good understanding of what Continuous Testing is.

Marc Hornbeek, the author of Engineering DevOps, describes it as:

“A quality assessment strategy in which most tests are automated and integrated as a core and essential part of DevOps. Continuous Testing is much more than simply ‘automating tests.’”

In this whitepaper, we’ll address the best practices you can adopt for implementing Continuous Quality and Testing on the AWS Cloud environment in the context of the DevOps model.

We will explore: • How you can effectively implement Continuous Testing and enable your engineering and quality assurance (QA) teams to get seamless self-service access to both the environments and tooling using AWS • How to develop a roadmap for Continuous Testing implementation based on maturity and assessment models • How to get started with DevOps testing

These suggested guidelines can help you reduce costs and time-to-market and allow your developer teams to stay focused on delivering value. Why Is Continuous Quality and Testing Maturity Important to DevOps?

The acceleration in DevOps adoption has fundamentally transformed how testing is done. Before the advent of DevOps, most quality testing happened at the end of the cycle. Now, testing increasingly takes place in parallel with development and is engineered for quality.

A comprehensive set of tests are, in aggregate, a prescription of expected behaviors for applications within the production environments. The expected behaviors are more than application features and functions. Non-functional requirements, such as performance, maintainability, and reliability, are also essential. Continuous tests are designed to cover both functional and Non-Functional Requirements (NFRs). The capabilities needed to operate an application in each of its deployment environments are part of Continuous Testing.

Although Continuous Integration adoption is becoming ubiquitous – with 88% of enterprise software companies adopting it in 2020, test suites are still plagued by a growing backlog of failing tests. Failures in integration and automated tests cause bottlenecks in the development pipeline and substantially increase engineering costs. When engineered well, Continuous Testing can provide impressive benefits, including reduced time-to-market, improved quality, reduced cost, improved innovation, and team satisfaction.

According to a May 2020 research report published by UNDO1:

88% 35% of enterprise software companies adopting Continuous of respondents say they deploy code changes hourly.

Integration in 2020 compared to 70% in 2015.

26% $61billion of developer time is spent an average of 13 hours reproducing to $1.2 trillion annually total salaries spent in enterprise value

and fixing falling tests, which equals 620 million developer lost for shareholders per year.

hours a year wasted on debugging software failures.

1The Business Value of Optimized Pipeline – UNDO 2020

Continuous Quality and Testing to Accelerate Application Development // 4 Continuous Testing Engineers Quality into DevOps

Continuous test engineering requires a complete re-think of software QA test strategies, so they align with Dev-through-Ops pipeline stages. Fortunately, best practices for DevOps testing are available and can be learned and applied to any product or service.

Five tenets underlie the best practices for DevOps test best practices are indicated in Figure 1 and explained below.

Fail often Run tests frequently and with many different conditions

Fail Early Arrange the tests so that the most likely problems are found Relevance in the earliest possible stage in Focus on the most important the DevOps pipeline tests and results

Shift Left Test Fast Conduct each test as early in the Arrange tests to run pipeline as possible in quick cycles

Figure 1: Continuous Testing Tenets

Continuous Quality and Testing to Accelerate Application Development // 5 The CD pipeline starts with the release of candidate artifacts delivered from the CI pipeline into an artifact repository. The CD pipeline is automated, and artifacts are deployed to a pre-production staging environment for system and user acceptance testing. These tests are automated as much as possible to minimize processing delays and bottlenecks. Once the tests pass, the artifacts are assessed with release policies to win approval for deployment. The release acceptance criteria are often determined by a Change Approval Board (CAB) and implemented as codified policies (if possible).

Once approved, the artifacts are deployed to production according to a deployment schedule. To minimize risk, the deployments are conducted using the safe Green/Blue and Dark Launch methods. Once deployed, additional post-deployment tests may be used to evaluate software options using A/B testing and Canary testing methods.

Figure 2: DevOps Continuous Testing Blueprint

Testing, an essential portion of DevOps, is responsible for continuous assessment of incremental changes and is part of establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, reliably, and safely.

DevOps Testing – the Primary Key to DevOps and Continuous Delivery” by Marc Hornbeek

Continuous Quality and Testing to Accelerate Application Development // 6 Best Practices for Well-Engineered Continuous Testing

Continuous Testing concepts must be mastered. Every test that can be automated should be automated. Tests should be executed in production-equivalent test environments. The results must be assessed in each stage of the end-to-end pipeline to determine if the artifacts are acceptable or need remediation before promotion.

Culture for Continuous Testing is critical. Quality is everyone’s responsibility. Leaders need to sponsor, inspire, fund, and motivate cross-functional teams to collaborate, learn, implement, and improve Continuous Testing practices.

Test strategies for Continuous Testing include specific objectives. Automated and even manual tests are triggered at each pipeline stage. Elastic cloud test environments are automatically orchestrated to match testing requirements of each change. Tests are shifted-left an accelerated as early as possible to find customer- relevant verdicts fast and early. The use of the A/B feature flag, Blue-Green, and Canary deployments strategies help validate user alternatives and performance safety in production.

Test Frameworks and Tools are selected for integration and performance. A test framework acts as a “backbone” for integrating and automating tasks including test plans, all types of test tools, test artifact version management, test resources, test data, test schedules, test environment orchestration, test results, test reports, and test results analysis.

Test Planning includes test strategies that describe test requirements, management policies, resources, topologies, and priorities for each product module to meet test strategy requirements. Test plans define test cases and priorities for each product module to meet test strategy requirements.

Test Automation is used extensively. Automated tests are created per requirements described in test strategies and plans, using a test automation creation strategy such as TDD. Tests and test tools are automated, orchestrated, operated, and results analyzed through an API by test cases scheduled by a framework.

Test Management and Analysis. Manage resources (financial, labor, schedules, and flexible scaling) to meet specific CT quality, delivery, and response time goals determined by key stakeholders.

Continuous Quality and Testing to Accelerate Application Development // 7 Continuous Testing Maturity Levels

Many organizations have not achieved the level of continuous test maturity described above. For an organization to mature its continuous test capabilities, it needs to know its current state of maturity and the target of its future state.

There is no standard continuous test maturity model. A useful way to approach it is to apply a five- level maturity model adapted from the “standard“2. The following paragraphs describe the five levels of continuous test automation maturity, as defined by Marc Hornbeek.

There are three dimensions to consider at each level: People, Process, and Technology. Technology, such as better test tools, will not accomplish a higher maturity level unless the People and Processes use them effectively.

Continuous Testing Maturity Level 1: Chaos

Figure 3 shows the key characteristics of People, Process, and Technology, evident at this maturity level. There is little indication of Continuous Testing skills at this level, test automation processes are not apparent, and test automation technology has significant gaps.

Typical outcomes are releases that are unpredictable, reactive, and the pipeline is wasteful. Response to resolve the problems found by testing is slow.

People Process Technology

• Silo team organization • Testing not part of • Missing tools to planning test performance of • Little knowledge of applications, pipelines Continuous Testing • No test standards and infrastructure • Blame, finger-pointing • Few automated tests

Figure 3: Continuous Testing Maturity Level 1: Chaos

2The original software Capability Maturity Model (SW-CMM) was developed by the Software Engineering Institute (SEI) at Carnegie Mellon University in the mid-1980s.

Continuous Quality and Testing to Accelerate Application Development // 8 Continuous Testing Maturity Level 2: Continuous Integration

Figure 4 shows the key characteristics of People, Process, and Technology, evident at this maturity level. At this level, there is some knowledge of Continuous Testing. Processes and technology well support the integration and build processes for Testing. End-to-end Testing, especially delivery and deployment stages, are not well supported with automated Testing.

At this stage, there are deployment quality problems and significant gaps in the infrastructure and process of the testing system. The test processes tend to have insufficient fidelity to indicate the source of failure unambiguously. This results in a situation where the reaction to unknown sources of failure is chaotic. End-to-end pipeline failure events must be correlated manually.

People Process Technology

• Limited knowledge of CT, • Most tests other than • Version management Ad-hoc training build tests are manual • Automated build test • Some Dev/QA • Minimal test version co-ordination management • Painful but repeatable releases

Figure 4: Continuous Testing Maturity Level 2: Continuous Integration

Continuous Testing Maturity Level 3: Continuous Flow

Figure 5 shows the key characteristics of People, Process, and Technology, evident at this maturity level. At this level, test automation is used from end-to-end across the pipeline. Dev and QA teams cooperate to ensure a good level of test coverage is automated. Release standards are using automated testing metrics.

Release standards are using automated metrics and some sophisticated measures correlating application test results performance to business risk.

Typical outcomes include repeatable quality releases with some bottlenecks and inefficiencies. There is sufficient testing to identify the most likely source of failure. Proactive testing results are integrated into one place for analysis.

Continuous Quality and Testing to Accelerate Application Development // 9 People Process Technology

• CT skills and training • E2E CI/CD pipeline, tests • Most tests automated for program visible app, infra, pipeline

• Risk management • Test/release standards • Release metrics use test results • Dev/QA joint plan • Test management

Figure 5: Continuous Testing Maturity Level 3: Continuous Flow

Continuous Testing Automation Maturity Level 4: Continuous Feedback

Figure 6 shows the key characteristics of People, Process, and Technology, evident at this maturity level. At this level, Continuous Testing processes extend from end-to-end across the pipeline. Metrics are set for test automation. The culture includes training and mentoring for test automation. Automation is applied to test environment orchestration and analytics. There is a focus on end-to-end testing rather than spot tests. Automation is applied to testing extensively.

Release standards are using automated metrics and some sophisticated measures correlating application test results performance to business risk.

Typical outcomes include repeatable quality releases with some bottlenecks and inefficiencies. There is sufficient testing to identify the most likely source of failure. Pro-active, testing results are integrated into one place for analysis.

People Process Technology

• Collaboration using shared • E2E performance trend • Test environment test metrics drive test design orchestration

• Goals: SLI/O/As • Focus on removing test • Predictive testing bottlenecks • Mentors and Guilds

Figure 6: Continuous Testing Maturity Level 4: Continuous Feedback

Continuous Quality and Testing to Accelerate Application Development // 10 Continuous Testing Maturity Level 5: Continuous Improvement

Figure 7 shows the key characteristics of People, Process, and Technology, evident at this maturity level. At this level, there is a high level of knowledge and confidence regarding continuous test automation. Dev and QA teams are tightly integrated to optimize knowledge and efficiency. End-to-end tests focus on the end customer experience and more sophisticated risk-based strategies. Test creation strategies leverage automation.

Dev, QA, and Ops teams collaborate using shared testing strategies. End-to-end tests focus on the end customer experience and more sophisticated risk-based strategies.

Typical outcomes include extremely high confidence and satisfaction by all stakeholders and innovation-driven culture. Tests can predict, with a high accuracy, where improvements are needed to avoid failures in the applications, pipelines, and infrastructure components. The test systems are self-correcting, allowing for experimentation. Achievement at this level provides a platform for autonomous continuous improvement strategies in which automation and intelligence drive innovation. “Here is the fix for the thing that will break.”

People Process Technology

• Experimentation • Risk based test design • E2E value stream test analysis, orchestration and • Integrated Dev/QA • Automated test creation execution and test results analysis • E2E user experience focus • Intelligent test creation

Figure 7: Continuous Testing Maturity Level 5: Continuous Improvement

Continuous Quality and Testing to Accelerate Application Development // 11 Continuous Testing Maturity Assessment

While the above five levels of Continuous Testing maturity provide a practical guide for defining maturity against characteristics of People, Process, and Technology, they are not an absolute measure of maturity. Organizations, or specific applications within an organization, may match some of the characteristics for different levels.

The following Continuous Maturity Model Assessment Model (Figure 8) is a useful tool to determine the “best fit” for the maturity of an organization or application within an organization. Matching the people’s characteristics, process, and tech columns that best describe your organization provides you with a quick assessment of your dominant level of testing maturity. You can also use this as a fast way to determine areas most in need of improvement to get to the next level of maturity.

People Process Technology

• Silo team organization • Testing not part of planning • Missing tools to test 1 performance of applications, • No test standards Chaos • Little knowledge of test pipelines and infrastructure automation • Few automated tests • Blame, finger-pointing

• Limited knowledge of • Most tests other than build • Version management 2 testing, Ad-hoc training tests are manual Continuous • Automated build tests Integration • Some Dev/QA co- • Minimal test version ordination management • Painful but repeatable releases

• Test automation skills and • E2E CI/CD pipeline, test • Most tests automated for visible Continuous 3 training program app, infra, pipeline Flow • Risk management • Test/release standards • Release metrics use test results • Dev/QA joint plan • Test management

• Collaboration using shared • E2E performance trends drive • Test environment test metrics orchestration Continuous 4 test design Feedback • Goals: SLI/O/As • Focus on removing test • Predictive test analytics • Mentors and Guilds bottlenecks

• Experimentation • Risk based test design • E2E value stream test Continuous 5 analysis, orchestration and • Integrated Dev/QA • Automated test creation execution Improvement and test results analysis • E2E user experience focus • Intelligent test creation

Figure 8: Continuous Testing Maturity Assessment Model

Continuous Quality and Testing to Accelerate Application Development // 12 How Do You Get Started with DevOps Testing?

A primary key to success with DevOps is continuous test engineering. As this whitepaper has explained, DevOps Continuous Testing is much more than automating tests or integrating a tool into the continuous delivery toolchain. The following 7-step methodical approach can help you realize effective and efficient DevOps Continuous Testing:

1 Skill yourself and your team on DevOps testing. Explore formal training.

2 Create a dedicated role to guide your organization through DevOps testing or engage a DevOps test consultant and appoint a DevOps test architect to conduct an assessment of your current practices and define a roadmap. The solution roadmap should define a test strategy for the product. The strategy will define the test coverage strategy, how test will be created, which test will be conducted at each stage of the continuous delivery pipeline, how test results will be handled, and metrics to monitor testing performance.

3 Select a pilot application.

4 Implement test infrastructure, test frameworks, and tools according to the solution roadmap.

5 Train cross-functional team members on test creation methodologies and tools.

6 After a pilot project with a one product application and team, expand the implementation to more teams and applications.

7 Monitor progress and put in place continuous improvement procedures.

Continuous Quality and Testing to Accelerate Application Development // 13 Continuous Testing in the Cloud

Cloud computing offers flexibility to test software services on the most appropriate environments and, at the proper time, throughout the software development cycle. The most effective software testing relies on the execution of tests that most accurately represent real-world user conditions.

Early in development, basic verification testing of code on a local laptop or SaaS IDE is acceptable. But as soon as code is compiled and merged with that of other developers, validating that the resulting services meet user expectations under various functional, performance, scalability, and security conditions become critical consideration of overall quality and user satisfaction. When utilizing a mature Continuous Integration/Continuous Delivery (CI/CD) pipeline, it is possible to trigger test jobs at any point in the delivery pipeline and automate the deployment and tear-down of appropriate cloud- based test environment to run the tests using the Infrastructure as Code approach.

Using AWS, engineering and QA teams can get seamless self-services access to both the environments and tooling they need to implement Continuous Testing effectively.

Figure 9: Continuous Testing with AWS

Continuous Quality and Testing to Accelerate Application Development // 14 Choosing the right tools for Continuous Testing

A significant pipeline strategy aligns DevOps phases with established practices and easily integrates with your chosen tools. In addition to the people and process elements of your Continuous Testing strategy, you’ll need to ensure that you have the right testing tooling that integrates with your pipeline.

Engineering and QA teams can use DevOps Solutions available in AWS Marketplace to design and deploy secure, scalable modern applications and infrastructure that can accommodate your current and future business needs as application usage grows. AWS Marketplace is a digital catalog with thousands of listings from independent software vendors that enables you to quickly and easily find, test, try, buy, and deploy software to orchestrate your entire DevOps toolchain on AWS.

Flexible consumption Quick and easy Helpful humans to and contract models deployment support you

The next section explains how to perform Continuous Testing across the application delivery pipeline and introduces the most commonly used third-party testing tools to help get the job done.

Continuous Quality and Testing to Accelerate Application Development // 15 On-demand Development and Testing Environments with Infrastructure as Code

One of the most significant benefits to developing applications in the cloud is defining your development, test, and production environments (infrastructure and software configuration) in machine-readable definition files. The files are managed in version control, like your application code, so that you can automate the creation and tear-down of consistent production-like environments for development and testing.

This practice is key to building quality into your applications as quality starts with the code your developers commit. Suppose development teams can work in self-service production-like environments without waiting long periods for operations to prepare them. In that case, developers can write, test, and run code to receive immediate feedback as they work.

Enabling development and QA teams with self-service access to needed environments facilitates frequent code commits, integration, and automated testing required for high-functioning DevOps pipelines. Adopting an Infrastructure as Code approach that is version-controlled and immutable also prevents configuration drift and errors that surface after the release because the development, testing, and production environments are all identical.

Infrastructure as Code (IaC) tools available in AWS Marketplace:

HashiCorp Terraform Chef Automate SaltStack Enterprise Puppet Enterprise Quali’s CloudShell Colony Try it > Try it > Try it > Try it > Try it >

View all IaC tools and learn more >

Continuous Quality and Testing to Accelerate Application Development // 16 The Right Tests at the Right Time

Continuous Integration and Continuous Delivery (CI/CD) Most DevOps activities revolve around a source code repository, and most of them utilize Git as the repo of choice. Continuous Integration orchestrates the builds of code into applications and services. It also automates the triggering of tests and other tasks, including the actual deployment of testing and production environments by integrating with your Infrastructure as Code tools.

CI/CD tools available in AWS Marketplace:

GitLab Armory Harness Circle CI - CI/CD Try it > Try it > Try it > Try it >

CloudBees Products GitHub Enterprise Try it > Try it >

View all CI/CD tools and learn more >

Continuous Quality and Testing to Accelerate Application Development // 17 Quality and Test Management Continuous Testing involves complex of interrelated artifacts and processes that often require their own management and execution frameworks. These solutions keep track of test requirements, specifications, scripts, and results sets. They can also control test execution, role base tasks, test results, and comprehensive quality reporting and analysis. DevOps solutions include integrations with key AWS Developer tools and other CI/CD tools.

Functional Testing Testing the functional inputs and outputs of software and services really picks up when integration and complexity regressions and bugs that will impact users. Cloud-based functional test automation enables teams to build and automate tests of this type. By deploying cloud-based test environments, hardware and network variables can be considered in the tests.

Load Testing, including Performance Testing These important test types measure the user experience and system breaking points as services scale to the needs of any number of users — from tens to millions. To any scale and form different regions, clouds resources instant availability has provided the best platform for driving continuous and on-demand load testing tools.

Mobile Testing Because users expect that every website and application will run on their mobile device, Continuous Testing must include the mobile user experience and mobile scalability, and even localization. With the proliferation of different mobile device types and the two primary mobile operating systems (iOS and Android), testing can be best accomplished with a mix of actual physical devices combined with device emulators that run on cloud servers.

Production Testing Formerly taboo, testing in production — with the proper tools, training, guidelines, and rollback capabilities — has become a relatively standard and accepted approach for organizations focused on end-to-end quality and reliability. Chaos Engineering, which includes randomly shutting down servers, injecting latency, dropping packets, and breaking connections, creates real-world scenarios that can best be replicated on actual production infrastructure. Feature flagging, which allows the testing of new features by toggling on/off for select users, and closely related A/B testing, which presents different capabilities to different users, enables the live testing of services to determine both functionalities as well as user adoption in live production environments.

Continuous Quality and Testing to Accelerate Application Development // 18 Testing tools available in AWS Marketplace:

Quality and Test Management: Load and Performance Testing: Mobile Testing:

Micro Focus ALM Octane Genymotion Cloud Try it > Gatling FrontLine Try it > Try it >

Production Testing:

Tricentis qTest Try it >

Gremlin Micro Focus Try it > Functional Testing: LoadRunner Cloud Try it >

CloudBees Feature Management Try it > Micro Focus UFT One Testable Try it > Try it >

LaunchDarkly View all testing tools and learn more > Try it >

Continuous Quality and Testing to Accelerate Application Development // 19 Get Started Advance your Continuous Testing practice with the right tools and more knowledge on DevOps best practices. To ensure your teams have the best tools for the job, explore and try new solutions in AWS Marketplace.

Key benefits of building your DevOps practice with solutions in AWS Marketplace

Find solutions 46% faster* Adopt new tools 53% faster* Deploy your way, 48% faster* Trust AWS verified technologies Find the market-leading tools you Try leading-edge developer tools Deploy DevOps solutions with Count on tools that are selected need to orchestrate your ideal and simplify procurement with methods that best fit your use for AWS interoperability and are DevOps toolchain. integrated AWS billing. case using containers, SaaS, regularly scanned for security AMIs, APIs, or CloudFormation vulnerabilities. Templates, and more. To get started, visit: https://aws.amazon.com/marketplace/solutions/devops/continuous-testing

Conclusion

The acceleration to DevOps transformed the way applications are developed and delivered and how they are tested. Successful implementation of the blueprint for well-engineered Continuous Testing depends on understanding the stage of your maturity level and implementing the appropriate best practices outlined in this whitepaper.

The first step requires an in-depth assessment across all three dimensions of the model: people, processes, and technology, which will enable you to develop a roadmap for your own version of Continuous Testing.

As you develop your roadmap, consider cloud computing’s flexibility to test software services in the most appropriate environments throughout the development cycle. Continuous Testing in the cloud enables your development and QA teams with self-service access to needed environments.

To implement Continuous Testing effectively, engineering and QA teams can get seamless self-service access to both the environments and the tooling using AWS. The resulting continuous quality management and testing will reduce costs and time-to-market and ensure development teams stay focused on delivering customer value.

*2020 Amazon Web Services Marketplace online survey with 200 IT decision-makers and influencers in the U.S.

Continuous Quality and Testing to Accelerate Application Development // 20 About AWS Marketplace

AWS Marketplace is a curated digital catalog that simplifies software discovery, procurement, provisioning, and management. With AWS Marketplace, customers can also utilize features that speed up product evaluation, improve governance and cost transparency, and enhance control over software spend. AWS Marketplace offers third-party solutions across software, data, and machine learning tools that enable builders to find, and deploy solutions to expedite innovation.

Customers can launch pre-configured solutions in just a few clicks in both Amazon Machine Image (AMI) formats and SaaS subscriptions, with entitlement options such as hourly, monthly, annual, and multi-year contracts.

AWS Marketplace is supported by a global team of solution architects, product specialists, and other experts to help IT teams connect with the tools and resources needed to streamline migration journeys to AWS.

About DevOps Institute

DevOps Institute works to advance the human elements of DevOps. We create a safe and interactive environment where members can network, gain knowledge, grow their careers, support enterprise transformation, and celebrate professional achievements.

We are not just an information source; we connect and enable the global member community to drive human transformation in the digital age.

Contributors: Marc Hornbeek Kanchan Waikar Ambassador, DevOps Institute Senior Partner Solutions Architect, CEO and Principal Consultant AWS at Engineering, DevOps Consulting

Alex Jones Brad Johnson Sr. Digital Demand Generation Manager, Global DevOps Category Leader, AWS Marketplace AWS Marketplace