Continuous Quality and Testing to Accelerate Application Development How to assess your current testing maturity level and practice continuous testing for DevOps Continuous Quality and Testing to Accelerate Application Development // 1 Table of Contents 03 Introduction 04 Why Is Continuous Quality and Testing Maturity Important to DevOps? 05 Continuous Testing Engineers Quality into DevOps 07 Best Practices for Well- Engineered Continuous Testing 08 Continuous Testing Maturity Levels Level 1: Chaos Level 2: Continuous Integration Level 3: Continuous Flow Level 4: Continuous Feedback Level 5: Continuous Improvement 12 Continuous Testing Maturity Assessment 13 How to Get Started with DevOps Testing? 14 Continuous Testing in the Cloud Choosing the right tools for Continuous Testing On-demand Development and Testing Environments with Infrastructure as Code The Right Tests at the Right Time 20 Get Started 20 Conclusion 21 About AWS Marketplace and DevOps Institute 21 Contributors Introduction A successful DevOps implementation reduces the bottlenecks related to testing. These bottlenecks include finding and setting up test environments, test configurations, and test results implementation. These issues are not industry specific. They can be experienced in manufacturing, service businesses, and governments alike. They can be reduced by having a thorough understanding and a disciplined, mature implementation of Continuous Testing and related recommended engineering practices. The best place to start addressing these challenges is having a good understanding of what Continuous Testing is. Marc Hornbeek, the author of Engineering DevOps, describes it as: “A quality assessment strategy in which most tests are automated and integrated as a core and essential part of DevOps. Continuous Testing is much more than simply ‘automating tests.’” In this whitepaper, we’ll address the best practices you can adopt for implementing Continuous Quality and Testing on the AWS Cloud environment in the context of the DevOps model. We will explore: • How you can effectively implement Continuous Testing and enable your engineering and quality assurance (QA) teams to get seamless self-service access to both the environments and tooling using AWS • How to develop a roadmap for Continuous Testing implementation based on maturity and assessment models • How to get started with DevOps testing These suggested guidelines can help you reduce costs and time-to-market and allow your developer teams to stay focused on delivering value. Why Is Continuous Quality and Testing Maturity Important to DevOps? The acceleration in DevOps adoption has fundamentally transformed how testing is done. Before the advent of DevOps, most quality testing happened at the end of the cycle. Now, testing increasingly takes place in parallel with development and is engineered for quality. A comprehensive set of tests are, in aggregate, a prescription of expected behaviors for applications within the production environments. The expected behaviors are more than application features and functions. Non-functional requirements, such as performance, maintainability, and reliability, are also essential. Continuous tests are designed to cover both functional and Non-Functional Requirements (NFRs). The capabilities needed to operate an application in each of its deployment environments are part of Continuous Testing. Although Continuous Integration adoption is becoming ubiquitous – with 88% of enterprise software companies adopting it in 2020, test suites are still plagued by a growing backlog of failing tests. Failures in integration and automated tests cause bottlenecks in the development pipeline and substantially increase engineering costs. When engineered well, Continuous Testing can provide impressive benefits, including reduced time-to-market, improved quality, reduced cost, improved innovation, and team satisfaction. According to a May 2020 research report published by UNDO1: 88% 35% of enterprise software companies adopting Continuous of respondents say they deploy code changes hourly. Integration in 2020 compared to 70% in 2015. 26% $61billion of developer time is spent an average of 13 hours reproducing to $1.2 trillion annually total salaries spent in enterprise value and fixing falling tests, which equals 620 million developer lost for shareholders per year. hours a year wasted on debugging software failures. 1The Business Value of Optimized Pipeline – UNDO 2020 Continuous Quality and Testing to Accelerate Application Development // 4 Continuous Testing Engineers Quality into DevOps Continuous test engineering requires a complete re-think of software QA test strategies, so they align with Dev-through-Ops pipeline stages. Fortunately, best practices for DevOps testing are available and can be learned and applied to any product or service. Five tenets underlie the best practices for DevOps test best practices are indicated in Figure 1 and explained below. Fail often Run tests frequently and with many different conditions Fail Early Arrange the tests so that the most likely problems are found Relevance in the earliest possible stage in Focus on the most important the DevOps pipeline tests and results Shift Left Test Fast Conduct each test as early in the Arrange tests to run pipeline as possible in quick cycles Figure 1: Continuous Testing Tenets Continuous Quality and Testing to Accelerate Application Development // 5 The CD pipeline starts with the release of candidate artifacts delivered from the CI pipeline into an artifact repository. The CD pipeline is automated, and artifacts are deployed to a pre-production staging environment for system and user acceptance testing. These tests are automated as much as possible to minimize processing delays and bottlenecks. Once the tests pass, the artifacts are assessed with release policies to win approval for deployment. The release acceptance criteria are often determined by a Change Approval Board (CAB) and implemented as codified policies (if possible). Once approved, the artifacts are deployed to production according to a deployment schedule. To minimize risk, the deployments are conducted using the safe Green/Blue and Dark Launch methods. Once deployed, additional post-deployment tests may be used to evaluate software options using A/B testing and Canary testing methods. Figure 2: DevOps Continuous Testing Blueprint Testing, an essential portion of DevOps, is responsible for continuous assessment of incremental changes and is part of establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, reliably, and safely. DevOps Testing – the Primary Key to DevOps and Continuous Delivery” by Marc Hornbeek Continuous Quality and Testing to Accelerate Application Development // 6 Best Practices for Well-Engineered Continuous Testing Continuous Testing concepts must be mastered. Every test that can be automated should be automated. Tests should be executed in production-equivalent test environments. The results must be assessed in each stage of the end-to-end pipeline to determine if the artifacts are acceptable or need remediation before promotion. Culture for Continuous Testing is critical. Quality is everyone’s responsibility. Leaders need to sponsor, inspire, fund, and motivate cross-functional teams to collaborate, learn, implement, and improve Continuous Testing practices. Test strategies for Continuous Testing include specific objectives. Automated and even manual tests are triggered at each pipeline stage. Elastic cloud test environments are automatically orchestrated to match testing requirements of each change. Tests are shifted-left an accelerated as early as possible to find customer- relevant verdicts fast and early. The use of the A/B feature flag, Blue-Green, and Canary deployments strategies help validate user alternatives and performance safety in production. Test Frameworks and Tools are selected for integration and performance. A test framework acts as a “backbone” for integrating and automating tasks including test plans, all types of test tools, test artifact version management, test resources, test data, test schedules, test environment orchestration, test results, test reports, and test results analysis. Test Planning includes test strategies that describe test requirements, management policies, resources, topologies, and priorities for each product module to meet test strategy requirements. Test plans define test cases and priorities for each product module to meet test strategy requirements. Test Automation is used extensively. Automated tests are created per requirements described in test strategies and plans, using a test automation creation strategy such as TDD. Tests and test tools are automated, orchestrated, operated, and results analyzed through an API by test cases scheduled by a framework. Test Management and Analysis. Manage resources (financial, labor, schedules, and flexible scaling) to meet specific CT quality, delivery, and response time goals determined by key stakeholders. Continuous Quality and Testing to Accelerate Application Development // 7 Continuous Testing Maturity Levels Many organizations have not achieved the level of continuous test maturity described above. For an organization to mature its continuous test capabilities, it needs to know its current state of maturity and the target of its future state. There is no standard continuous test maturity model. A useful way to approach it is to apply a five- level maturity model adapted