Issue 2 Evolving from Automated to Continuous Testing for DevOps As agile development practices mature and There are many nuances associated with the 1 DevOps principles begin to infiltrate our transformation from automated testing to Evolving from Automated to corporate cultures, organizations realize the Continuous Testing. In this introduction, we’ll Continuous Testing for DevOps distinct opportunity to accelerate software focus on three key distinctions: delivery. However, when you speed up 2 Research from Gartner: any process, immature practice areas and • Aligning “test” with business risk Market Guide for API Testing and roadblocks become much more pronounced. Service Virtualization It’s the difference between driving over a • Ubiquitous access to complete test speed bump at 5 MPH versus 50 MPH … environments 8 at 50 MPH, that speed bump is going to be Top 5 Continuous Testing quite jarring. • Extreme test automation at the API/ Questions from Gartner message layer Conferences Accelerating any business process will expose systemic constraints that shackle Aligning Test and Business Risk 9 the entire organization to its slowest moving The most fundamental shift required in Parasoft Continuous Testing component. In the case of the accelerated Accelerates Testing and moving from automated to continuous is SDLC, testing has become the most significant Improves Quality aligning “test” with business risk. Especially barrier to taking full advantage of iterative with DevOps and Continuous Delivery, 11 approaches to software development. releasing with both speed and confidence About Parasoft For organizations to leverage these requires having immediate feedback on the transformative development strategies, they business risks associated with a software must shift from test automation to Continuous release candidate. Given the rising cost and Testing. Drawing a distinction between test impact of software failures, you can’t afford automation and Continuous Testing may to unleash a release that could disrupt the seem like an exercise in semantics, but the existing user experience or introduce new gap between automating functional tests features that expose the organization to new and executing a Continuous Testing process security, reliability, or compliance risks. To is substantial. prevent this, the organization needs to extend from validating bottom- Extreme Test Automation at the API/Message Layer up requirements to assessing the system requirements associated Testing at the API/message layer (services, message queues, with overarching business goals. database abstraction layers, etc.) offers several distinct advantages for enabling Continuous Testing at the speed of DevOps: Ubiquitous Access to Complete Test Environments One of the biggest constraints associated with exercising meaningful • Stability: While GUI tests often fail due to inconsequential tests is accessing a complete test environment–including the myriad application changes, a failure at the API/message level typically dependent systems that the application under test (AUT) interacts signals a fundamental flaw in the application logic–something with. Given the composite nature of today’s applications, it is nearly likely to impact the core user experience. If you’re configuring a test impossible to stage a complete test environment. This is where suite failure to serve as a “gate” along the automated deployment Service Virtualization comes into play. Service Virtualization enables pipeline, it’s important to ensure that every failure indicates a truly you to emulate the behavior of specific components in heterogeneous show-stopping problem. component-based applications such as API-driven applications, cloud-based applications, and service-oriented architectures. By • Speed: Traditional methods of testing, which rely heavily on simulating the AUT’s interactions with the missing or unavailable manual testing and automated GUI tests that require frequent dependencies, Service Virtualization helps you ensure that data, updating, cannot keep pace with the speed required for DevOps. performance, and behavior is consistent across the various test runs. Testing is delayed until the GUI is available, which is typically late Moreover, it also helps you “shift left” testing so it can begin much in the process. Moreover, GUI tests are notoriously brittle and earlier in each iteration and expose defects when they’re fastest and require significant updating with each application modification. easiest to fix. API tests can be defined as soon as the service description (e.g., Swagger or RAML) is available, can be executed much earlier in As a general rule, you should be testing against the most production- the implementation process than GUI tests, and require minimal like environment that you can access …if not in production. maintenance. However, this typically presents a sizable challenge in terms of cost, security, and privacy. Using simulation technologies such as Service • Accurate risk assessment: In modern applications, the Virtualization allows you to bypass the constraints associated with the functionality exposed at the GUI layer is just the tip of the iceberg. dependent systems outside of your control in order to run meaningful The core of the application logic is controlled by the API/message end-to-end tests. layer. Without exhaustive testing of critical user transactions at the API/message layer, it’s hard to rest assured that today’s highly- distributed systems truly function as expected. Source: Parasoft Evolving from Automated to Continuous Testing for DevOps is published by Parasoft. Editorial content supplied by Parasoft is independent of Gartner analysis. All Gartner research is used with Gartner’s permission, and was originally published as part of Gartner’s syndicated research service available to all entitled Gartner clients. © 2016 Gartner, Inc. and/or its affiliates. All rights reserved. The use of Gartner research in this publication does not indicate Gartner’s endorsement of Parasoft’s products and/or strategies. Reproduction or distribution of this publication in any form without Gartner’s prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. The opinions expressed herein are subject to change without notice. Although Gartner research may include a discussion of related legal issues, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner is a public company, and its shareholders may include firms and funds that have financial interests in entities covered in Gartner research. Gartner’s Board of Directors may include senior managers of these firms or funds. Gartner research is produced independently by its research organization without input or influence from these firms, funds or their managers. For further information on the independence and integrity of Gartner research, see “Guiding Principles on Independence and Objectivity” on its website, http://www.gartner.com/technology/about/ombudsman/omb_guide2.jsp. 2 3 Research from Gartner: Market Guide for API Testing and Service Virtualization Demand for service API testing and Recommendations offering solutions for automating the testing virtualization solutions will quickly Heads of development and testing should: functionality and/or performance for APIs increase with faster delivery of new or Web services and solutions, to simulate business functionality to more complex • Evaluate your current software or “virtualize” interdependent components. IT environments. This research highlights development and delivery processes, Rather than driving the user interface as important market trends and attributes to identifying the biggest issues and most automation tools do, these tools test help heads of development and testing challenges from process and skills the system-to-system interactions driving adopt the right solution. perspectives. This will lead toward an the APIs and services. Virtualization is a effective solution and guide the required technique that expands on the concept Key Findings skills to develop or acquire. of object mocking to simulate the behavior • The need for API testing and service of services by adding capabilities such as virtualization is becoming increasingly • Involve roles from different functions timing responses, error handling and the important as IT organizations develop and share ownership across several ability to simulate unavailable resources. bimodal capability to support mobile, competency leaders in adopting This reduces system costs and allows a digital business transformation initiatives technologies and tools. This will help deeper set of tests to be run by playing with and the Internet of Things (IoT). identify key obstacles to overcome and service parameters. provide work-arounds so the project can • The rapid adoption of agile and SOA- move forward smoothly. Service virtualization was first introduced based solutions to support business to the market by ITKO (acquired by CA agility demands that development and • Develop a plan for acquiring technologies Technologies) in 2007, and early entrants test organizations change approaches and tools that will enable much earlier have been the most dominant in the market. and tools to drive a broader, more testing in the development life cycle, accurate view of complex system quality.