Continuous API Testing and Monitoring: Best Practices & Buy-In

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide

Evaluate continuous API testing and monitoring tools to significantly improve API quality, security, and reliability

v2.1 Contents

Four Key API Testing Strategies 4 1. Use Dynamic DDT to Check Entire User Flows 4 2. Ensure API Tests Are Created with Domain Expertise 5 3. Invest in an API Testing Tool with a Modern Architecture 5 4. Adopt a New API Metric: E2E Functional Uptime 5

Four Important API Testing Solutions 6 Solution 1: Solve QA/Testing Bottlenecks with Transparent API Testing 6 Solution 2: Monitor APIs With or Without a CI/CD Pipeline 7 Solution 3: Reuse API Tests as Functional Uptime Monitors in Production 8 Solution 4: Create a Single Pane of Performance Analytics 9

API Testing Automation: Evaluation Criteria 11 Features Evaluation Checklist 11 Conclusion 12

About API Fortress 13

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 2 Shift-left API testing automation has been increasingly embraced by developers and QA teams as essential to successful agile software development or a CI/CD pipeline. In theory, automated testing early in the lifecycle should reduce costly late-lifecycle QA/testing bottlenecks. Yet bottlenecks continue to plague developers and testers, and the number of undetected API bugs that go live and reach end-users or hackers remains alarmingly high.

$2.8 trillion was spent in the U.S. due to poor quality software (CISQ)

One of the critical issues behind a failure of testing effectiveness at agile organizations involves the complexity of managing the scope and scale of API testing, that is, in striking the right balance between time-to-market and quality-to-market. Companies need to do more testing in less time, but a major roadblock stands in the way - the diverse teams that share in API success tend to work in silos.

Development, QA/Testing, and DevOps teams may struggle to find consensus on which API metrics should be tracked to set KPIs

A core reason for silos in an agile organization stems from the wide gap in priorities and accountability among the teams that own APIs. While developers may be most concerned about delivering a shippable product on time, QA/testing teams may be more concerned about validating whether the product satisfies the user story. Then DevOps or DevSecOps leaders have uptime and reliability concerns that extend beyond the delivery sprint. Banking, financial services, healthcare, and other types of enterprises that deal with advanced API security and compliance issues run into more complications that divide siloed teams even further.

With the right API testing and monitoring tools, agile organizations can bring insight-driven visibility and collaboration to distributed teams. This is needed to set companies on the right path to transform API testing reliability and performance.

This white paper offers a breakdown of recommended API testing strategies along with insights about using the right API metrics to set up more viable KPIs based on our years of experience at API Fortress in working with several of the world’s most innovative retail, banking, financial services, insurance, healthcare, telecom, education, entertainment, and aeronautics/aviation companies. Additionally, this white paper includes a breakdown of best practices for implementing good API testing and monitoring tools.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 3 Four Key API Testing Strategies

Prior to agile development, CI/CD pipelines and microservices, UI and Unit testing dominated the testing pyramid. And then, the only essential role of API testing was to verify Test Automation the contract and check for a ping round trip. Now, API tests must effectively act as end- to-end tests that validate entire user stories. UI Tests This new set of strategies (along with new metrics) are needed to determine API testing E2E Testing success. The following “Big Four” strategies have been derived from what we have seen in next-generation approaches to API testing for modern systems with contemporary toolchains and CICD flows. We hope these strategies help leaders of development, QA and testing, enterprise architecture, DevOps, and product teams to better quantify the efficiency and efficacy of their current or proof-of-concept API testing tools.

Strategy 1: Use Dynamic DDT to Check Entire User Flows Modern API testing tools must be able to conduct multi-step integration tests across APIs to check on how well API endpoints work together (the “API flow”). In the real world, APIs use data that is not fixed or prebuilt. It follows that API testing tools should create API testing paths that are unpredictable. The best way to do this is to leverage Data-driven Testing (DDT) from active databases or highly variable (fake) testing data. The goal is to create tests that involve multiple steps, including calling an API that can trigger a sequence of unique API calls in an array. In some cases, testing systems include components that convert databases into APIs to be used as these dynamic APIs. Additionally, modern API testing tools should automate OAuth 2.0 flows to avoid disruption to DDT while also validating SSO and multi-factor authorizations.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 4 Strategy 2: Ensure API Tests Are Created with Domain Expertise

While developers can certainly write tests that verify the technical capabilities of the APIs they build, it can be ineffective and/or inefficient for developers to create API tests that also validate the user story. Instead, API tests from developers that simply prove whether an API is working when used correctly can form the basis for more stringent and comprehensive tests written by professional QAs with high knowledge of the problem and solution domains. These holistic tests may include end-to-end integration tests that can check entire flows involving arrays of APIs that may have been built or changed by many different teams. Ultimately, the goal is to leverage one unified test with consistent domain knowledge for proactive and real- time insight about API health throughout constant changes to code and databases.

Strategy 3: Invest in an API Testing Tool with a Modern Architecture Modern API testing tools, in particular, tools that excel at testing APIs in agile development and CI/CD pipelines, must offer a robust set of APIs for plug-and-play integrations with existing and modern DevOps tools. Today, many companies benefit from sending all test results to a best-in-breed analytics platform such as Elastic or Splunk, while delegating notifications to best-in-breed solutions such as PagerDuty or Slack.

Additionally, with modern architectures, API testing tools can evolve along a far more innovative roadmap. If a company is evaluating a number of API testing tools that are divided between unified testing suites that piecemeal together multiple apps/services versus a best- in-breed unified testing platform, the best-in-breed platform is almost always the better choice. Most API owners should avoid vendor lock-in, and future-proof their increasingly agile toolchains for cloud maturity and distributed services.

Strategy 4: Adopt a New API Metric: E2E Functional Uptime On its own, API uptime (an HTTP 200 OK) is an outmoded metric for continuous API quality. Modern API testing tools must go beyond uptime, and simplify validation of the user story by checking the business logic and service layers for problems across functionality, reliability, performance, and compliance as well as potential security vulnerabilities. A holistic verification of these layers allows a good API testing tool to significantly improve on the accuracy and consistency of uptime reporting, particularly, in reducing false-positives and ensuring adequate testing coverage. Testing Centers of Excellence or smaller QA/Testing teams sometimes refer to this new API metric as “Functional Uptime” or “End-to-End (E2E) Uptime.”

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 5 Four Important API Testing Solutions

Before breaking down the best practices of a good API testing tool, this section provides a framework for how the best practices were selected. The focus of the assessment is geared toward solving four key API testing solutions that are critical for any organization moving from a monolithic stack to distributed services and cloud maturity.

Solution 1: Solve QA/Testing Bottlenecks with Transparent API Testing

22 23

Plan Build Test/QA Release / Deploy

Most of the costly and time-consuming bottlenecks that hold up software development happen in the QA/testing stage. Insufficient or poor collaboration between technical and product teams (or line of business owners) with poor or no visibility into end-to-end API testing is often at fault. True end-to-end API testing tools tackle this problem by making it easy for stakeholders of all technical and coding backgrounds to work in parallel. Ultimately, the goal is to minimize the risk of falling short of validating the user story without delaying go-to-market.

Another vital aspect of API testing transparency is to ensure that all teams are clear on the scope of testing coverage. Insufficient attention to this detail may result in high numbers of false-positives, which allow malfunctioning APIs to exist in production environments for prolonged periods of time.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 6 Solution 2: Monitor APIs With or Without a CI/CD Pipeline

Continuous API Testing and Monitoring in a CI Flow

Developers & QA CI/CD Pipeline Centralized API Testing and Debugging

Repo

Command Line Interface Test Eecution esults IE

Successful shift-left API testing automation frees development teams to confidently run a Continuous Integration (CI) flow. Most API testing tools simplify the CI flow by seamlessly integrating with popular CI/CD platforms such as Jenkins, BitBucket, Azure DevOps, Bamboo, TravisCI, GitLab CI, CircleCI and more. The latest API testing tools go further by additionally offering a built-in scheduler. This empowers teams to deploy continuous API tests and monitors without requiring a CI/CD pipeline. Critical advantages of this approach include:

• Faster Builds, Less Overhead: Continuous API testing (monitoring) via a scheduler achieves the same automation goals of a CI/CD pipeline with added capabilities to ensure that development/staging environments are working for teams 24/7; this risk reduction can return many hours back to teams

• Enhanced Testing Consistency and Accuracy: Free from dependence on the CI/CD pipeline, continuous API tests/monitors remain accurate and consistent throughout changes to the code repository and databases

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 7 If API testing automation is an implied part of any CI flow, then it follows that end-to-end API monitoring should be an implied part of any CI flow that must avoid disastrous QA/testing bottlenecks. With proper API monitoring feeding the right (end-to-end) API testing data points to analytics tools, QA/testing or DevOps leaders can properly curate analytics for more useful KPIs to track API performance.

True continuous testing does not exist without end-to-end API monitoring throughout the API lifecycle

Solution 3: Reuse API Tests as Functional Uptime Monitors in Production

Reusing Functional API Tests as Monitors Improves Quality, Security, and Reliability

Knowledge Live APIs

Build

I IT

PRE-PRODUCTION PRODUCTION

• CONTINUOUS • DATA-DRIEN • FUNCTIONAL • END-TO-END Test

API Mocks

Many of the same benefits of shifting API testing left also apply when shifting API testing right (into production environments). Good API testing tools help to significantly reduce the risk of any unforeseen bugs or crashes by making it easy to reuse shift-left tests built with sufficient domain knowledge and coverage as shift-right tests, aka, API monitors. Automated tests in

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 8 production are exposed to the knowns and unknowns that APIs will encounter when live. Production environments typically contain complex systems and data sets that staging does not. Ultimately, by extending proper API testing across the entire API lifecycle, companies can solve QA/testing bottlenecks and ship products faster with greater confidence.

In the best cases, a good API testing tool makes it easy to build continuous API tests or API monitors from a holistic combination of data-driven, functional, load, and end-to-end (integration) tests. With this level of modular reusability, both QA/testing and live monitoring/ SRE teams can test more and test better in less time. Further, good API testing tools should be able to schedule these functional/E2E monitors in development, pre-production, and production environments. In this way, technical teams can verify an API’s technical capabilities in harmony with product/testing teams validating the user story.

Solution 4: Create a Single Pane of Performance Analytics

Most companies already have proven solutions in place for best-in-breed UI and infrastructure monitoring. A good API testing tool makes it easy to extend existing best-in-breed investments with best-in-breed API testing by allowing functional tests to be reused as end-to-end load

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 9 tests that can be scheduled continuously as API performance monitors. Automated API load tests generated in this manner offer built-in domain knowledge along with coverage that stretches across entire user flows to help teams quickly connect the dots about how and why functional and integration problems may have an impact on API performance. Now, performance bugs that may have escaped detection altogether or required costly QA/testing in a fully virtualized environment to find can be detected with a continuous end-to-end API load test.

End-to-end load testing helps companies raise the bar on SLA guarantees for internal and public APIs, accelerating an API-led future

With a good API testing tool, monitoring and SRE teams can instantly upgrade their analytics dashboards with powerful new metrics about API performance under real world conditions.

The Build vs. Buy Decision An alternative to purchasing an API testing and monitoring platform is to build the suite in-house. Traditional API testing platforms may have pushed companies toward DIY solutions due to high costs, a lack of software budget, unique requirements, lack of configurability, and/or problems integrating with an existing toolchain. Building a testing suite requires the same planning and development required of any new product or application . For some companies, they have no choice, but there are important items to consider when choosing to build in-house including:

• Dependent on a few staff members or service providers who know how to fix, customize, and update the suite. This leads to a problem if they choose to leave the organization. • Usually there is a lack of UI and reporting to help understand results. • Documentation to train new users on using the suite rarely exists. • No formal support procedure for teammates. • User management and auditing is rarely implemented, leading to a lack of collaboration. • Building from scratch keeps you always behind the curve in terms of new standards and requirements.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 10 API Testing Automation: Evaluation Criteria

This final section of the white paper offers two grids that capture the key points of this paper, and may help in the pursuit of the right API testing and monitoring tool.

API Testing/Monitoring: Features Evaluation Checklist

API Testing & Monitoring: Features Evaluation Checklist

Y or N Category Feature Score (1-5)

Cloud Installation & Setup On-Premises

Single Setup Integrations (Slack, JIRA, etc...)

Team Setup & User Roles

Flexibility in Methods of Use (GUI, command-line, plugins, etc...)

Test Generation Test Creation Test Editing

Assertion Library

Easy to Learn / Understand

Ability to Create End-to-End Tests

Ease of Creating End-to-End Tests

Reuse Code Snippets Between Tests

Universal Variables

Support for Custom Coding Test Creation - Scripting Scripting Libraries

Data Source Driven Tests Test Data Management Supported Data Formats

Source Control Management Integration with VCS (eg: Github)

Run Tests Against Any Environment Test Execution

Works With Your CI/CD Platform (eg: Jenkins, Bamboo, TFS, etc...)

APIs & Webhooks for Execution Flexibility

Built in Scheduling

WWW.APIFORTRESS.COM

Download PDF Download Spreadsheet

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 11 Conclusion

The API economy has greatly accelerated sweeping changes across many industries, including an even greater dependence on RESTful apps and web services. Like many organizations, your enterprise may be rushing with greater urgency toward digital transformation. Or you may be finalizing strategies to bolster your leading position with cloud and API technologies. Wherever you are in your journey in the API economy, a key takeaway from this white paper is to make sure that you know what makes modern API testing so different from traditional API testing as well as UI, unit, browser, and infrastructure testing/monitoring. By knowing why modern API testing must effectively become end-to-end testing, you can establish the right API metrics and KPIs for API testing success.

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 12 About Us

API Fortress is the only API testing platform built from the ground up for collaborative API testing in modern architectures. Our customers integrate us easily into their SDLC toolchains for shift-left testing and functional uptime monitoring of internal APIs and external/public APIs. The cloud-native API Fortress platform can be deployed via an on-premises container or as a SaaS with low total cost of ownership. Many API Fortress customers build or improve upon single pane of analytics across APIs, UI, and infrastructure via DataDog, Sauce Labs, MuleSoft, Apigee, Kong, Jenkins, Splunk, BitBucket, Azure DevOps, TIBCO Mashery, WSO2, Oracle Cloud, Axway, PagerDuty, xMatters, and more.

Want to Learn More?

Visit www.APIFortress.com to chat live, sign up for a Free Trial, or Schedule a Demo with a sales or solutions engineer.

Start a Free Trial Schedule a Demo

Continuous API Testing and Monitoring: Best Practices & Buy-in Guide 13