WHITE PAPER AUTONOMOUS API TESTING THROUGH SCRIPT-LESS AUTOMATION

Shefali Garg Rajiv Ramanjani Rakesh Ramakrishna Autonomous API Testing through Script-less Automation

Next Generation Testing Model: The Autonomous API Testing Solution This white paper explores the possibilities for how concepts like Data Analytics and Machine Learning can be leveraged to move towards autonomous and script-less API testing. We will discuss challenges with the current Conventional Testing model, and explore how innovations such as Script-less Automation, the Automation Framework Agnostic model, Data Analytics, Artificial Intelligence (AI) and Machine Learning (ML) can be consolidated into a single new testing model/approach – the next generation Autonomous API Testing Solution.

AI and ML modules have evolved to assimilate incredible amounts of data that can accumulate, and use this data to dynamically identify key relationships and patterns that could evade human detection during testing.

Delay detection of crtical defects

Incomplete Repeat test coverage

Challenges of Conventional Last minute Flaky tests requirement Software changes Testing

Large density Meet of production aggressive bugs QA Need for continuous innovation

2 Autonomous API Testing through Script-less Automation

Following are examples of the underlying challenges Opportunities with Autonomous with conventional models: API Testing Solution

● Manual Efforts in Test Design Data Analytics can provide the required feedback loop to product The introduction of each new requirement or feature enhancement development which will enable the model to learn and adapt in real in a product suite under test impacts the overall API Test Design. time. Intelligent analytics solutions are now able to translate This leads to labor-intensive manual efforts in areas such as test exhaustive test results into actionable information for future script creation, maintenance, and subsequent test improvement. Machine Learning algorithms can utilize current and design reviews. previous validation results to identify potential causes for failure and patterns which are beyond the reach of conventional testing models. ● Incomplete Test Coverage Test coverage is complex to create and maintain. When manually The ever-evolving concepts and advancements of Machine Learning identifying the test conditions for a given API product suite, there is empower us to realize an Autonomous API Testing Solution which: a chance that flaky test cases may be created or important business flows missed. There’s also the need to understand ● Operates independent from human intervention and establish relationships among the test cases, ensuring that all possible business flows for a given product suite are covered. ● Governs and sustains continuous test creation, execution, reporting and self-learning from the data ● Focused Testing & Test Suite Optimization Today’s agile development methodologies focus on reducing time to market for product releases and enhancements; this rapid pace poses a challenge for Test teams to meet increasingly aggressive delivery milestones. An optimized testing model must have the capability to analyze test set runs, prioritize business flows, enable focused testing for a given API product change/new feature, and build the corresponding focused/optimized test suites. Focused testing offers the benefit of alleviating the need to run the full / large / lengthy regression testing suite for every change.

● Model The current Conventional API Test Model is too tightly coupled to a selected Test Automation Framework and its corresponding test scripts. This leads to considerable efforts for maintaining the test scripts over time, and may add to license costs as well.

3 Autonomous API Testing through Script-less Automation

Transformation Progression to Autonomous API Testing The transformation to Autonomous API Testing is consolidated into phases shown below:

Manual Testing Automated Testing Test Accelerators Autonomous Testing

● Requirements Analysis ● Preparation of Test ● Automated Test Script ● Script-less Automation Automation Scripts Generation ● Test Scenarios ● Automation Framework ● Data-independent Tests ● Dynamic Input Requests Agnostic Approach ● Test Case Preparation ● Script Reviews ● Dynamic Test Data ● Comprehensive Test Coverage ● Test Execution ● Change Detection ● Data Analytics (Existing Tests) ● Smart Testing ● Automated Defect Reporting ● Continuous Learning

Script-less test automation does not depend on coding or The automated solution provides higher test coverage by utilizing programming skills in any way; the test cases are automated, the feedback provided by the AI-ML module for each set of test run focused, structured, and customizable. results. Hence the system actually learns from each test run and accordingly alters the test suite which is executed in the next test The inputs to this solution include design documentation captured run to optimize the test coverage. Machine Learning and Data in Swagger, WSDL etc. which are available as part of standard Analytics concepts are embedded into the Autonomous Testing product development models. These inputs act as single trigger Solution to provide smart testing capability wherein the solution point for the solution to operate. The solution automates the test runs in self-execution and self-optimization mode. case design and test execution phase, thus eliminating the manual effort, and increases the overall test coverage for the product. This next generation solution is built to utilize a continuous learning Further, the data generated from this extensive testing is analyzed model. The solution can analyze test set runs and project actionable to generate patterns and clusters which further refines the test test set patterns which yield a deeper understanding of failures and design and execution components of the autonomous solution. their impact on the product, and enables an appropriate prioritization of business flows to enable focused testing for a given product The Autonomous API Testing Solution is designed to be agnostic change/new feature. to the Test Automation Framework and thereby is not tied to be operational with any specific automation framework. This enables retention of the automation framework that has already been chosen and best suited for the specific need.

4 Autonomous API Testing through Script-less Automation

Benefits of Autonomous Testing

● Decrease Costs – Script-less Automation: The solution provides savings on product development costs by eliminating the manual efforts involved in test script creation, test script maintenance and test execution. – Automation Framework Agnostic: This solution is not tied to a specific Automation Framework and is designed to work with any automation framework of choice. This eliminates the cost involved in purchasing new licensed test automation tools.

● Increase Quality – Testing Coverage: 100% test scenario/flow coverage is achieved to ensure the API product suite under test is delivered with higher quality and far fewer defects. – Smart Testing: The solution learns from each test run and accordingly alters the test suite for the next test run, including elimination of flaky tests.

● Increase Velocity and Predictability – Adaptability and Throughput: The solution is capable of dynamically adapting to product updates by implementing a more comprehensive mapping of business flows and their tests. This enables quicker creation/selection and a more focused test suite designed to validate a specific change to the product, thus improving the test velocity and risk evaluation of any planned change/fix.

● Value Add for Product Development and DevOps The different functions within a value stream can also harness this solution. For example: – Design: Helps with identifying the impact of a requirement on the associated business flows and their corresponding tests. – Development: Establishes and provides clear relationships and dependencies between business flows and tests which can be used to avoid introducing regression issues during a code change.

● Modular Solution The architecture of the Autonomous Testing Solution is designed in a way that each component is modular and independent. Each component can be run individually or as a single integrated solution.

5 Autonomous API Testing through Script-less Automation

Architecture and Building Blocks Following is an overview of how the autonomous concepts and approach are being implemented for application programming interface (API) validation. The architecture of the Autonomous API Testing Solution is designed so that each component is modular and independent. Each component can be run individually and as an integrated solution.

Swagger/WSDL Utility Test Data Generator Test Data Accelerator TA Combination TA TA Generator Business Flows Test Data

Combinations

Single Service Combinations

Multi Service Combinations

Combination Optimizer DATABASE DRIVER

Data Analytics

TEST RESULTS

Combination Generator Database The Combination Generator is a program written in Python language All the execution results from the Driver are continuously stored in a to provide complete test coverage, generating all combinations based MySQL database, including the test results data corresponding to on the input design documents. The combinations are generated as a each business flow and for each individual API. Invoking a Business single service as well as multiple service levels covering end-to-end Flow may involve more than one API, and the response of each API business flows with all possible combinations of data. The within the business flow is captured and stored in the database and Combination Generator uses the individual API specifications from used by the Data Analytics module. the Swagger utility and the Business Flows to map and create all the possible combinations. Data Analytics The plethora of data getting stocked in the database is used for mining API Business Flows useful information and generating reports. Machine Learning is used to Input from API Business Flows defines the API composition of all cluster the data and provide an analysis that will help identify critical business flows identified within a product, the sequence of the business flows, predict the impact of changes or failures, and better individual APIs in each of the flows, and the metadata that is understand the relationship among business flows. exchanged between the APIs in every given business flow. Combination Optimizer Driver The Combination Optimizer makes the whole solution more robust An integral part of the solution, the Driver is written in C# and runs the and thus enables the Autonomous API Testing Solution to optimize combinations in the Test environment setup and stores the results in the overall test coverage. During the first test run the Combination the database. The API Business Flows enable the Driver to understand Generator will generate all the possible business flows/test conditions. the flow of data between the individual APIs within a combination In this initial phase the Combination Optimizer consumes the during the execution flow. The Test Data Generator utility responsible combinations and their corresponding test results to learn about for dynamic data creation processes each input combination/ request which business flows are valid and invalid. being fed to the Driver. As the Combination Optimizer matures it works independently without further inputs from the combinations, and will have the capability to select more rational combinations and design combination sets based on a given priority or risk-based input. This results in further optimization of the test coverage/combinations that are executed in the Driver. 6 About FIS

FIS is a leading provider of technology solutions for merchants, banks and capital markets firms globally. Our more than 55,000 people are dedicated to advancing the way the world pays, banks and invests by applying our scale, deep expertise and data-driven insights. We help our clients use technology in innovative ways to solve business- critical challenges and deliver superior experiences for their customers. Headquartered in Jacksonville, Florida, FIS is a Fortune 500® company and is a member of Standard & Poor’s 500® Index.

www.fisglobal.com [email protected]

twitter.com/fisglobal linkedin.com/company/fis

©2021 FIS FIS and the FIS logo are trademarks or registered trademarks of FIS or its subsidiaries in the U.S. and/or other countries. Other parties’ marks are the property of their respective owners. 1157115_US