API Testing: Challenges and Best Practices

Modern composite applications are aggregating and consuming private, partner, and public APIs at a staggering pace in order to achieve business goals. ProgrammableWeb reports that there are over 10,000 APIs published today— which is well over twice as many than there were two years ago.1 Beyond these publically-exposed APIs, the number of private APIs is estimated to be in the millions. As the risks associated with application failure have broader business impacts, the integrity of the APIs you produce and consume is now more important than ever. An API that fails to deliver the expected level of security, reliability, and performance can thus have tremendous business impacts—both to the organization producing it and to those consuming it. If you are integrating exposed APIs into your business critical transactions, you are essentially assuming the risks associated with that API's integrity (or lack thereof). As the number of external APIs integrated into a business process increases, so do the potential points of failure. The business impact of any application failure is the same, regardless of whether the fault lies within the components you developed or the APIs you are consuming. Finger pointing does little to foster customer satisfaction and brand loyalty. If you are exposing an API, the assumption is that it will work as described. Once the organizations consuming that API integrate this exposed functionality into their own applications, API failure jeopardizes the transactions that now depend on this functionality. If your API is popular, you can guarantee that a glitch will make the headlines. The more secure, reliable, and dependable your API, the better the chance of consumption and the greater the potential for business expansion. If you’re providing a questionable interface and there are viable alternatives to your API, you’re likely to lose business since switching costs associated with API integration are so low.

4 Integrity Challenges with the API Economy In today's API economy, your ability to protect your brand hinges upon your awareness of (and response to) the following challenges…

1. Broader Attack Surface Area Simply exposing an API through an internal infrastructure undeniably increases an application’s attack surface area from a security perspective. It could be vulnerable to API-level attacks (injections, payload-based attacks, etc.) as well as exploits that take advantage of ineffective authentication, encryption, and access control. The challenge of broader attack surface area is compounded if the API is hosted by a public cloud service. In traditional computing, the producer is aware of (and has full control over) the parameters of the network security. With cloud services, this level of control is significantly diminished. If you’re now leveraging a third-party set of services, the onus is upon you to ensure that your APIs will provide the level of security that your organization expects.

1 http://blog.programmableweb.com/2013/09/23/programmablewebs-directory-hits-10000-apis-and-counting/

1

2. Elevated Potential for Unexpected Misuse Considering the range and number of people who will have access to published APIs, it’s virtually inevitable that they will be exercised in a number of unexpected ways: both by people innocently using them in ways the producer never anticipated and by attackers maliciously trying to exploit them. In the early days of SOA—when services were exposed internally through controlled networks—you could be fairly certain that your services would be used by colleagues or partners who were familiar with their intended uses cases. Now, when an API is exposed to the public, the producer surrenders all control and certainty over how those APIs are consumed.

3. Exceptionally Unpredictable Demand APIs typically need to meet established performance SLAs; however, validating performance vs. SLAs is complicated because it’s so difficult to predict how and when the API might be accessed. Thus, it’s important to validate performance SLAs against a broad range of performance scenarios, including the sudden surges that could occur if an API garners unexpected attention. Moreover, if you’re testing an API that interacts with additional layers of services, the potential for variable or unacceptable performance increases exponentially. This makes it both more critical — and more challenging—to execute a robust set of performance testing scenarios and pinpoint whether the system under test satisfies expectations.

4. API Consumers Need Test Environments To promote widespread adoption of APIs, it’s often desirable to provide API consumers test environments (a.k.a. sandboxes) that enable them to develop and test against exposed services with zero impact on the production system. Such test environments are also critical when API producers want to jumpstart adoption by allowing integration to begin before their API is actually implemented or before new features are fully completed. Another key driver for providing a test environment is to give API consumers easy access to the broad range of behavior, data, and performance profiles they might want to leverage for testing their integration with that API.

5 API Testing “Must Haves” Ensuring that APIs are delivering the necessary level of security, reliability, and performance that’s vital to success in today’s API ecosystem inevitably involves developing, continuously executing, and religiously maintaining a broad array of complex tests. Following are several key API testing “must haves” that will help you achieve those goals in light of the above challenges.

1. Intelligent Test Creation and Automated Validation With APIs, testing a broad range of conditions and corner cases is critical, so automation comes to the forefront. The creation and execution of simple automated tests with limited or manual validation might have sufficed for internal given web services that were used internally (e.g., via SOA), but more sophisticated and extensive automation is required to be confident that APIs are robust enough to meet business expectations. You need a level of automation that gives you a comprehensive set of functional test cases that can be repeated in a systematic manner.

2

Recommended capabilities for this goal include an intuitive interface for automating complex scenarios across the messaging layer, ESBs, databases, and mainframes:

 Defining automated test scenarios across the broad range of protocols and message types used in APIs: REST, WADL, JSON, MQ, JMS, EDI, fixed-length messages, etc.

 Automating rich multilayer validation across multiple endpoints involved in end-to-end test scenarios.

 Parameterizing test messages, validations, and configurations from data sources, values extracted from test scenarios, or variables.

 Defining sophisticated test flow logic without requiring scripting.

 Visualizing how messages and events flow through distributed architectures as tests execute. These are all capabilities that should—or at least could—have been applied to web service testing for SOA. In fact, most of these capabilities were invented, tested, and refined in the context of SOA testing. APIs—with their extreme exposure and myriad opportunities for misuse— brings us to the tipping point that makes these automated testing and validation capabilities a “must have” for organizations serious about delivering APIs that satisfy user needs and expectations.

2. Change Management for Test Assets and Environments Continuously evolving APIs helps organizations stay a step ahead of the competition while responding to business demands. Yet, this frequent change presents significant quality risks if the automated test suite fails to keep pace with the evolving API. A system for fast, easy, and accurate updating of test assets is critical for keeping test assets in sync with the changing API. If you can automatically assess the impact of changes to existing tests and then quickly update existing tests (or create new ones) in response to the identified change impacts, you can vastly reduce the amount of time required to ensure that your tests don’t fail due to expected changes…or overlook critical new functionality.

3. Service Virtualization for Simulated Test Environments Service Virtualization technology creates simulated test environments that provide anytime, anywhere access to the behavior of dependent resources that are unavailable, difficult to access, or difficult to configure for development or testing. “Dependent resources” might include mainframes, mobile app front-ends, databases, web services, third-party applications, or other systems that are out of your team’s direct control. Service virtualization can be used in conjunction with hardware/OS virtualization to access the environments you need to test earlier, faster, or more completely. In the context of API testing, service virtualization can be applied in two key ways:

 To simulate access to the dependent resource behavior (e.g., from a mobile app, database, legacy system, or third-party service) that you need in order to thoroughly validate your API.

3

 To simulate the behavior of your APIs, creating a test environment that API consumers can develop and test against without impacting your production environment—or to enable development and testing to begin before APIs are completed.

4. Extensive Performance Testing—Ideally, with Service Virtualization Due to the highly-exposed nature of APIs, there’s a high potential for unpredictable and often volatile traffic volumes. To determine whether your API will satisfy SLAs in the event of the erratic or surging demand that APIs commonly face, it’s essential to ramp up the scope of performance testing. You can use service virtualization (covered above) to create simulated test environments that help you test against different performance scenarios that would otherwise be difficult to create in the test environment. For instance, you can easily set performance conditions (e.g., timing, latency, delay) to emulate peak, expected, and slow performance—perhaps to help you plan for cloud bursts or determine how the API might respond when someone is accessing it from China. You can also configure various error and failure conditions that are difficult to reproduce or replicate with real systems— for instance, if your APIs rely on Amazon Web Services, you can easily simulate a scenario where AWS is down. This ability to rapidly configure a broad range of conditions in dependent systems is essential for determining if your APIs provide reasonable responses—or at least fail gracefully—under exceptional conditions. One final way that adopting service virtualization helps performance testing: you can “virtualize” any connections to third-party systems, reliably eliminating the risk that your stress tests might impact services you aren’t permitted (or budgeted) to barrage with test messages.

5. Extensive —Ideally, with Service Virtualization Considering APIs’ increased attack surface area, a multi-faceted security testing strategy is essential for ensuring that development has built the appropriate level of security into your application. This includes:

 Executing complex authentication, encryption, and access control test scenarios.

 Generating a broad range of penetration attack scenarios involving parameter , injections, large payloads, etc.

 Running penetration attack scenarios against your existing functional test scenarios.

 Monitoring the back-end during test execution in order to determine whether security is actually compromised. In addition, if you’re adopting service virtualization (covered above) you can leverage it to take your security testing to the next level:

 It provides rapid ways to emulate attack scenarios as well as emulate different security behaviors of dependencies. This lets you derive more value from your existing functional test scenarios (since you can run them vs. different security scenarios that would otherwise be difficult to configure and unfeasible to test against).

 It enables extensive security testing to be performed without a security expert. Existing test scenarios can be easily executed against a broad set of preconfigured security scenarios.

 It helps you isolate and zero in on your APIs response to various attack scenarios and different security behaviors of dependencies.

4

About Parasoft For 25 years, Parasoft has researched and developed software solutions that help organizations define and deliver defect-free software efficiently. By integrating Development Testing, cloud/API testing, and service virtualization, we reduce the time, effort, and cost of delivering secure, reliable, and compliant software. Parasoft's enterprise and embedded development solutions are the industry's most comprehensive—including static analysis, , requirements traceability, functional & load testing, dev/test environment management, and more. The majority of Fortune 500 companies rely on Parasoft in order to produce top-quality software consistently and efficiently. For more information, visit the Parasoft web site and ALM Best Practices blog.

Contacting Parasoft USA Phone: (888) 305-0041 Email: [email protected] NORDICS Phone: +31-70-3922000 Email: [email protected] GERMANY Phone: +49 731 880309-0 Email: [email protected] POLAND Phone: +48 12 290 91 01 Email: [email protected] UK Phone: +44 (0)208 263 6005 Email: [email protected] FRANCE Phone: (33 1) 64 89 26 00, Email: [email protected] ITALY Phone: (+39) 06 96 03 86 74 Email: [email protected]

OTHER See http://www.parasoft.com/contacts

Author Information This paper was written by:

 Wayne Ariola ([email protected]), VP of Strategy at Parasoft  Cynthia Dunlop ([email protected]), Lead Technical Writer at Parasoft

© 2013 Parasoft Corporation All rights reserved. Parasoft and all Parasoft products and services listed within are trademarks or registered trademarks of Parasoft Corporation. All other products, services, and companies are trademarks, registered trademarks, or servicemarks of their respective holders in the US and/or other countries.

5 Parasoft API Integrity

Modern composite applications are aggregating and consuming private, partner, and public APIs at a staggering pace in order to achieve business goals. As applications grow increasingly interdependent, the security, functionality, and performance of the composite “mashup” is only as strong as its weakest link. The success of an end-to-end transaction depends upon all the parts working flawlessly all the time; even small glitches from a popular API can singlehandedly choke thousands of transactions. Ensuring API integrity is complicated by a number of factors: Extensive testing is required to ensure that APIs will satisfy expectations under the extreme conditions they might face in the field Validating end-to-end test scenarios typically requires access to third-party systems that are often unavailable or offline Each of the many continuously-evolving components involved in modern applications requires very specialized domain expertise to test and analyze efforts lack the breadth, depth, and repeatability that is critical for identifying application risks prior to production Without an enterprise-level automated solution for ensuring the integrity of APIs and API-driven applications, organizations risk: Brand erosion as faulty software drives away customers Time-to-market delays that diminish market share Exposure to legal liability associated with application failure Failure to comply with applicable regulatory standards and technical contracts Ensure that interconnected applications meet expectations Parasoft’s API Testing solution was built from the ground up to simplify the complex testing that’s vital for secure, reliable, and compliant composite applications. The result: Reduced costs by reducing testing costs, reducing technical debt, and exposing defects earlier in the SDLC Reduced risks by applying more exhaustive testing techniques, increasing test coverage, and immediately exposing any defects introduced by modifications Increased efficiency by leveraging sophisticated automation and enabling artifact reuse

An API Testing solution is commonly applied in the following situations: Complex composite applications: To visualize and validate how messages and events flow through the distributed architecture as tests execute Cloud-based applications: To facilitate cloud migration and ensure functionality, security, and performance expectations are met in dynamic environments Mobile development: To ensure the continued functionality, security, and performance of the frequently-evolving APIs that drive mobile applications End-to-end functional testing: To automate and analyze test scenarios across the many disparate, specialized endpoints involved in a single business transaction

1 Parasoft API Testing Solution / Data Sheet Parasoft API Testing Solution Real Results Parasoft’s comprehensive enterprise-grade solution dramatically simplifies the complex testing needed for today’s interconnected Parasoft API Testing successes include: business systems. AT&T: To ensure the reliability of iPhone billing systems Simple Generation of Flexible, Extensible Tests From a user-friendly interface, generate tests by monitoring live IRS: To ensure the accuracy and performance application traffic or analyzing key application resources. Intuitive of corporate tax e-filing systems GUIs visualize message structures, making it simple to fine-tune test CDC: To validate rule-based specimen messages and validations for GUI-less services and APIs. Parasoft’s management systems tests are engineered for easy sharing, reuse, and extension. Sabre: To ensure the reliability of the world’s largest travel network Repeatable Automated End-to-End Testing Using automatically-generated tests as building blocks, rapidly Cisco: To audit business processes define complex test scenarios that exercise and validate business IBM: To enforce governance policies transactions across multiple endpoints. From the messaging layer, Fidelity Investments: To create and to the web UI, to the database, ESB, and mainframes, the intuitive manage emulated service assets interface makes it simple to validate whether business logic HP: To validate complex business scenarios satisfies expectations. Easy integration with platforms ensures that critical errors are exposed immediately Vanguard: To ensure expected quality of service upon introduction. Bloomberg: To validate performance expectations Simulate the Behavior of Dependencies MedicAlert: To safeguard personal health Testing efforts are often delayed and/or compromised due to record management services difficulty accessing or configuring dependent components (3rd Siemens: To ensure secure, reliable account party applications or services, databases, mainframes, etc.). management and email services Parasoft Service Virtualization enables rapid, flexible simulation of these dependencies’ behavior, giving functional and performance Lufthansa: To ensure that cargo shipments testers unconstrained access to—and unparalleled control are planned and fulfilled flawlessly over—the dependent components they need to test against. This promotes earlier, faster, and more complete testing. Key Features

Built from the ground up for message-layer testing; industry “gold standard” since 2002 From a single intuitive interface, automate complex end-to-end scenarios across multiple endpoints (services, databases, Web UI, ESBs, mainframes…) Leverage broad, flexible support for protocols, transports, message formats Automatically generate tests that are robust, reusable, and easily shareable Keep test assts in sync with evolving systems Technologies & Protocols via automated intelligent updating Parasoft supports all the standard protocols and technologies, including: Visualize and validate how messages and REST / SOAP / JSON / JMS / MQ / NET WCF / TCP/IP / FTP / EDI / EDIFACT events flow through distributed architectures as tests execute HL7 / X12 / HIPAA / ISO 8583 / WSDL / WADL / WS-* / XML/PoX / UDDI Seamlessly integrate functional testing WSIL / BPEL / RMI / EJB / webMethods / TIBCO / SMTP / ISO 8583 / JSON / with load testing, service virtualization, and HTTP/HTTPS / ebXML / Equifax / JDBC / ODBC / Java Objects / Fixed Length / development testing Bytes/Binary / custom / more

USA PARASOFT HEADQUARTERS / 101 E. Huntington Drive, Monrovia, CA 91016 Phone: (888) 305-0041 / Email: [email protected]

© Parasoft Corporation All rights reserved. Parasoft and all Parasoft products and services listed within are trademarks or registered trademarks of Parasoft Corporation. All other products, services, and companies are trademarks, registered trademarks, or servicemarks of their respective holders in the US and/or other countries.