Continuous Functional Test Automation Suites, Q2 2020 the 15 Providers That Matter Most and How They Stack up by Diego Lo Giudice June 18, 2020
Total Page:16
File Type:pdf, Size:1020Kb
LICENSED FOR INDIVIDUAL USE ONLY The Forrester Wave™: Continuous Functional Test Automation Suites, Q2 2020 The 15 Providers That Matter Most And How They Stack Up by Diego Lo Giudice June 18, 2020 Why Read This Report Key Takeaways In our 26-criterion evaluation of continuous Eggplant, Tricentis, Parasoft, And ACCELQ functional test automation (CFTA) providers, Lead The Pack we identified the 15 most significant ones — Forrester’s research uncovered a market in which ACCELQ, Broadcom, Cyara, Eggplant, Experitest, Eggplant, Tricentis, Parasoft, and ACCELQ are IBM, mabl, Micro Focus, Parasoft, Perforce Leaders; mabl, Broadcom, Micro Focus, Sauce Software, Ranorex, Sauce Labs, SmartBear Labs, Perforce Software, IBM, and SmartBear Software, Tricentis, and Worksoft — and Software are Strong Performers; Cyara, researched, analyzed, and scored them. This Worksoft, and Experitest are Contenders; and report shows how each provider measures up Ranorex is a Challenger. and helps application development and testing A Smart, Integrated Test Experience Is The professionals select the right one for their needs. Key Differentiator Smart, integrated suites of testing tools outperform the less effective, siloed best-of- breed ones. On top of the typical integration and maintenance functions, capabilities like AI and machine learning (ML) infused in test automation design and execution, combined heterogenous channels, and API testing dictate which providers will lead the pack. Vendors that can provide a smart, integrated suite with seamless access to an extended set of capabilities position themselves to successfully deliver faster, lower- cost, and smarter testing to their customers. This PDF is only licensed for individual use when downloaded from forrester.com or reprints.forrester.com. All other distribution prohibited. FORRESTER.COM FOR APPLICATION DEVELOPMENT & DELIVERY PROFESSIONALS The Forrester Wave™: Continuous Functional Test Automation Suites, Q2 2020 The 15 Providers That Matter Most And How They Stack Up by Diego Lo Giudice with Christopher Mines, Julia Caldwell, Sara Sjoblom, and Kara Hartig June 18, 2020 Table Of Contents Related Research Documents 2 Test Automation Becomes Smarter, The 12 Must-Dos For Achieving Continuous Autonomous, And Universal Software Testing 3 Evaluation Summary Now Tech: Continuous Functional Test Automation Suites, Q1 2020 7 Vendor Offerings Now Tech: Shift-Left Performance Testing Suites 8 Vendor Profiles Of Tools, Q4 2019 Leaders Strong Performers Contenders Share reports with colleagues. Challengers Enhance your membership with Research Share. 15 Evaluation Overview Vendor Inclusion Criteria 17 Supplemental Material Forrester Research, Inc., 60 Acorn Park Drive, Cambridge, MA 02140 USA +1 617-613-6000 | Fax: +1 617-613-5000 | forrester.com © 2020 Forrester Research, Inc. Opinions reflect judgment at the time and are subject to change. Forrester®, Technographics®, Forrester Wave, TechRadar, and Total Economic Impact are trademarks of Forrester Research, Inc. All other trademarks are the property of their respective companies. Unauthorized copying or distributing is a violation of copyright law. [email protected] or +1 866-367-7378 FOR APPLICATION DEVELOPMENT & DELIVERY PROFESSIONALS June 18, 2020 The Forrester Wave™: Continuous Functional Test Automation Suites, Q2 2020 The 15 Providers That Matter Most And How They Stack Up Test Automation Becomes Smarter, Autonomous, And Universal The pandemic, recent financial crises, and market volatility are forcing enterprises to become adaptive. Adaptive companies move fast and deliver software quickly and continuously to react to market- changing conditions while keeping a strong focus on customer experience.1 Automation is at the core of being adaptive. While test automation is not new, past automation approaches have not helped app dev teams keep up with increased demands of modern delivery.2 Testing becomes strategic in new areas like automation (e.g., testing of robotic process automation [RPA], low-code, and AI- infused apps); smarter with infusion of AI and ML; and more efficient by converging different types of nonfunctional and functional testing. With business, technical, and developer personas all stewards of quality, modern software testing requires a technology environment that enables collaboration and easy navigation from one type of testing to another and among the different persona testers. As a result of these trends, CFTA customers should look for providers that: › Make functional test automation smarter and more autonomous. In the past 12 to 18 months, vendors have made progress in infusing AI and especially ML to augment user testing intelligence. From infusing natural language processing (NLP) in testing domain-specific languages to supporting automated self-healing of UI object recognition and turning testing results data into insights, expect help from CFTA to address many use cases.3 › Provide multiple testing capabilities plugged into functional test automation. As continuous delivery matures and Agile teams compress sprints, time for testing shrinks. Functional testing is converging with load and performance testing to save time in creating duplicate test cases and automation bots. Look for CFTA suites that provide converged types of functional automated testing, with load-speed performance testing, API testing, service virtualization testing (SVT), and more. › Provide an engaging, uniform, and straightforward test experience. The best tools provide a uniform, consistently easy-to-use experience: an environment that facilitates collaboration across the software development lifecycle (SDLC) and testing stakeholders but within which is always straightforward to operate. All testing personas should be able to smoothly navigate across the different testing types (e.g., functional, nonfunctional, API, web, and mobile) without feeling that they are moving to a different tool. Forty-four of the 56 reference customers we interviewed for this evaluation prefer an integrated suite versus a bunch of best-of-breed tools. © 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 2 [email protected] or +1 866-367-7378 FOR APPLICATION DEVELOPMENT & DELIVERY PROFESSIONALS June 18, 2020 The Forrester Wave™: Continuous Functional Test Automation Suites, Q2 2020 The 15 Providers That Matter Most And How They Stack Up Evaluation Summary The Forrester Wave™ evaluation highlights Leaders, Strong Performers, Contenders, and Challengers. It’s an assessment of the top vendors in the market and does not represent the entire vendor landscape. You’ll find more information about this market in our “Now Tech: Continuous Functional Test Automation Suites, Q1 2020,” report as well as “The 12 Must-Dos For Achieving Continuous Software Testing” report.4 We intend this evaluation to be a starting point only and encourage clients to view product evaluations and adapt criteria weightings using the Excel-based vendor comparison tool (see Figure 1 and see Figure 2). Click the link at the beginning of this report on Forrester.com to download the tool. © 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 3 [email protected] or +1 866-367-7378 FOR APPLICATION DEVELOPMENT & DELIVERY PROFESSIONALS June 18, 2020 The Forrester Wave™: Continuous Functional Test Automation Suites, Q2 2020 The 15 Providers That Matter Most And How They Stack Up FIGURE 1 Forrester Wave™: Continuous Functional Test Automation Suites, Q2 2020 Continuous Functional Test Automation Suites Q2 2020 Strong Challengers Contenders Performers Leaders Stronger current offering Tricentis Eggplant Broadcom ACCELQ Perforce Software Parasoft IBM Worksoft mabl Micro Focus Experitest SmartBear Software Sauce Labs Cyara Ranorex Weaker current offering Weaker strategy Stronger strategy Market presence © 2020 Forrester Research, Inc. Unauthorized copying or distributing is a violation of copyright law. 4 [email protected] or +1 866-367-7378 FOR APPLICATION DEVELOPMENT & DELIVERY PROFESSIONALS June 18, 2020 The Forrester Wave™: Continuous Functional Test Automation Suites, Q2 2020 The 15 Providers That Matter Most And How They Stack Up FIGURE 2 Forrester Wave™: Continuous Functional Test Automation Suites Scorecard, Q2 2020 s ester’ o Focus oadcom Forr weighting ACCELQ Br Cyara Eggplant ExperitestIBM mabl Micr Current offering 50% 3.67 3.85 1.87 4.47 2.63 2.85 2.94 2.85 Operating environments 10% 4.00 3.00 2.50 4.00 4.00 3.00 2.50 3.50 GUI design and automation 10% 3.66 3.02 1.66 3.66 4.34 3.00 3.66 3.00 API test design and automation 10% 3.00 5.00 1.00 4.00 1.00 5.00 2.00 4.00 Test and automation design 15% 4.00 4.00 2.00 5.00 1.00 2.00 2.50 3.00 Packaged application testing 10% 5.00 3.00 1.00 3.00 1.00 3.00 3.00 3.00 Automation execution/continuous 15% 5.00 5.00 1.00 5.00 5.00 3.00 3.00 1.00 testing Testing of BP and RP automation 10% 1.00 5.00 3.00 5.00 1.00 1.00 1.00 3.00 Qualities and integration 20% 3.25 3.00 2.50 5.00 3.00 3.00 4.50 3.00 Strategy 50% 3.80 3.00 3.60 4.40 2.40 2.60 4.00 4.00 Product vision 20% 5.00 3.00 3.00 5.00 3.00 3.00 3.00 5.00 Execution roadmap 20% 3.00 3.00 3.00 5.00 1.00 3.00 3.00 3.00 Innovation roadmap 20% 5.00 3.00 3.00 5.00 5.00 1.00 5.00 5.00 Partner ecosystem 20% 3.00 3.00 5.00 3.00 1.00 3.00 5.00 5.00 Market approach 10% 3.00 3.00 5.00 5.00 1.00 3.00 5.00 3.00 Commercial models 10% 3.00 3.00 3.00 3.00 3.00 3.00 3.00 1.00 Market presence 0% 1.50 4.50 1.50 3.00 2.50 5.00 1.00 4.50 Revenue 50% 1.00 4.00 2.00 3.00 2.00 5.00 1.00 4.00 Customers 50% 2.00 5.00 1.00 3.00 3.00 5.00 1.00 5.00 All scores are based on a scale of 0 (weak) to 5 (strong).