Magic Quadrant for Integrated Software Quality Suites 28 August 2014 ID:G00259602 Analyst(s): Mark Driver, Thomas E. Murphy, Nathan Wilson, Ray Valdes, David Norton, Maritess Sobejana VIEW SUMMARY NOTE 1 MOBILE APPLICATION TESTING Testing is one of the key processes in the software development life cycle. The market has been The mobile application testing market is a hotbed of steadily maturing to deliver improved user success, but technology changes like agile practices, mobile activity, with a growing number of providers and computing and cloud platforms can complicate vendor evaluation for testers and test managers. varying approaches, as well as growing capabilities. This can include broadening the platforms supported and device lab solutions. Techniques for the tools to access and drive the application are varied, exhibiting different strengths Market Definition/Description and weaknesses in platform coverage, and time­to­ market for new functionality. Depending on the types The integrated software quality (integrated quality assurance [IQA]) tool suite market is a subsegment of applications you are creating — mobile browser, of the overall software development life cycle market. It is a composite market composed of an native, hybrid and the level of interactivity (use of integrated suite of related toolsets, rather than a single technology. Within this suite, three traditional gesture sets, access to platform features), access path areas stand out as the principal focus: needs may vary. Most vendors in this research rely on partnerships with third parties to provide their mobile Test management — tools to manage and plan testing activities, defects and report results testing support, but a growing number have engineered their own solutions. The market has Automated stress and load testing — tools that simulate the load of multiple users against a developed slower than expected, as much of the server infrastructure to understand and tune performance mobile development has been outsourced business­to­ Automated functional and regression testing — tests that mimic a single user to find defects consumer development (e.g., Gorilla Logic) and business­to­enterprise, and B2B development (and in the application thus testing) has lagged. However, mobile is a critical element of future computing strategies. Testing will These three areas are the centralized focus of the market, and remain the most critical features when not find all the defects in mobile apps. We expect this Gartner evaluates vendor offerings. We expect market leaders to support all three functional areas. will lead to a combination of ad hoc testing Most leading vendors in this market analysis also provide tools for broader quality tasks, such as code approaches, using facilities such as Applause's Apphance, and other crowd or outsourced facilities, quality (static analysis), code review and unit test support. and a greater need for testing beyond traditional functional tests. In particular, usability, security and The breadth of types of technology and application layers that can be tested varies, with vendors in the localization tests will be a key part of a robust mobile Leaders quadrant generally supporting a broad set of technologies. Leading vendors provide app strategy. This will lead to a very dynamic market, integration/connectivity/synchronization between the suite components and other tools within the with the continued entry of new tools and services, software development life cycle (e.g., requirements management, continuous integration, etc.). partnerships and acquisitions, and the demand on organizations to increase funding for training. Because Vendors that don't have this breadth will generally act as those in the Visionaries quadrant or end up of the inherently dynamic nature of the market, expect tool acquisitions to be tactical at this point. more to the left of complete solutions (as specialized market players). In many cases, due to the large shifts in technology and development styles, we expect that many enterprises will have a core provider that is enhanced with other best­of­breed components. Our focus here is on vendors with the intent and NOTE 2 ability to provide offerings that serve as a core element of an enterprise quality assurance (QA) stack. COTS TESTING At its core, the IQA testing market is quite mature, with a strong stratification of vendors by annual We have not included COTS­testing­focused solutions revenue. In this Magic Quadrant, we are focused on the more mature elements of the market, as many in this research. Some of the included vendors do have of the more visionary elements supporting the Nexus of Forces (social, mobile, information and cloud) support for various COTS packages (typically SAP and are evolving rapidly and are fragmented, thus subject to tactical adoption. Therefore, the market Oracle), but focused solutions often provide additional acceleration, including management of test data, weightings emphasize execution­oriented criteria. Even today, we are seeing vendors that, in the past, connection to change management to automate may have been Visionaries now shifting into other quadrants, based on their ability to execute on sales selection of tests that need to be run and change and customer growth, partnerships, acquisitions, and filling out their offerings. As the market shifts impact analysis. Focused vendors are also beginning to toward mobile and service architectures, it is possible that market Leaders could be disrupted; provide support for SaaS solutions, and, as a whole, however, this will again be dictated by the execution of partnerships and acquisitions, and the ability to their focus on dealing with COTS implementations enable client success, which will require more than tools. As a result, we expect to make changes in the drives faster innovation in the specific market and a metrics and weightings in next year's Magic Quadrant that may more significantly shift positions in focus on the type of users that is different from custom development. In addition, a number of outsourcing future analyses. vendors have developed specific practices for various COTS solutions. Magic Quadrant Package­specific vendors include: Panaya SAP Figure 1. Magic Quadrant for Integrated Software Quality Suites Worksoft General vendors with COTS support include: Borland HP Oracle Original Software Tricentis EVALUATION CRITERIA DEFINITIONS Ability to Execute Product/Service: Core goods and services offered by the vendor for the defined market. This includes current product/service capabilities, quality, feature sets, skills and so on, whether offered natively or through OEM agreements/partnerships as defined in the market definition and detailed in the subcriteria. Overall Viability: Viability includes an assessment of the overall organization's financial health, the financial and practical success of the business unit, and the likelihood that the individual business unit will continue investing in the product, will continue offering the product and will advance the state of the art within the organization's portfolio of products. Sales Execution/Pricing: The vendor's capabilities in all presales activities and the structure that supports them. This includes deal management, pricing and negotiation, presales support, and the overall effectiveness of the sales channel. Market Responsiveness/Record: Ability to respond, change direction, be flexible and achieve competitive success as opportunities develop, competitors act, customer needs evolve and market dynamics change. This criterion also considers the vendor's history of responsiveness. Marketing Execution: The clarity, quality, creativity and efficacy of programs designed to deliver the organization's message to influence the market, promote the brand and business, increase awareness of the products, and establish a positive identification with the product/brand and organization in the minds of buyers. This "mind share" can be driven by a combination of publicity, promotional initiatives, thought leadership, word of mouth and sales activities. Customer Experience: Relationships, products and services/programs that enable clients to be successful with the products evaluated. Specifically, this includes the ways customers receive technical support or account support. This can also include ancillary tools, customer support programs (and the quality thereof), availability of user groups, service­level agreements Source: Gartner (August 2014) and so on. Operations: The ability of the organization to meet its Vendor Strengths and Cautions goals and commitments. Factors include the quality of the organizational structure, including skills, Automation Anywhere experiences, programs, systems and other vehicles that enable the organization to operate effectively and Automation Anywhere offers a strong solution for test automation and should be considered for teams efficiently on an ongoing basis. with nontechnical testers, good cross­platform support (including mobile), and coverage of functional Completeness of Vision and performance tests. Building on its success in desktop automation, the product has strong Market Understanding: Ability of the vendor to recognition technology and ease of use. The vendor has a hands­on approach to enabling successful understand buyers' wants and needs and to translate product trials, and makes use of a hybrid (online and field) sales model. those into products and services. Vendors that show the highest degree of vision listen to and