Testing Platforms by Theresa Lanowitz, Lisa Dronzek | June 5, 2012 Voke Research
Total Page:16
File Type:pdf, Size:1020Kb
MARKET MOVER ARRAYTM REPORT: Testing Platforms By Theresa Lanowitz, Lisa Dronzek | June 5, 2012 voke Research MARKET MOVER ARRAYTM REPORT: Testing Platforms By Theresa Lanowitz, Lisa Dronzek | June 5, 2012 ~ SUMMARY ~ TABLE OF CONTENTS The testing of software at every Market Mover Array Overview 2 stage of the lifecycle and across • Testing Market Overview 2 all aspects of the supply chain is a • State of the Testing Market 4 standard and required practice. As • Market Mover Array – Looking Forward 10 software and platforms continue to • Market Mover Array Methodology 11 increase in complexity and demand, testing is paramount. • Market Mover Array: Testing Platforms 12 Market Mover Array Vendors 13 Businesses must understand the risk posed by quality issues in the • Coverity 15 software in use. It is critical to deliver • Electric Cloud 19 software that will not compromise • Experitest 22 business, safety, or security. • HP 24 This Market Mover ArrayTM Report • IBM 29 examines the current state of the • Keynote DeviceAnywhere 32 software testing market and provides • Micro Focus 35 analysis of the vendors vying to • Microsoft 37 move the testing market beyond • Parasoft 42 the status quo through • Perfecto Mobile 46 innovation, • QMetry 49 technology, • QualiSystems 52 and the • Replay Solutions 55 conversation in • SmartBear 58 the market. • SOASTA 61 • Spirent 65 • TRICENTIS 68 • Wind River 71 • ZAP technologies.com 75 © 2012 voke media, llc. All rights reserved. voke and vokeStream are trademarks of voke media, llc. and are registered in the U.S. All other trademarks are the property of their respective companies. Reproduction or distribution of this document except as expressly provided in writing by voke is strictly prohibited. Opinions reflect judgment at the time and are subject to change without notice. voke disclaims all warranties as to the accuracy, completeness or adequacy of information and shall have no liability for errors, omissions or inadequacies in the information contained or for interpretations thereof. Contact www.vokeinc.com for additional information. 2 Market Mover Array™ Report: Testing Platforms MARKET MOVER ARRAY OVERVIEW Zero tolerance for errors. This is how software is viewed in the modern world where software is an on-demand commodity. With software always-on and expected to work as intended, a software failure may bring about catastrophic crises for its owner. The commoditization of software, along with the fact that it runs the business and is the differentiator, seems to be a paradox. Software is no paradox; its value is now inextricably linked to the brand of an organization. A software failure is a brand failure. In the post-global financial crisis (GFC) environment, high profile software failures are occurring at an alarming rate. Executives from organizations of all sizes must publicly answer to the reasons for software failures. Testing is an integral part of the entire lifecycle and has moved from obscurity to prominence. Proper quality of software means production, or production-ready, software, but it also means known and assessed business risk. The inextricable link of software and the brand has now made business leaders aware of the need for quality software with minimal business risk. The business of software must move from a singular focus on schedule to managing cost, quality, and schedule as it relates to business risk. Faster is not always better; organizations need to understand the risks associated with schedule demands. Software and platforms continue to increase in complexity and demand. Businesses must understand the risk of the software in use. This means that the critical task of delivering software that will not compromise business, safety, or security is paramount and ongoing. The testing of software at every stage of the lifecycle across all aspects of the supply chain is a standard and required practice. This Market Mover ArrayTM Report examines the current state of the software testing market and provides analysis of the vendors vying to move the market beyond the status quo through innovation, technology, and the conversation in the market. ~ TESTING MARKET OVERVIEW Since our last published report on the testing market in 2010 (see voke Market Mover ArrayTM Report: Testing Platforms – August 11, 2010) there has been a renaissance in the testing market by both the software vendors and the testing professionals themselves. Testing professionals are far more visible, vocal, and confident than at any other time in the history of the market. This newfound confidence is based on the value delivered through their dedication to software excellence. Testing professionals in proactive organizations have transformed themselves from being viewed as just necessary to release to roles of customer advocates and change agents. June 5, 2012 © 2012 voke media, llc. All rights reserved. Reproduction and distribution prohibited. Market Mover Array™ Report: Testing Platforms 3 In the role of customer advocate, they help to identify the risks associated with each and every software release. These strategic efforts help to deliver higher quality software throughout the enterprise, manage relationships, and place laser focus on assessing the risk associated with every piece of software. This rise of the testing professional is aided in part by the innovation in the market through the solutions and tools that vendors are delivering. One can almost make the classic “chicken and egg” argument about which came first—more vocal testing professionals or more sophisticated tools and solutions. Regardless of the answer, the innovation of tools and solutions over the past two years in the market is just as apparent as the more confident and strategic test professional. Innovation in the testing market is being driven by eight key factors: 1. HP testing assets are an overwhelming market force. By HP testing assets we refer to the breadth of testing artifacts (test cases, scripts, automation, et cetera) across all types of organizations using HP testing solutions. These assets protect HP’s market share and provide opportunity for vendors that create solutions leveraging these assets. 2. Lifecycle virtualization, the category that encompasses the benefits of virtualization applied to the application lifecycle, is the engine that is delivering unprecedented breakthroughs and transforming modern testing tools to new levels of productivity. (See voke Category SnapshotTM Report: Lifecycle Virtualization – November 7, 2011.) 3. Development testing is used to prevent defects from leaving the development phase. Preventing defects from leaving development makes software more predictable, traceable, visible, and transparent throughout the lifecycle. If software is more predictable, traceable, visible, and transparent, there is also better collaboration across the lifecycle as well. 4. Infrastructure test optimization (ITO), defined as the need to ensure that infrastructure is not a bottleneck and that anything added to the infrastructure does not negatively affect the network or other connected elements, is driving the need for end-to-end quality. (See voke Category SnapshotTM Report: Infrastructure Test Optimization – January 12, 2012.) 5. Mobility and the need to ensure and deliver a quality experience in the world of on- demand software is critical for all organizations. 6. The cloud—its economics and scalability—is a powerful delivery platform, yet it must meet the challenges of performance and security requirements. 7. Integration of testing products enables access to new, innovative solutions and openness, enabling collaboration across organizations and supply chains. June 5, 2012 © 2012 voke media, llc. All rights reserved. Reproduction and distribution prohibited. 4 Market Mover Array™ Report: Testing Platforms 8. Embedded software and the ongoing drive to convergence is a reality that organizations need to understand and embrace. In this report we refer to convergence as a long-term trend in which embedded and infrastructure groups migrate to commercial tools with the ability to share best practices and specialized testing resources across the organization. With innovation in the testing market being driven by these key factors, the market is undergoing a dramatic change. It is no longer sufficient to test just for functionality and performance. Today’s testing professional must work collaboratively across functional groups, with others in the supply chain, and throughout the lifecycle to ultimately be able to identify the risk to the business associated with software quality issues. Vendors of all sizes are having an impact on technology and innovation while shaping the discussion about what software testing means. As we identified in our 2010 report, the testing market is no longer solely defined by the need to test business applications. This statement carries even more heft in 2012. In 2012, we see vendors tackling testing problems on new platforms, innovating to focus on the total value of software, placing value on testing in every phase of the lifecycle, and working together to solve big, ongoing, classic computing problems. The testing market of the future will not have one dominant vendor; rather the market will be defined by software from testing vendors that is open and integrated with a wide variety of tooling options. This new view is being driven by on-demand software. Organizations can no longer dictate where, when, or how software is used. Workers are mobile, customers are global,