DECISION MATRIX Decision Matrix: Selecting an Application Lifecycle Management Vendor
Selecting an ALM suite provider in a fragmented market Reference Code: OI00068-002 Publication Date: March 2011 Authors: Chandranshu Singh, Tony Baer, Michael Azoff
SUMMARY
Catalyst As software development environments become increasingly complex and IT organizations continue to face budget pressures, application lifecycle management (ALM) solutions become part of the toolset necessary to manage development environments and distributed teams, and to ensure timely delivery of increasingly complex application software in a resource-efficient way. Ovum has developed this Decision Matrix report to help enterprises select the most appropriate ALM vendors based on technology strength, reputation among the ALM user community, and presence in the market. Ovum aims to provide a complete view of vendor capabilities, and advises on which vendors enterprises should shortlist, consider, and explore.
Ovum view Delivery Management is increasingly gaining prominence to address growing software reliability concerns.
HP, IBM, Microsoft, and MKS occupy the high end of the market and are the top overall performers.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 1 This report is a licensed product and is not to be photocopied HP and MKS have moved to the "shortlist" category since Ovum’s last Decision Matrix: Ovum believes HP's latest offering, HP ALM 11.0, with the leading position in the software quality management market, makes the company a formidable force in the ALM market. MKS advanced on the strength of its integrated platform and reach into the specialized, highly lucrative segment of the ALM market: embedded software in product automation and systems engineering.
Atlassian, CollabNet, and ThoughtWorks Studios are smaller vendors with impressive ALM suites.
Rally’s, Polarion’s, and TechExcel’s respective market positions have improved since the 2009 Decision Matrix report.
Micro Focus is the sleeping giant of the ALM space.
Key messages With economic recovery under way, organizations look for additional value from ALM investments.
Regulatory Compliance and Agile adoption remain major drivers of the fragmented ALM market.
SaaS, as a business model and an application delivery model, has had pockets of success, and ALM is one of these.
Software development for mobile devices and embedded software development are red-hot areas.
Application security, previously an overlooked area, has been prioritized by vendors such as HP, IBM, and Microsoft.
ALM vendors are struggling to respond to the DevOps movement.
Change and configuration management (CCM) functionality has been commoditized by popular open source tools.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 2 This report is a licensed product and is not to be photocopied MARKET DEVELOPMENTS
With economic recovery under way, organizations look for additional value from ALM investments Since Ovum published its last application lifecycle management (ALM) Decision Matrix report in June 2009, the largest geographic markets for enterprise IT have gone through a severe economic downturn. Predictably, the downturn led to large-scale IT initiatives being shelved, and ALM was no exception. This downturn has again brought the age-old problem of better management of software development processes to the fore. It is common knowledge that IT projects often suffer with budget and schedule overruns and delivered functionality not meeting business requirements. Given the frequency of project failures, organizations have done well to look at alternative approaches to managing software development. The focus is not just on better execution of the traditional software development lifecycle (SDLC) from requirements to release, but also on areas such as application performance management, application security, and bridging the gap between development and operations. Furthermore, emergent solution delivery channels such as SaaS ALM have gained traction in the market.
Although the overall ALM market has not grown appreciably in the past year, there are hot spots in areas such as Agile planning, complex/embedded systems development, and expanded take-up of tool offerings through the cloud. Spend rationalization has led organizations to look at ways to both maximize the benefits of existing ALM tool investments by re-examining their development processes, and to explore less capital-intensive options such as open source, cloud-based ALM. Additionally, organizations are keen to break up the siloed IT environment and bring the development function closer to the customer- or business-facing disciplines (or both) such as IT service management, with the overall goal of reliable and secure IT services and applications. On the supply side there is an overall trend towards broader integration, driven by the opportunities to upsell and to be the strategic tools provider that becomes the customer’s focal point for third-party tool integration. There is also growing support for enhanced collaboration and globally distributed development.
Regulatory compliance and Agile adoption remain major drivers for the fragmented ALM market Research reveals that regulatory compliance across industry verticals is a leading driver for the ALM market. Compliance efforts lead to ALM adoption in two ways. Firstly, such initiatives necessitate changes to various business processes and application software that need to be
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 3 This report is a licensed product and is not to be photocopied managed at enterprise level. This could mean hundreds or even thousands of application change requests for the development team. Projects of this scale require concerted effort across the organization and involve multiple stakeholders. The need for end-to-end visibility into the lifecycle that spans multiple projects and departments creates a strong case for ALM adoption. Secondly, organizations view ALM as an effective tool for strengthening the governance of development processes themselves. This is because ALM enables organizations to trace business requirements throughout the development lifecycle and further on to application code running in production environments, which creates audit trails for all process artifacts, tasks, and development team members.
The software development landscape has shifted in favor of Agile development practices. However, the hype around Agile has died down, and organizations now take a pragmatic approach to Agile – there is no single, pure Agile implementation in practice. Therefore, Ovum now sees hybrid environments in which the development methodology adoption is governed by the constraints of the project. Meanwhile, Agile practices also continue to evolve as organizations pick Agile elements that best suit their requirements, such as continuous integration, test-driven development, and (electronic) whiteboards. Agile has also given impetus to related management practices such as Lean and Kanban. These techniques, Kanban in particular, have resonated well with the market, and vendors have followed up with tools to support these processes. Another crucial factor is the availability of skilled and experienced developers. Since its inception, the Agile movement has been led by developers who have wanted lightweight processes that facilitate software development without hindering developer creativity. Thus, organizations faced with skills shortages have not had much success with Agile.
Agile ALM vendors have seen significant traction in the market; however, this is not based solely on the rapid rate of Agile adoption. It also has to do with the vendors being agile themselves and offering holistic solutions that couple products supporting Agile with superior customer support and end-user training at an attractive price. On a different note, Ovum expects that the recent consolidation activity in the Agile ALM market will continue, and the functionality offered by various Agile ALM tools will converge.
Overall, the market for ALM tools remains fragmented along several fault lines. (See Figure 1.) On the demand side, factors such as the organization’s size and the complexity of the IT environment play an important role in determining buying behavior. The bottom end of the market includes many small and medium-sized enterprises (SMEs) and freelance developers, development teams with 10 members or less, and consultants. This segment typically has highly skilled developers who can work alone or in small teams, and prefers Agile methodologies and open source software. The next stratum is comprised of medium to large organizations that often look at ALM from a cost
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 4 This report is a licensed product and is not to be photocopied containment perspective. ISVs are a special case, with presence across all size bands large, medium, and small; here developer productivity and tool interoperability remain key issues and, predictably, open source software is used heavily. Large, globally distributed enterprises are at the top of the pyramid; these organizations have existing and ongoing investments in ALM tools and rely heavily on advanced requirements engineering processes. This continues to be the most lucrative segment for ALM vendors in general and Agile ALM vendors in particular as Agile practices permeate into traditional waterfall process territory.
Figure 1: The ALM system market today: who is buying what?
Higher-value Advanced RD & RM, QA deals Full ALM stack for traditional Global enterprises, industrials development and, increasingly, Agile Sophisticated users Productivity is the #1 issue Software houses, ISVs Will use OSS to premium Mixed patterns– major opportunities Large businesses Reduce costs strong message ALM as a message does not sell to Agile developers Mass market: SME and freelance/consultants OSS ALM tools do well Addressable market
Source: Ovum O V U M
SaaS ALM: the big switch Industry experts have made the business case for SaaS many times over, with arguments typically revolving around terms such as total cost of ownership, enterprise data security, pre-integrated applications, freedom from managing application infrastructure, and so forth. As a business model and application delivery model, SaaS has had pockets of success, and the ALM market is one enterprise software segment where it has done well. ALM tools buyers usually have a software engineering background and can better appreciate the benefits of SaaS; predictably, SaaS ALM
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 5 This report is a licensed product and is not to be photocopied vendors have benefited from this aspect of the ALM market, registering strong double-digit growth both terms of in revenues and installed base. The average revenue growth rate for ALM vendors whose primary delivery model is SaaS (namely Rally Software, Atlassian, CollabNet, and TechExcel) is 32.5%. Although this growth is impressive, one must also keep in mind that it has come off a relatively small base compared with larger ALM vendors; whether such high-growth figures can be sustained year-on-year remains to be seen.
Ovum views ALM as an ideal candidate for SaaS, and believes SaaS penetration will grow further, reinforced by other major growth drivers such as Agile development and mobile application development. SaaS has the majority share in the mass market for ALM, and its presence in other segments is bound to increase with time.
Notably, the bottom of the ALM pyramid is also the hub for technological innovation in this market; for instance, these are the organizations that have been willing to extend the envelope by working with innovative, lightweight open source tooling. As these new technologies make their way into enterprise environments, the case for SaaS is bound to get stronger. Although technology adoption will largely depend on software refresh cycles, the premise of SaaS ALM can no longer be ignored by large enterprises and large ALM vendors alike. To clarify the point, consider the recent developments in the software configuration management segment of ALM – distributed version control systems (DVCSs) represent the next wave of innovation in SCM, and the key word is “distributed.” Any software application or process that is distributed in nature typically spans multiple persons, systems, and geographical regions. This makes such applications and processes ideal candidates for SaaS. Although DVCS adoption at present is greatest at the bottom of the pyramid segment, it is only a matter of time before it penetrates into other segments of the ALM market. Application development is an extremely collaborative activity, and increasingly distributed. Ovum believes organizations are already overcoming their psychological barriers to SaaS, and with increasing DVCS adoption there will be a compelling case for moving the rest of the application lifecycle stack to the cloud.
Vendor analysis
Rally Software and VersionOne have both set out to deliver hosted ALM services from inception, as their owners recognized the potential for SaaS early on. Not surprisingly, competitors have been drawn in.
CollabNet was a predominantly SaaS-based vendor from inception. Its latest acquisition, Codesion, brings into its fold a market-leading hosting provider for CollabNet's Subversion software version-control solutions. Codesion (formerly CVSdude) expanded from offering CVS to Subversion, its most popular offering, as
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 6 This report is a licensed product and is not to be photocopied well as up-and-coming DVCSs such as Git and Mercurial. The CollabNet portfolio now spans small-scale to enterprise-scale projects, the target market for TeamForge.
TechExcel has announced that its next major release in 2011 will offer expanded SaaS capabilities. The company expects growing demand for its hosted solutions and says customers will benefit from faster ramp-up capabilities.
Atlassian has been improving the capabilities of its SaaS offering, while continuing to offer desktop solutions. It recently announced the acquisition of Bitbucket.org, a rival to Codesion and principally a hosting provider for Git and Mercurial. Bitbucket provides another step in expanding Atlassian’s SaaS capabilities and, by offering DVCSs, it fits well in a SaaS context. Although Atlassian shies away from calling its platform an ALM system, it is one in everything other than name.
Although IBM Rational lagged in providing access to ALM tooling in the cloud, in 2010 it embraced the cloud with a vengeance. The vendor began with a modest base of offering AppScan security testing and Rational Policy Manager. In 2010, IBM Rational followed up with a multifaceted strategy for offering public and private cloud services. The initial rollout comprised Rational Team Concert, Rational Requirements Composer, Rational Quality Manager, Rational Build Forge, and Rational Asset Manager, hosted by IBM or business partners on IBM or third-party infrastructure, such as Amazon EC2.
Serena delivers some of its offerings via the cloud, including the Serena Business Manager orchestration engine, Serena Agile, and Serena PPM. However, it has yet to host any of the core Dimensions requirements or SCCM products through the cloud.
ThoughtWorks provides subscription-based offerings through a third-party hosting provider.
Other more recent vendors in ALM SaaS are also visible in the market. Tomos, Digite, and Kovair all have offerings, and the overall picture is a market going through a significant transition to SaaS ALM.
For now, neither MKS nor Polarion have been proactive. Polarion allows its tooling to be hosted, but does not offer the subscription-based licensing that often accompanies cloud-based deployment. MKS does not yet have active plans as it has not yet encountered sufficient demand from its customer base.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 7 This report is a licensed product and is not to be photocopied Mobile development – the new gold rush Mobile devices’ computing power and performance is increasing at a breakneck pace. With the increasing popularity of Apple iOS- and Google Android-based devices, the market for mobile application development is heating up, and Ovum expects that it will only grow as businesses go about extending their customer-facing applications to mobile devices. Ovum believes the next few years will see a great spike in mobile application development, led by target platforms such as Apple iPhone and iPad, Google Android-powered smartphones and tablets, and Windows Phones. However, application development for mobile devices, or even extending enterprise applications to such devices, places new constraints with respect to the device’s form factor, user behavior, and so forth.
While mobile applications emerge in the consumer market, the need for extending development to mobile targets is also starting to surface in the enterprise for employee-facing applications. Although in the short term the transition will occur relatively slowly, Ovum believes enterprise adoption of mobile devices as application targets will grow in a fashion similar to that of PCs a few decades ago.
Vendor analysis
As a group, ALM vendors are still figuring out how to support this emerging area of need in the software development market. So far ALM vendors do not recognize a special need to support mobile development; the main tool activity is in supporting core application development, especially around open web-based technology and translating such models into native apps. None currently provide mobile development environments (MDEs) or partner with third parties that offer them.
For now, MDE support is the domain of niche players such as:
Rhomobile, with its Ruby-based development framework
open source offerings such as Appcelerator Titanium platform, which supports development through multiple common web languages, and mobile deployment management platforms such as GPXS
Sybase, which at enterprise level maintains the most complete offering, spanning from application development and asset and security management to delivering business applications and messaging services.
Among ALM vendors, only IBM Rational has addressed mobile targets; it has done so to a limited extent. For instance, its Rhapsody realtime UML and SysML modeling tool can deliver
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 8 This report is a licensed product and is not to be photocopied preconfigured API libraries for Android platforms, plus a more generic capability for targeting other mobile clients; Rational Application Developer offers code assist, emulation, and validation capabilities for Android and RIM Blackberry devices only. Across IBM Software Group, there are numerous products that address some aspect of mobile application development or deployment. However, IBM has yet to unify its mobile application development tool strategy.
As mobile development support tools gain popularity and prevalence, ALM vendors must be able to connect to these and draw out metrics and track work items. In 2011, Ovum expects some ALM vendors to ramp up mobile support; watch for partnerships and acquisitions in this space.
Application security is an area of concern While software quality has always been a key factor driving ALM adoption in organizations, recent developments suggest that software reliability and application security are the new problem areas. As organizations have proceeded to secure their perimeters, networks, and infrastructure from attacks, application security has been left largely unaddressed and therefore vulnerable. In our view, application security should be factored into the development lifecycle itself, rather than left for third-party tools to address. Security should be an integral part of the application lifecycle, and organizations should conduct security risk assessments at various stages of the lifecycle, such as requirements, design, development, and integration testing. Features such as integration with static and dynamic code testing tools, security issue impact analysis, defect management tool integration, risk prioritization of security issues, and security assessment reporting are no longer simply “nice to have” from an ALM functionality perspective.
Vendor analysis
Application security is an area that most ALM vendors have overlooked. Of all the vendors profiled in this Decision Matrix report, most integrate with static and dynamic source code testing and verification tools. However, in our view, this is a basic feature of ALM products; ALM solutions offering application security management functionality must also offer other required features.
Thus, in our assessment, only IBM and HP offer functionality to manage application security issues during the lifecycle; they have already taken the first steps toward integrating security testing into the application lifecycle, but have not yet completed the job. Both offer static and dynamic security testing in their portfolios; IBM Rational has integrated static and dynamic testing capabilities, which came from separate acquisitions two years apart, in a consolidated offering that is available in role- based editions. For HP, the big news in 2010 was its acquisition of long-time partner Fortify, which provided static and dynamic testing capabilities plus a new risk management offering.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 9 This report is a licensed product and is not to be photocopied By contrast, MKS integrates with several automated testing tools that only provide spot coverage of security testing issues. Of note, Compuware began developing security testing capabilities within the automated testing tooling that it subsequently sold to Micro Focus; Ovum awaits Micro Focus's forthcoming next-generation release, which will consolidate the quality management assets that came with the Compuware and Borland tool acquisitions.
Microsoft has also tackled application security through introducing a security framework. Ovum believes that in 2011 other ALM vendors will need to acquire or partner with niche players in order to start offering their own roundtrips between security monitoring and defect tracking and requirements management (RM) tools.
Software plays a greater role in product engineering Leading vendors such as IBM Rational and MKS are already seeing the brunt of growth in their businesses in supporting development of software that is embedded in complex systems such as engineered products or civil infrastructure. Additionally, Polarion and Serena are targeting this segment for growth. This market segment has grown in spite of the recession because of continued demand from consumers for high-tech products, and from segments such as public sector and biomedical devices for smarter equipment.
Revenue growth in this segment of the ALM market is significant in spite of the fact that it is a niche market in which the number of potential customers is dwarfed by those in the mainstream enterprise sector. However, these customers have a need for large implementations addressing requirements, quality, design, change, and release management implementations that often number in the hundreds or thousands of seats. The average size of deals in this segment is vastly outweighing those of mainstream enterprise IT development.
Vendor analysis
In this segment, there is entrenched rivalry with large CADCAM players such as PTC, Siemens, and Dassault Systemes. The main attraction for ALM players is that the PLM space is huge compared with their traditional enterprise software development space. However, only a few ALM vendors are rising to the challenge because targeting product lifecycle management (PLM) dictates highly complex project requirements and quality management functionality that might be excessive for enterprise software development.
IBM Rational and MKS have taken the most forward positions in targeting this space. Through acquisition of Telelogic, IBM Rational added products such as DOORS for requirements management, Focal Point for portfolio management, and Rhapsody for SysML modeling, which
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 10 This report is a licensed product and is not to be photocopied are targeted at complex systems. MKS has refocused its entire ALM business around this fast- growing segment. Additionally, Polarion and Serena have customers in this segment, and plan to ramp up in this space in 2011.
ALM vendors are struggling to respond to the emerging DevOps movement Like mobile, this is a segment that has caught many ALM vendors off guard. That is because software development and IT operations have traditionally been separate end markets served by different technology vendors. DevOps is the consequence of two trends:
top-down adoption of ITIL process initiatives among IT operations organizations, especially for incident, problem, and change management
more recently, grassroots movements, especially in the Agile software development community, to make the release management aspect of continuous integration a more predictable activity.
Vendor analysis
Today, only a handful of vendors with software development and operations/service management/application performance management tools have built feedback loops between service-desk and defect-tracking tools. Most ALM vendors now include DevOps in the "future development" slides of their product roadmaps.
The logical starting point is for software-related trouble tickets reported by the IT service desk (help desk) to be automatically submitted to defect-tracking systems. Most ALM vendors with defect- tracking tools currently support this capability, some with formally supported integrations to popular support desks such as BMC Remedy, and others with more generalized APIs. The next logical step is to enable repurposing of performance test scripts to be used for conducting synthetic tests on the operational side; a few ALM vendors offer this capability. The more advanced vendors in this space will begin to add links between source code and change management (SCCM) and the configuration management databases (CMDBs) that are emerging for more ambitious ITIL implementations.
Currently:
IBM Rational, MKS, Serena, and HP have productized links to specific service-desk systems, and Polarion has released a general-purpose API.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 11 This report is a licensed product and is not to be photocopied IBM Rational and HP can repurpose performance tests scripts as synthetic operational performance tests, and IBM also supports integration of change management through interoperation of Rational Asset Manager and Tivoli CCMDB.
Serena's new partnership with Nolio offers possibilities for integration of change management processes as part of release automation; at this point, both vendors need to set their priorities for the direction of their relationship.
Remaining ALM vendors have yet to formulate their DevOps product strategies. The primary challenge is marketing, not technology, as ALM and IT service and infrastructure management tools providers have the systems in place that could form the building blocks of combined solutions.
Ovum expects a virtuous cycle: vendors with solutions will benefit from the DevOps movement, and the DevOps movement in turn will gain momentum as automation support reinforces good DevOps practices. This field is ripe for innovation; for instance, Nolio's unique release automation opens possibilities for linking deployment scripts to classic runbook automation.
The Application Lifecycle Management Decision Matrix The Ovum Decision Matrix provides a summary of the market standing of the major ALM vendors based on a quantitative and objective assessment of their technology capabilities, market impact, and customer sentiment. The Decision Matrix provides vendor selection guidance for enterprises looking to deploy ALM solutions and advises them on whether a vendor should be shortlisted, considered, or explored further. In this release of the ALM Decision Matrix, the clustered technology assessment and insufficient customer sentiment survey scores for some vendors created difficulties in classification. Ovum’s ALM Decision Matrix therefore offers two angles of analysis:
The Decision Matrix: Figure 2 includes the vendors that have significant responses in all three assessment categories, namely Customer Sentiment, Technology, and Market Impact. The core decision matrix prioritizes Customer Sentiment and Technology assessment scores, placing them on the X and Y axis, with Market Impact represented by the size of the bubble. Technology forms the bedrock of a successful ALM offering, and Ovum finds that high Customer Sentiment scores are indicative of successful go-to-market strategies.
The Extended Decision Matrix: This includes the vendors that have registered considerable Market Impact and Technology assessment scores but do not have significant Customer Sentiment data to be rated objectively. Due to the high
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 12 This report is a licensed product and is not to be photocopied Technology and Market Impact assessment scores garnered by these vendors, Ovum explores an alternative means of representing all vendors studied: an extended ALM decision matrix (Figure 3). This scatter diagram represents each profiled vendor, with the X axis reflecting a vendor's Technology assessment score and the Y axis representing its Market Impact score.
Figure 2: ALM Decision Matrix
Source: Ovum O V U M
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 13 This report is a licensed product and is not to be photocopied Table 1: ALM Decision Matrix
Shortlist Consider Explore HP Micro Focus Atlassian IBM Polarion CollabNet Microsoft Serena Software Rally Software MKS TechExcel ThoughtWorks Studios
Source: Ovum O V U M
Figure 3: Extended ALM Decision Matrix
Source: Ovum O V U M
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 14 This report is a licensed product and is not to be photocopied As discussed in the market developments section, the ALM market has undergone many changes. The Ovum ALM Decision Matrix also reflects this shift in market direction. Vendors that were part of Ovum’s 2009 ALM Decision Matrix report but have not been included in this report are Aldon, Compuware, and Borland. Aldon was excluded because its coverage of ALM was not broad enough to meet Ovum’s definition. Similarly, having sold its software quality business to Micro Focus, Compuware’s resultant portfolio was too narrow for it to be called an ALM vendor. Borland was excluded because it had been acquired by Micro Focus.
Atlassian, ThoughtWorks, Rally Software, and CollabNet are smaller vendors with impressive ALM suites New entrants to 2010’s Decision Matrix include Atlassian, CollabNet, ThoughtWorks Studios, and Micro Focus. Atlassian, CollabNet, and ThoughtWorks Studios all have strong propositions for the ALM market backed by strong offerings, and Ovum expects that these vendors will continue to impact the market in the years to come.
Polarion’s and TechExcel’s respective market positions have improved A significant aspect worth noting is that Polarion and TechExcel fall under the Consider category in this edition of the ALM Decision Matrix. In the previous report these vendors were included in the Explore category. The Consider category is the name Ovum has given to the set of vendors a tier below overall market leaders. The market traction these vendors have gained is impressive, given that only two years have passed since Ovum published its previous ALM Decision Matrix report and that we take numerous factors into account before arriving at respective market positions for leading vendors in any enterprise software segment. In addition, Polarion and TechExcel do not have the overall scale enjoyed by the market leaders (HP, IBM, and Microsoft).
Micro Focus is the sleeping giant of the ALM market In the previous ALM Decision Matrix report, Borland was positioned in the Shortlist category, and Compuware in the Consider category. Although it seems counter-intuitive that Micro Focus, which acquired Borland and Compuware’s software quality business, should be positioned in the Consider category in the present edition of the Decision Matrix, at time of writing Micro Focus has not released an ALM suite. It is only making available a select number of tools from its acquired portfolio. Ovum characterizes Micro Focus’s latent potential as that of a sleeping giant: we will wait to see what emerges, but in our opinion the vendor’s message post acquisition was not
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 15 This report is a licensed product and is not to be photocopied communicated clearly to the market. In contrast, Micro Focus’s Application Management and Quality segment has registered double-digit revenue growth, and Ovum expects that the vendor’s impact on the ALM market will grow further in 2011 and beyond.
HP and MKS emerge as ALM market leaders HP was placed in the Consider category in the previous ALM Decision Matrix report. This year’s report ranks HP third in the Shortlist category. This reflects HP’s expansion of its tools portfolio from quality management, application performance management, and project performance management to a more well-rounded stack. The missing elements in HP’s newly released ALM bundle include CCM, model-driven development, database modeling and design, build management, and more advanced Agile project management features; significantly, some of these capabilities are also covered by CollabNet, with which HP maintains a strategic alliance. With the launch of HP Software ALM 11.0 in late 2010, the vendor has brought to market a significantly broader offering. HP has communicated clearly its product roadmap, although Ovum has questions about the ultimate role that CollabNet’s products will or will not play in HP’s future. Thanks to its long-time dominance of quality management and test automation, HP also brings a well-developed partner network and a broad base of third-party integrations. Many of these include rival ALM products such as IBM Rational. HP’s prime weaknesses include the newness of its ALM offerings, of which several portions are only starting to be market tested, and its low score on Customer Sentiment. (It scored second to last – just above IBM Rational – some of which can be attributed to the newness of the offering.)
Similarly, MKS also advanced from Consider to Shortlist this time around. As the vendor that pioneered ALM as a single unified product, MKS earned the second-highest Technology rating in the Decision Matrix. Its product, MKS Integrity, ties RM, SCCM, project planning, and quality management (although not test automation) into the same platform. Along with IBM Rational, MKS has taken a forward position in the fast-growing, potentially lucrative embedded software development space. That is both MKS’s strength as an ALM solution provider, and its weakness when positioned in the overall market. As a modest-sized company, MKS has been smart to focus its guns and not be all things to all people; however, the flip side is that the company has low recognition in the broader ALM market, which is dominated by enterprise IT software development.
IBM and Microsoft maintain their leadership positions IBM and Microsoft have maintained their respective leading positions in the market. Since the last ALM Decision Matrix report, IBM has released the Jazz platform, launched new Jazz-based ALM products, migrated some of its existing ALM products to Jazz, and used Jazz as a connection
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 16 This report is a licensed product and is not to be photocopied bridge for the remaining ALM portfolio. This goes to show that IBM as a market leader recognizes that tool interoperability is a key issue faced by organizations, and has moved quickly to address this, even though it involved re-architecting most of the company’s legacy ALM portfolio.
Microsoft’s ALM message has got stronger in the past two years; in addition to ALM, the vendor has a complete software development stack as well as an interactive development environment (IDE), and a large developer community. Microsoft has also made its ALM branding stronger, leveraging the popularity of the Visual Studio IDE; however, requirements definition (RD) and management and software quality management remain areas of concern for the IT behemoth.
In this Decision Matrix report, IBM has the top score in the Technology and Market Impact dimensions; yet the company has not done well in the Customer Sentiment dimension. From a technology perspective, IBM has the broadest and arguably the deepest portfolio among all vendors profiled in this report. IBM’s Technology score is 94.3%. Furthermore, the vendor has ensured that it stays ahead of the pack by having a good grasp of market demand and meeting it through tool support. IBM has the highest Market Impact score (10), and all other vendors have been ranked relative to IBM. The Market Impact dimension takes into account parameters such as: revenues from the ALM market and its spread in terms of organization size, vertical presence, and geography; year-on-year ALM revenue growth in percentage terms; and market recognition. IBM has the top score in all these parameters except revenue growth, which is expected due to IBM’s large base.
Customer sentiment is one area in which IBM’s performance is significantly lower than other vendors’. It also has the lowest Customer Sentiment score among ALM market leaders, whereas in the previous ALM Decision Matrix report IBM was one of the top scorers in this category. Ovum believes that the survey results provide many significant takeaways for IBM. Notably, IBM has scored low on product usage, service levels and response times, client engagement, and customer support. Customers aren't always clear on IBM's ALM roadmap for several reasons. It has come about through multiple acquisitions, and IBM's Jazz initiative has led to re-architecting and rebranding of several Rational products.
Microsoft is second only to IBM in the ALM market. Microsoft has the third-highest Technology score (83.4%), bettered only by IBM and MKS. Microsoft has a broad ALM portfolio, though certain gaps exist. On the whole, Microsoft’s performance on the Technology scale is commendable, reflecting the vendor’s transition from primarily IDE vendor, to ALM vendor with heavy dependence on partners (VSTS 2008), to ALM vendor with a well-rounded portfolio (VSTS 2010), to ALM market leader (Visual Studio 2010) with a strong brand and consistent performance. Microsoft’s Market Impact score is second only to IBM; this is largely due to Microsoft’s high recognition and
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 17 This report is a licensed product and is not to be photocopied ALM revenues. On the Customer Sentiment scale Microsoft has performed better than IBM, the overall market leader; however, it is not the top scorer in this category even though it has the highest Customer Sentiment score among the market leaders. There is room for improvement, primarily in customer support.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 18 This report is a licensed product and is not to be photocopied MARKET LEADERS
Overview The application lifecycle management market is broad and includes many subsets. Vendors vary in focus and market strength. In this section, Ovum takes a deeper look into the underlying patterns in the market by comparing the top vendors in each of the criteria that contribute to the Technology, Market Impact, and Customer Sentiment ratings.
Technology
Figure 4: Market leaders analysis: Technology (suite architecture)
Source: Ovum O V U M
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 19 This report is a licensed product and is not to be photocopied The state of technology leadership reflects ALM market maturity
The “Market leaders analysis: Technology (suite architecture)’ schematic (see Figure 2) reflects market maturity in terms of technological capabilities. A cursory glance at the schematic shows that most vendors are sensitive to key issues such as internal tool interoperability, integration with third-party ALM tools, and the collaborative nature of software development activities. The market has also converged in terms of overall solution architectures – nearly all vendors now offer common repository-based solutions with support for repository federation, as well as other essential pieces such as workflow, analytics, web-based client access, and support for Agile development methodologies.
Having figured out the general market direction, vendors are keen on plugging the gaps
Although no other vendor offers an ALM portfolio as comprehensive as IBM’s, the intent of filling in the missing pieces through either product development or third-party integrations is apparent. The ability to fit seamlessly into customers’ IT environments and interoperability with incumbent tools is no longer a key differentiator, as is clear from vendors’ scores in this category. It is interesting to note that IBM has the lowest score in the external tool interoperability dimension; given that IBM offers all pieces of the ALM puzzle in its large installed base, it is apparent that the company’s strategy is to facilitate and incentivize customers’ further reliance on IBM technology. In addition, IBM is working towards closing this gap through its Jazz platform and the OSLC initiative; as OSLC has failed to attract other mainstream ALM players so far, its industry impact cannot be qualified at this point.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 20 This report is a licensed product and is not to be photocopied Figure 5: Market leaders analysis: Technology (suite components)
Source: Ovum O V U M
Project management is the gateway to ALM
Six vendors out of 12 have perfect scores in the project management dimension, and nine have scores over 90%. This goes to show that lifecycle visibility, traceability of artifacts, and the ability to act or take corrective measures based on near-realtime information is a top priority for most ALM buyers. More importantly, executing project management correctly is the key to better fortunes in the ALM market, as demonstrated by emerging vendors such as Rally Software, ThoughtWorks Studios, Polarion, TechExcel, and CollabNet.
A commonly held view is that application project portfolio management (APPM) follows naturally from project management. This is true, but only with respect to managing multiple projects or
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 21 This report is a licensed product and is not to be photocopied project hierarchies. Most vendors provide a hierarchical or tree structure with rolled-up reporting capabilities. However, other aspects of APPM, such as critical-path analysis, financials management, “what-if” scenario modeling, risk management, and demand management, are not natural extensions of project management capabilities. Therefore, if a vendor performs project management well, it likely will also offer multiple project management functionalities; however, adding other capabilities will require substantial development effort. Hence, the APPM dimension is not as clustered as the project management dimension; only two vendors have perfect scores, and only four vendors have scores of more than 90%.
Requirements definition, requirements management, and change and configuration management remain specialist businesses
No vendor has a perfect score in the RD category and, on the whole, vendors’ scores are lower than corresponding project management or APPM scores. This goes to show that RD is a specialist area insofar as vendors with established RD products have been able to protect their businesses, and emerging ALM vendors have not devoted significant product development resources in this direction. Interestingly, Microsoft has the lowest RD score of all vendors.
Only IBM has a perfect score in the RM dimension; other leading vendors include Micro Focus, Serena, Rally, and TechExcel. For most vendors, RM amounts to requirements tracking through lifecycle, dependency mapping, requirements baselining, and reporting. Requirements-based test case generation and visual requirements modeling do not seem to be focus areas for most vendors.
CCM is a specialist area of ALM, and the same can be inferred from the clustering of vendors in this dimension. However, it must also be noted that popular open source CCM tools such as Subversion have created a strong undercurrent in this ALM discipline, and led to the commoditization of this segment. Some vendors that scored high marks in the CCM dimension, such as Atlassian and Polarion, embed Subversion into their ALM solutions.
IBM, Microsoft, Polarion, and Serena share the top spot with perfect scores; MKS and Atlassian are tied for the second spot with scores marginally lower than those of category leaders; CollabNet and Micro Focus share the third spot with scores just under 80%. The gulf between these eight vendors and the remaining four is extremely wide, with the next-best scores (those of HP, Rally, and ThoughtWorks) just over 20%. It is interesting to note that CollabNet, the lead vendor on the Apache Subversion project, scores lower than Polarion and Atlassian (vendors that embed the open source CCM tool into their ALM suites). In the main, this is due to CollabNet’s lack of visual workflow with automation and access management, visual exploration of branch inheritance, and automated branch inheritance – features that Polarion’s version of Subversion provides. Although
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 22 This report is a licensed product and is not to be photocopied Atlassian does not offer visual workflow with automation and access management, it offers automated branch inheritance, and visual exploration capabilities for branch inheritance.
Software assurance and application security remain the stronghold of established ALM vendors
The longer a product has been on the market, the more third-party integrations it is likely to have; this is largely true for the software assurance space, which encompasses requirements management, defect management, quality management, application performance management, and IT service desk, with established ALM vendors scoring higher on software assurance integrations than their recent competitors. Application performance management is an interesting case in point: only IBM, HP, Micro Focus, and Microsoft have APM integrations. Micro Focus, HP, Microsoft, Polarion, and MKS are the leading vendors in the software assurance integration dimension. IBM and Polarion are exceptions, and the rationale behind IBM’s limited number of third-party integrations has already been discussed. (The vendor is working towards an industry- wide ALM integration framework and is not keen on developing integrations with point products.) Polarion and TechExcel – although TechExcel does not feature in the software assurance dimension due to weak requirements management and QA integration – are the only emerging players in the ALM space, with significant investments in quality management and software assurance integrations.
Quality management scores are largely along expected lines, with HP, IBM, TechExcel, and Micro Focus scoring well. TechExcel and Micro Focus are tied for the second spot ahead of HP, and that may come as a surprise to many; it’s primarily due to TechExcel’s test lab management capability for cloud based test environments that HP lacks. HP offers test lab management only for on premise environments. Micro Focus misses out on a perfect score due to the lack of test lab management capability for on premise environments.
Application security has largely been overlooked by ALM vendors. Hence, in the application security dimension, established ALM vendors with existing investments in application security testing scored well, and a wide gulf exists between these vendors and the rest of the pack. IBM, Microsoft, HP, and MKS each have a perfect score. Vendors were evaluated on parameters such as static and dynamic testing tool integration, automated defect creation for security issues, security issue impact analysis, security risk prioritization, and security assessment reporting.
Established ALM vendors are now aboard the Agile train
Many emerging ALM vendors have been championing Agile practices for many years, for which they have reaped commensurate benefits as the market has now largely adopted the Agile mantra.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 23 This report is a licensed product and is not to be photocopied There have been ups and downs and learning opportunities for both the Agile vendor community and its early adopters. Not all organizations have had smooth sailing with Agile adoption, and tool support for Agile methodologies has evolved over the years as vendors learned what worked and what did not. However, this market is now hot and, in Ovum’s view, is moving towards feature set convergence. Ovum finds all leading ALM vendors back up their Agile marketing communication with actual tool support, with the exception of Micro Focus. Established ALM vendors such HP, IBM, Microsoft, MKS, and Serena scored well in this dimension and, overall, nine vendors scored more than 80%.
Micro Focus’s Agile support is limited. Micro Focus StarTeam’s capabilities can be extended using the SDK for multi-project Agile planning. StarTeam is process agnostic, thus users can configure process templates to Agile or Agile hybrid variants. SilkCentral Test Manager and SilkTest support API-based integration with Agile project management and planning tools.
Delivery management is increasingly gaining prominence to address growing software reliability concerns
Delivery management encompasses build, release, and deployment management disciplines. As the Technology (suite components) schematic illustrates, most ALM vendors realize that downstream processes of the software lifecycle are as important as their upstream counterparts, if not more so. Build, release, and deployment management precede production deployment of applications. A concerted approach towards delivery management is the result of the numerous production application failures, and it aims to enhance the reliability of delivered software by closing the gaps between build, deployment, and release processes, thereby making software releases more predictable.
Micro Focus and MKS are the leaders in this category; however, they are just marginally ahead of IBM and ThoughtWorks. Other vendors that scored well in this dimension include Microsoft and HP, followed by Atlassian, CollabNet, and Polarion.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 24 This report is a licensed product and is not to be photocopied Customer Sentiment
Figure 6: Market leaders analysis: Customer Sentiment
Source: Ovum O V U M
Emerging ALM vendors are overall leaders in Customer Sentiment
Emerging ALM players Atlassian, Polarion, and Rally are the overall leaders in Customer Sentiment. Market leaders IBM, Microsoft, MKS, and HP performed relatively poorly on the Customer Sentiment scale in this edition of the ALM Decision Matrix. IBM and Microsoft were rated
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 25 This report is a licensed product and is not to be photocopied by their customers as leaders in the financial stability dimension. IBM also makes an appearance in the vertical specialization, service capabilities, and product integration dimensions; however, its score is a notch lower than category leaders in all three cases. In addition to financial stability, Microsoft was rated a joint leader in the product usage and portfolio depth dimensions. Microsoft’s scores in other Customer Sentiment dimensions, such as product integration, vertical specialization, client engagement, and service levels, are a notch lower than respective Customer Sentiment category leaders. Microsoft also has the highest overall Customer Sentiment score among the Shortlist vendors. Surprisingly, IBM is not among the leaders in the portfolio depth dimension, even though it has arguably the most feature-complete toolset in the market and is the undisputed leader in Technology ratings. This anomaly is likely due to the surveyed customers’ dissatisfaction with IBM, and is reflected in IBM’s low scores in dimensions that have direct correlation with customer satisfaction, such as customer support, service levels, and client engagement.
MKS finds a mention in the product usage dimension as a joint category leader; the vendor is also present in the client engagement, service levels, service capabilities, and vertical specialization dimensions. HP is present in the product integration, vertical specialization, portfolio depth, and financial stability dimensions. The vendor is the category leader in the vertical specialization dimension and a joint leader in the portfolio depth dimension.
The key factor behind market leaders’ poor performance on the Customer Sentiment scale appears to be low customer satisfaction, an aspect in which the smaller vendors have excelled. The dimensions that indicate customer satisfaction with the vendor’s quality of service (such as customer support, service capabilities, service levels, and client engagement) collectively contribute 35% to the overall Customer Sentiment score. Each of these dimensions has an emerging ALM vendor as the category leader, with all four Shortlist vendors out of reckoning. Ovum believes the following factors have led to this result:
As they are smaller in both company size and customer base, the emerging ALM vendors are able to have more intimate relationships with a larger cross-section of customers.
Customers probably do not have as grand of expectations of smaller companies as they would of the larger, more diversified vendors.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 26 This report is a licensed product and is not to be photocopied Market Impact
Figure 7: Market leaders analysis: Market Impact
Source: Ovum O V U M
Shortlist vendors are the overall leaders in Market Impact
HP, IBM, and Microsoft are the top-three vendors in all dimensions of Market Impact, except revenue growth and vertical presence. These three vendors, together with Rally Software and Atlassian, take the second spot in vertical presence. The vertical presence dimension indicates the degree of presence a vendor has across verticals with respect to its own total revenues, not against absolute revenue earnings. This metric therefore measures how well a vendor’s income is
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 27 This report is a licensed product and is not to be photocopied spread across verticals – a high score indicates that the vendor has customers across many verticals.
Interestingly, MKS, a Shortlist vendor, finds no mention in any Market Impact dimension. This is due to the fact that MKS is not among the top-three vendors in any Market Impact dimension, and makes the Shortlist category purely on the strength of its ALM offering. Polarion, Rally, and Atlassian are the leaders in the revenue growth dimension; as with vertical presence, this dimension does not take into account the absolute value of earnings.
The “Market leaders: Market Impact” schematic (Figure 6) shows the gulf between market leaders and the rest of the pack in sheer revenue terms. This divide is also apparent in the Extended Decision Matrix chart. It is also worth noting that relative positions of the market leaders in each of the dimensions (except revenue growth) are unchanged, with IBM as the overall leader, followed by Microsoft and HP.
The smaller vendors are growing faster than the ALM giants
As the list of leaders in terms of revenue growth indicates, the smaller vendors are growing faster than the largest ALM vendors. That said, the difference in scale is of two orders of magnitude, and it is unlikely that the market structure will change in any meaningful way any time soon. However, the good showing by smaller ALM vendors on the Technology and Customer Sentiment scales, incorporating factors such as technological capabilities, future product roadmap, customer perception of product quality, and customer perception of their quality of service, suggests that they will remain strong contenders in the ALM market.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 28 This report is a licensed product and is not to be photocopied VENDOR ANALYSIS
Atlassian
Figure 8: Atlassian ALM radars
Source: Ovum O V U M
Atlassian is an emerging ALM vendor, with an impressive ALM portfolio. Atlassian’s Technology and Market Impact scores reflect the vendor’s relative position in the ALM market. On the Technology front, Atlassian’s scores are close to the average on dimensions such as breadth of ALM coverage (joint fourth), solution architecture (joint fourth), external tool interoperability (joint leader), and collaboration (shares the second spot). In terms of lifecycle disciplines Atlassian’s offering merits a mention only in change and configuration management (joint second), due in
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 29 This report is a licensed product and is not to be photocopied great measure to the Subversion repository that Atlassian embeds in its hosted ALM solution, JIRA Studio. Atlassian’s overall score on the Technology scale is the lowest among all vendors profiled in this report. Atlassian’s Market Impact is two orders of magnitude lower than category leader IBM. Although these Market Impact comparisons suggest that Atlassian has not made a dent in the ALM market, which is dominated by mega-vendors such as IBM, HP, and Microsoft, Atlassian’s Customer Sentiment scores suggest otherwise. Ovum’s end-user survey results have placed Atlassian as the leader on the Customer Sentiment scale. This also reflects the popularity of Atlassian’s offerings and the market perception surrounding them. Ovum believes Atlassian will benefit from the flux in the ALM space, especially from trends such as SaaS ALM and enterprise adoption of distributed version-control systems.
Figure 9: Atlassian ALM components radar
Source: Ovum O V U M
Atlassian’s ALM portfolio comprises Atlassian JIRA, GreenHopper, Confluence, FishEye, Crucible, Bamboo, Clover, Bitbucket, and JIRA Studio (the hosted Agile development suite made up of JIRA, GreenHopper, FishEye, Crucible, Confluence, and Bamboo with Subversion as the embedded suite repository). Salient aspects of Atlassian’s ALM offering include single shared repository-based architecture, wiki-based collaboration and knowledge management, and strong defect and issue management capabilities. The offering lacks native analytics functionality, and
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 30 This report is a licensed product and is not to be photocopied requirements definition and management are other improvement areas. However, Atlassian’s offering has a strong developer focus insofar as most of the tools listed above are aimed at practitioners. Furthermore, Atlassian has shifted direction from being a developer tools vendor to offering a third-party hosted/SaaS ALM solution; although many functionality gaps remain in Atlassian’s solution, overall the vendor’s strategy on greater Agile adoption, enterprise adoption of DVCS, and SaaS ALM is likely to pay off.
Atlassian has embarked on the next phase of growth. The privately held Australian ALM vendor recently attracted €42,000,000 investment from Accel Partners. The company plans to use these funds to expand its presence in Europe. The EMEA region currently accounts for 40% of Atlassian’s revenue. Atlassian currently has over 8,000 customers in Europe. With a global customer base of 22,000 across 138 countries, Atlassian has a strong presence in industry verticals such as manufacturing, financial services, professional services, public sector and education, and healthcare. Nearly 70% of Atlassian customers are small to medium-sized enterprises (SMEs); however, the vendor aims to penetrate the large-enterprise segment further in the near term.
In the past two years, Atlassian has acquired GreenHopper (June 2009), and Bitbucket (September 2010). The GreenHopper acquisition was Atlassian’s foray into the Agile project management space; GreenHopper was a JIRA add-on offered by GreenPepper Software. Atlassian acquired Bitbucket in September 2010 to widen its support for the Mercurial distributed version control system (DVCS).
Recommendation: Explore
Atlassian earns an Explore rating primarily due to its moderate Technology score and high Customer Sentiment score. Although Atlassian’s products are not the best in class in either breadth of coverage or feature depth, the vendor has been able to carve out a niche for itself in the ALM market, which is evident from its large globally distributed installed base and impressive vertical presence. Notably, the vendor has created a large third-party developer ecosystem, and the list of its third-party plug-ins is impressive. Atlassian’s ALM portfolio has evolved in line with what Ovum believes is the way forward for Agile and globally distributed software development teams in SMEs. Atlassian’s tool set will also find favor with teams of 10 members or less.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 31 This report is a licensed product and is not to be photocopied CollabNet
Figure 10: CollabNet ALM radars
Source: Ovum O V U M
CollabNet is an emerging ALM vendor best known as the creator of Subversion, the popular open source repository. CollabNet’s Technology scores are above average In dimensions such as collaboration and internal and external tool interoperability; however, the vendor’s performance has been below par on parameters such as solution architecture, breadth of lifecycle discipline coverage, and architecture modeling support. CollabNet’s Market Impact score is predictably low in a market dominated by heavyweights such as IBM, Microsoft, and HP. However, the vendor has recorded double-digit revenue growth and still has ample room for expansion. CollabNet’s
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 32 This report is a licensed product and is not to be photocopied performance on the Customer Sentiment radar is close to the group average on most counts, and slightly below par on product quality, customer support, and portfolio depth.
Figure 11: CollabNet ALM Components radar
Source: Ovum O V U M
CollabNet’s ALM strategy has always revolved around the cloud to support globally distributed development efforts. CollabNet Subversion was the first major step in this direction, a centralized artifacts and source code repository with a web interface. CollabNet’s ALM portfolio now includes TeamForge, the hosted planning and lifecycle management solution; ScrumWorks, the Agile project planning tool; Subversion Edge, for managing globally distributed Subversion instances; and Lab Management, for managing development and test environments in the data center or in public clouds. CollabNet’s offering would benefit from enhanced reporting and analytics capabilities. The vendor has also sought to fill key gaps in functionality through partnerships, a recent example being its partnership with HP Software to round out QA and requirements definition and management functionality.
CollabNet has made some key acquisitions at various stages of its history. The flagship offering, CollabNet TeamForge, came about after SourceForge Enterprise (an on-premise solution acquired from VA Software in 2007) and CollabNet Enterprise (a hosted planning and lifecycle management
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 33 This report is a licensed product and is not to be photocopied tool) were merged. The vendor retired CollabNet Enterprise after migrating existing customers to TeamForge. The acquisition of Danube in early 2010 was another significant milestone; it brought the ScrumWorks Agile project planning and management tool to CollabNet, addressing a critical gap in the vendor’s Agile strategy. Furthermore, in late 2010 CollabNet acquired Codesion, the largest third-party hosting service for Subversion.
CollabNet has an installed base of over 6,000 organizations, of which 700 are TeamForge clients. CollabNet has a significant presence in the large-enterprise segment, and many of those deployments are globally distributed and enterprise wide, with 10,000 or more users in each case. This is noteworthy as CollabNet is an emerging ALM vendor and lacks the scale that market leaders such as IBM, Microsoft, and HP enjoy. CollabNet’s primary market in terms of geography is consists of the two Americas; it has few clients in EMEA and Asia-Pacific. Most CollabNet clients come from manufacturing and financial services verticals. The vendor also has significant presence among ISVs.
Recommendation: Explore
CollabNet gets an Explore rating due to its moderate Technology score. CollabNet is a strong contender in areas such as hosted and SaaS ALM, project planning and lifecycle management, and delivery management. However, its less-than-average score across important pieces of the ALM portfolio, such as requirements definition, requirements management, software assurance integrations, and quality management, led Ovum to assign this rating. However, CollabNet deserves a closer look from enterprises that struggle to manage globally distributed development efforts, as well as from ISVs.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 34 This report is a licensed product and is not to be photocopied HP
Figure 12: ALM radars
Source: Ovum O V U M
HP is one of the leaders of the ALM market. HP’s Technology and Market Impact radars reflect the vendor’s relative position in this market. On the Technology front, HP has secured the fourth position overall, behind IBM, MKS, and Microsoft. On the Customer Sentiment scale, HP’s performance is close to the average in some dimensions, and below average in dimensions such as product usage, client engagement, service levels, and customer support.
On the Technology scale HP has below-par scores in dimensions such as collaboration, change and configuration management, and project management. Ovum expects that HP’s recent
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 35 This report is a licensed product and is not to be photocopied partnership with CollabNet will help the vendor enhance capabilities in the CCM space. The Market Impact radar shows vendors’ positions relative to market leader IBM, and HP’s performance is above average in all dimensions except revenue growth. The vendor’s overall position on this scale is third, behind IBM and Microsoft.
Figure 13: HP ALM Components radar
Source: Ovum O V U M
HP’s suite of application lifecycle products falls under its business technology optimization portfolio. The ALM suite contains many point tools, which are strung together by HP’s latest ALM platform, HP ALM v11.0, to create an end-to-end offering that not only caters to IT practitioner roles such as developers and testers, but also provides project and portfolio management, planning capabilities, and lifecycle visibility and traceability to managers. In addition, it provides integration with enterprise systems such as project portfolio management, business process management, and SOA governance on one end, and with DevOps tools on the other end. Key products in HP’s ALM portfolio (in addition to HP ALM 11) include HP Quality Center 11, HP Performance Center 11, HP Sprinter, HP Unified Functional Testing 11, HP QC Agile Accelerator, and HP Test Data Management.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 36 This report is a licensed product and is not to be photocopied In addition to quality management, HP QC 11 serves as the solution repository and offers the requirements management, defect management, and software delivery management aspects of HP ALM 11. HP Sprinter, a new manual testing tool, is also part of this release. HP Unified Functional Testing 11 is a new product that combines HP QTP with HP Service Test (integration testing) through a single interface and offers unified reporting capabilities. HP Performance Center 11 provides performance testing capabilities for web-based and legacy applications and, as part of the ALM solution, enables end-to-end traceability of artifacts. HP QC Agile Accelerator is an Agile project management tool built from the ground up. This recent addition to HP’s ALM portfolio has earned the vendor a nearly perfect score in the enterprise Agile Technology dimension.
Overall, HP’s ALM strategy seems to center around consolidation of existing leading positions in key areas including quality management, application security, and moving quickly to capture market share in ALM hot spots such as Agile development. The vendor has also focused on filling functionality gaps through in-house development and partnerships. HP is a mega-IT vendor and, as such, the company has made quite a few big-ticket acquisitions in the past few years. The acquisition of Fortify in September 2010 is relevant from an ALM perspective. The Fortify acquisition provided HP with tools to address application security concerns in the development lifecycle. HP has a keen sense of the rapidly changing software development landscape, as well as applications. Many new capabilities added to existing tools such as QTP and HP Service Test, and new products such as HP LoadRunner TruClient, HP Sprinter, and HP QC Agile Accelerator, led us to believe that HP aims to stay abreast with the complex, ever-changing nature of modern applications.
Recommendation: Shortlist
HP has advanced on Ovum’s ranking from the Consider category in 2009’s Decision Matrix to the Shortlist category this year. The vendor’s good scores in the Technology dimension (marginally lower than Microsoft’s), near-average Customer Sentiment score, and third-highest Market Impact score have led to its Shortlist rating. HP’s ALM offering merits attention from organizations across size bands, geographies, and industry verticals. The offering’s support for a wide range of processes and project types, coupled with HP’s comprehensive QA functionality, make it a good fit for enterprises.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 37 This report is a licensed product and is not to be photocopied IBM
Figure 14: IBM ALM radars
Source: Ovum O V U M
IBM is among the largest vendors in the ALM space, and its Market Impact scores reflect its status as an application lifecycle management heavyweight. It registers the highest Technology score, ahead of MKS, Microsoft, HP, and Polarion. IBM scores the highest or close to the highest among the 12 ALM vendors profiled in this report across most Technology dimensions, including breadth of lifecycle discipline coverage, solution architecture, collaboration, application project portfolio management, project management, requirements management, change and configuration management, quality management, and delivery management among others. In terms of its
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 38 This report is a licensed product and is not to be photocopied Market Impact, IBM is the undisputed leader, much ahead of closest competitors Microsoft, and HP. In this research exercise the Customer Sentiment scores of the large ALM vendors have mostly been unimpressive, and same is the case with IBM. Its score is above average on just the financial stability dimension of Customer Sentiment.
Figure 15: IBM ALM components radar
Source: Ovum O V U M
Predictably, IBM has the largest ALM portfolio of all vendors included in this report, insofar as it has multiple products for lifecycle disciplines such as requirements management and change and configuration management. The following is a partial list of IBM Rational products:
Rational Team Concert – project management and lifecycle visibility
Rational DOORS – requirements management
Rational Quality Manager – test management
Rational Test Lab Manager – test environment provisioning and management
Rational ClearCase – CCM
Rational ClearQuest – defects and issue management
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 39 This report is a licensed product and is not to be photocopied Rational Build Forge – build and release management
Rational AppScan – application security
A key aspect of IBM’s ALM strategy is Jazz. Jazz is an architecture framework that aims to address tool interoperability, one of the biggest concerns regarding ALM. A few years ago IBM announced the Jazz Initiative, based on the understanding that organizations’ IT environments typically have tools from multiple vendors, and that making these tools work well together, orchestrated by a common framework, is an unsolved challenge given that often even tools from the same vendor have poor interoperability. Jazz supports a wide range of operating systems (including mainframes), and has a web services (ReST)-based federated architecture. Jazz’s associated industry body is the Open Services for Lifecycle Collaboration (OSLC), an IBM-led community that is working towards developing a common architecture framework to enable better integration among lifecycle tools. However, this initiative is still at an early stage, and industry adoption of OSLC specifications will depend on many factors other than technical. So far, OSLC has not attracted any other mainstream ALM vendors, and the success of this initiative cannot be ascertained at this point.
IBM is in the process of migrating legacy Rational products to the Jazz platform. (Rational here refers to all products that are part of IBM Rational, not to be confused with products that came to IBM through the Rational Software acquisition in 2002.) The company has also developed new products native to Jazz. These new products include Rational Team Concert, Rational Requirements Composer, and Rational Quality Manager.
IBM is a large IT vendor, and acquisitions play a major role in shaping its product portfolio and growth in the markets in which it operates. Significant acquisitions from an ALM standpoint include Rational Software (2003), BuildForge (2006), WatchFire (2007), Telelogic AB (2008), and Ounce Labs (2009).
Recommendation: Shortlist
High scores on the Technology and Market Impact scales have put IBM in the Shortlist category in this Decision Matrix. IBM’s ALM portfolio offers all-around functionality in virtually all ALM disciplines. The solution set supports the entire range of use cases, project types, and processes found in enterprise organizations with mature software development functions. Although IBM also sells to small enterprises and small development teams, the feature set will likely be expensive and overkill for their needs.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 40 This report is a licensed product and is not to be photocopied Micro Focus
Figure 16: Micro Focus ALM radars
Source: Ovum O V U M
Micro Focus, an enterprise applications management and legacy modernization vendor, entered the ALM space by acquiring Borland and the software quality management product line from Compuware in 2009. Micro Focus scored well in the Technology dimensions, such as breadth of lifecycle discipline coverage, architecture modeling support, delivery management, requirements definition, requirements management, and quality management, among others. Micro Focus’s overall position on the Technology scale is eighth, mainly due to below-average performance in several ALM disciplines. For instance, Micro Focus received below-par scores in key areas such
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 41 This report is a licensed product and is not to be photocopied as application project portfolio management, project management, enterprise Agile, and application security. Micro Focus’s Market Impact scores reflect its position in the market and the mindshare that the vendor has managed to capture. Micro Focus is the category leader in the Market Impact vertical presence dimension, due the fact that it has the most uniform vertical spread of revenues among all profiled vendors. Micro Focus placed fourth jointly with Rally Software on the Market Impact scale, behind IBM, Microsoft, and HP. Ovum anticipates Micro Focus launching its acquired Borland Agile ALM products, but its prolonged withdrawal from this part of the market is a weakness.
For this Decision Matrix, Micro Focus was not rated by enough customers for Ovum to aggregate and present statistically significant Customer Sentiment scores.
Micro Focus’s ALM portfolio includes Silk Suite (software quality management), StarTeam (change and configuration management), Caliber suite (requirements definition and requirements management), and Together (model-driven development). The vendor has a global presence, with the Americas being the primary geography. Micro Focus’s installed base is comprised mostly of large and medium-sized enterprises, with a relatively small percentage being small businesses. Significant customer verticals include media and communications, healthcare, professional services, financial services, and public sector, among others.
Recommendation: Consider
Moderate Technology and Market Impact scores earn Micro Focus a Consider rating. Micro Focus is a leader in categories such as requirements management, software quality management, and delivery management; however, less-than-average performance in other key ALM disciplines led us to assign this rating. Micro Focus’s offering must be considered by enterprises, in particular by those that wish to approach ALM from a software quality and reliability perspective.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 42 This report is a licensed product and is not to be photocopied Microsoft
Figure 17: Microsoft ALM radars
Source: Ovum O V U M
Research for the ALM Decision Matrix report places Microsoft in the leaders bracket. Microsoft is second only to IBM in terms of ALM market leadership. On the Technology scale, Microsoft placed third, behind IBM and MKS. The vendor scored well in all Technology dimensions except requirements definition and requirements management.
Microsoft is the second-place vendor on the Market Impact scale; Microsoft’s Market Impact score is predictably behind IBM, mainly due to the difference in revenues these two vendors earn from the ALM market. Microsoft’s performance on the Customer Sentiment radar is close to the average
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 43 This report is a licensed product and is not to be photocopied in most dimensions, and better than average in product integrations and financial stability. Notably, Microsoft’s product quality rating is below par. Overall, Microsoft’s performance on the Customer Sentiment scale is better than that of the other three Shortlist vendors (IBM, MKS, and HP). This goes to show that Microsoft has managed customer expectations better than the other market leaders.
Microsoft’s ALM portfolio revolves around the Visual Studio IDE; with the current version, Microsoft has done away with the various role-based editions, and now offers Visual Studio 2010 in Ultimate, Premium, and Professional editions. There is a separate VS2010 Test Professional edition for QA teams; it offers a non-programmer environment for QA professionals.
Over the past few years, Microsoft has not only managed to create space for itself in the ALM market, but also emerged as a market leader. Ovum’s 2009 edition of the ALM Decision Matrix also recognized Microsoft as an ALM market leader. Microsoft’s quick rise to the top in the ALM domain is not surprising. Microsoft Visual Studio IDE is one of the leading development environments in the market, and has earned Microsoft a large and globally distributed developer community. Given Microsoft’s leading position in the operating systems business (both client and server) and its ownership of the .Net technology stack, it was apparent that Microsoft would be an ALM market leader as soon as it focused on this market. Microsoft’s ALM offering, previously known as Visual Studio Team System 2010, is now simply called Visual Studio 2010. Additionally, all editions of the VS2010 offering now include the MSDN subscription. Microsoft’s ALM strategy has clearly moved away from selling point tools to selling an integrated ALM platform.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 44 This report is a licensed product and is not to be photocopied Figure 18: Microsoft ALM components radar
Source: Ovum O V U M
Recommendation: Shortlist
Good performance on all three scales (Technology, Market Impact, and Customer Sentiment) led to Microsoft’s Shortlist rating. The vendor’s ALM offering is comprehensive, supporting all roles, project types, and processes found in an enterprise-class development function. The solution merits closest attention by organizations that have standardized on the .Net stack; however, the Visual Studio IDE also supports non-.Net development. Organizations developing on and for the Microsoft cloud platform should also consider VS2010 for its deep integration with Azure.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 45 This report is a licensed product and is not to be photocopied MKS
Figure 19: MKS ALM radars
Source: Ovum O V U M
MKS secured the second-highest score on the Technology scale in this edition of the ALM Decision Matrix, and that led to the vendor being placed in the Shortlist category. However, the vendor’s performance on the Customer Sentiment radar is less than satisfactory. Significantly, like other market leaders, MKS received less-than-average scores in a few Customer Sentiment dimensions, such as portfolio depth, financial stability, product integration, and customer support. However, MKS managed a par score in the client engagement dimension. MKS’s overall Customer Sentiment position is relatively poor compared with the leaders in this dimension, but the vendor
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 46 This report is a licensed product and is not to be photocopied did better than two other Shortlist vendors, IBM and HP. Its score is marginally behind Microsoft, the Customer Sentiment leader among Shortlist vendors. Thus, Ovum is inclined to believe that MKS customers’ perception of the vendor is largely favorable.
MKS’s performance on the Market Impact scale is also less than satisfactory. This is attributable to MKS’s highly specialized target market, organic product development, and growth. MKS targets organizations with complex IT environments and, consequently, highly specialized business requirements that MKS addresses through its ALM offering. MKS customers are typically large enterprises in the Americas and EMEA, from industry verticals such as automotive, finance, electronics and high technology, medical devices, aerospace and defense, healthcare, and government. This led to MKS losing points in Market Impact dimensions size-band presence and regional presence, which typically favor vendors with equitable distribution in terms of customer size and geographical presence. MKS also lost points in the revenue and revenue growth dimensions compared with other vendors included in this report.
MKS’s ALM offering is MKS Integrity, a monolithic platform that covers all ALM disciplines. MKS’s solution scored less-than-average marks only in the architecture modeling support dimension. In all other Technology categories the vendor’s score is at least average, if not better.
Figure 20: MKS ALM components radar
Source: Ovum O V U M
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 47 This report is a licensed product and is not to be photocopied Recommendation: Shortlist
MKS’s close-to-highest score on the Technology scale and moderate score on the Customer Sentiment scale placed the vendor in the Shortlist category. The Customer Sentiment and Market Impact scores are lower than would be expected of an ALM vendor of MKS’s stature, largely attributable to MKS’s poor level of customer awareness. However, other leaders such as HP and IBM Rational scored even lower. There is little to doubt the comprehensive nature of MKS’s offering and its relevance to diverse ALM requirements, including Agile development.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 48 This report is a licensed product and is not to be photocopied Polarion
Figure 21: Polarion ALM radars
Source: Ovum O V U M
Polarion is the only emerging ALM vendor with an impressive Technology score. Polarion’s score is marginally behind Shortlist vendors Microsoft and HP, and the vendor placed fifth overall on the Technology scale. This is mainly due to Polarion’s wide coverage of the ALM disciplines and above-average scores in all ALM disciplines covered. Polarion offers a well-rounded ALM solution, and the only missing element is tool support for architecture modeling. Polarion’s performance on the Technology scale is impressive, particularly in change and configuration management, enterprise Agile, project management, solution architecture, and application security. The vendor
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 49 This report is a licensed product and is not to be photocopied scored highly on the Customer Sentiment scale as well, an area in which many established ALM vendors have faltered.
Polarion received above-average scores in all Customer Sentiment dimensions, and is the category leader in dimensions such as client engagement, service capabilities, product quality, and customer support. On the whole, Polarion placed second on the Customer Sentiment scale, marginally behind Atlassian. On the Market Impact scale, Polarion’s performance is predictably poor. Polarion is the leader in terms of revenue growth; however, the growth came off a small base, hence the overall impact is low. Polarion does not have much history behind it; however, if the vendor continues to grow at a high double-digit rate (as a percentage of revenue growth year- on-year), it will soon be in a position to challenge bigger, established ALM vendors and have a greater impact in the market.
Polarion’s ALM offering, Polarion ALM Enterprise, was first shipped in 2005. In this short time the vendor has put together an impressive solution, underpinned by a single solution-wide repository (Subversion). The offering is competent in most areas, as can be seen from Polarion’s ALM radars; however, certain gaps exist, such as lack of requirements modeling capability and lack of feature depth in application project portfolio management. Notably, the solution was developed from the ground up, caters well to the needs of SMEs, and is currently targeting the enterprise market.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 50 This report is a licensed product and is not to be photocopied Figure 22: Polarion ALM components radar
Source: Ovum O V U M
Recommendation: Consider
Polarion’s impressive performance on the Technology and Customer Sentiment scales led Ovum to assign the Consider rating. The vendor’s ALM revenues are miniscule compared with the market leaders; however, that does not take anything away from Polarion’s offering. Polarion’s ALM solution merits closer attention from organizations that use Subversion as a repository and are yet to deploy ALM, as well as from organizations engaged in product engineering.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 51 This report is a licensed product and is not to be photocopied Rally Software
Figure 23: Rally Software ALM radars
Source: Ovum O V U M
Rally Software was placed in the Explore category in the previous edition of the ALM Decision Matrix; the vendor managed to retain its position in this edition of the report. This is in line with the growing market adoption that Rally Software has enjoyed over the past couple of years. Rally’s performance on the Technology scale is along expected lines; the vendor lost points in areas that it does not cover, such as application security, delivery management, change and configuration management, and architecture modeling support. It lost points in other Technology dimensions in which its scores were below par. Rally scored well in dimensions that it does cover, such as project management, requirements management, and enterprise Agile. On the whole, Rally’s
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 52 This report is a licensed product and is not to be photocopied position on the Technology axis is second to last, slightly better than Atlassian. Ovum looks at breadth of ALM discipline coverage and feature depth in each discipline when evaluating vendors on the Technology scale; hence, vendors that operate in niche areas or offer only a subset of overall ALM functionality lose points. Rally is primarily an Agile ALM vendor, but with lifecycle traceability, Agile planning, and project management as major functionality areas, its offering addresses a broad and growing market. Ovum is also of the view that Rally Software is a leader in the ALM sub-segment (Agile ALM) in which it operates, and its impact on the ALM market is bound to grow in the near to medium term.
Figure 24: Rally Software ALM components radar
Source: Ovum O V U M
Rally has executed its market strategy well; hence, the vendor finds itself a leader in the Agile ALM space. Rally’s score on the Market Impact scale is predictably low compared with market leaders IBM, Microsoft, and HP. However, Rally has the highest impact among vendors placed in the Explore category, and is tied with Micro Focus for fourth place. In Ovum’s view, Rally’s Market Impact is also comparable to other established ALM vendors such as MKS and Serena Software.
Rally performed well on the Customer Sentiment scale; this goes to show that Rally has gauged the market well and captured mind share in its target market. The vendor has par or above-par
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 53 This report is a licensed product and is not to be photocopied scores in all Customer Sentiment dimensions, and leads the pack on dimensions such as product quality, client engagement, and service levels. This is significant because Rally has an installed base of 2,500 organizations in 60 countries.
Rally’s ALM solution is offered in many editions; the most comprehensive version is aptly named Rally Ultimate Edition. The vendor started shipping its Agile planning and project management solution in 2004; the solution has been on the market for nearly seven years, and this has enabled Rally to fine tune its product strategy and leverage the trends in the market to its advantage. Rally has also been able to rapidly integrate functionality from acquired solutions such as 6th Sense Analytics and AgileZen into the core platform. Having accomplished Agile planning and project management, the vendor now plans to expand into adjacent upstream areas such as application project portfolio management, and deepen its vertical expertise by offering vertical-specific functionality on top of its core Agile ALM platform. How the market reacts to Agile ALM offerings with vertical-specific functionality remains to be seen.
Recommendation: Explore
Rally’s low score on the Technology scale led us to assign the Explore rating. However, as the discussion above suggests, Rally’s relevance to the organizations practicing Agile development is on the rise. The vendor now seeks to help organizations scale their Agile practices from team or department level to enterprise wide. Ovum expects that Rally’s influence in the Agile ALM market will grow and the vendor will break into the Consider category in the next edition of the ALM Decision Matrix.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 54 This report is a licensed product and is not to be photocopied Serena Software
Figure 25: Serena Software ALM radars
Source: Ovum O V U M
Serena Software is an established ALM vendor with significant market impact and a large installed base. Serena’s main business is developing and selling tools that support the application lifecycle, from an automation perspective and from lifecycle visibility and artifact/work-item traceability perspective. The vendor offers tools for distributed and mainframe environments. Serena was placed in the Consider category in the previous edition of the ALM Decision Matrix; the vendor has managed to retain its position in this edition of the report.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 55 This report is a licensed product and is not to be photocopied On the Technology scale Serena performed better than most vendors in the areas that it covers, such as application project portfolio management, project management, requirements management, change and configuration management, and enterprise Agile. However, the vendor lost points in key areas such as delivery management and solution architecture due to lack of support for continuous integration, continuous testing, and product lifecycle management integration. Serena also scores below par in the collaboration dimension due to lack of wiki-based knowledge management capability. In the quality management dimension Serena lost points due to lack of support for test data management and unit test management. In all other ALM disciplines that Serena covers, its score is above par. Overall, Serena placed sixth on the Technology scale, behind IBM, MKS, Microsoft, HP, and Polarion.
Serena’s performance on the Market Impact scale is along expected lines; the vendor is predictably behind market share leaders IBM, Microsoft, and HP. That Serena has managed to hold onto its market share in a regressive economic climate is commendable. On the Customer Sentiment scale, Serena has on-par scores in most dimensions and a below-par score in product quality. Notably, Serena Software has managed customer expectations well; this is also reflected in the vendor’s high scores on customer support and service capabilities dimensions. In the previous edition of the ALM Decision Matrix, Ovum opined that Serena could pose a serious threat to the ALM market leaders due to its strong market presence and technology leadership; however, the vendor needed to improve its Customer Sentiment ratings. In this edition of the report, Serena has come down a few notches on the Market Impact scale; however, it managed to better its Customer Sentiment rating and placed fourth on the Sentiment scale behind Atlassian, Polarion, and Rally Software.
Over the past few years, Serena’s ALM strategy has undergone some changes. Until late 2010, the vendor was focused on composite applications, commonly known as mashups, and had a product called Serena Business Mashups, which aimed to facilitate cloud-based composite application development. Serena has shifted focus back to its main business – application lifecycle management. Serena’s ALM message is focused on demand management, development management, and release management, with overall governance and automation of the application lifecycle offered through Serena Business Manager. It is interesting to note that Serena takes a process-oriented view of the application lifecycle and considers ALM a set of processes and activities that need to be orchestrated for delivering applications that meet business requirements on budget and on schedule. Key to this approach is Serena Business Manager, a product that not only acts as the process management and integration platform, but also provides the bulk of the reporting functionality inherent in Serena’s ALM offering. Other key products include Serena Dimensions CM, Serena Dimensions RM, Serena PPM, Serena Prototype Composer, Serena PVCS, and Serena Agile.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 56 This report is a licensed product and is not to be photocopied Figure 26: Serena Software ALM component radar
Source: Ovum O V U M
Recommendation: Consider
Serena’s moderate Technology score coupled with its significant Customer Sentiment score earned the vendor a Consider rating. As opined in the previous edition of the ALM Decision Matrix, Serena is well placed to challenge the Shortlist vendors for market leadership. Serena’s large installed base is comprised mainly of large and medium-sized enterprises from financial services, professional services, public sector, manufacturing, and retail/wholesale. In our view, Serena’s process-oriented approach to ALM will resonate well with its target market, and may well elevate the vendor to the ALM market leaders’ category in future.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 57 This report is a licensed product and is not to be photocopied TechExcel
Figure 27: TechExcel ALM radars
Source: Ovum O V U M
TechExcel was placed in the Explore category in the previous ALM Decision Matrix; the vendor has been placed in the Consider category in this edition of the report. This is mainly due to TechExcel’s above-average Technology scores in all dimensions that the vendor covers through its ALM offering. TechExcel has below-par scores in Technology dimensions such as change and configuration management, application security, and delivery management; however, the vendor is not focused on these areas and does not offer native tool support for these disciplines. Ovum rates TechExcel’s ALM offering highly, and this is reflected in the vendor’s scores in individual Technology dimensions. However, the Decision Matrix report looks at both feature depth and
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 58 This report is a licensed product and is not to be photocopied breadth of functionality offered; hence, TechExcel lost some crucial points on the Technology scale. Overall, the vendor placed seventh on the Technology axis, behind IBM, MKS, Microsoft, HP, Polarion, and Serena.
For this Decision Matrix, TechExcel was not rated by enough customers for Ovum to aggregate and present statistically significant Customer Sentiment scores.
TechExcel’s Market Impact is two orders of magnitude lower than that of market leaders IBM, Microsoft, and HP. Although the gulf between TechExcel and the market leaders is wide and will not be bridged in the near to medium term, the ALM tools market has weathered the economic slump and TechExcel has ample room for growth. Moreover, the vendor registered high-double- digit revenue growth over the past year, and Ovum expects this growth to continue. On the whole, Ovum expects TechExcel to grow at a rapid pace for the next few years. This growth will likely come from TechExcel’s investments in third-party hosted and SaaS variants of its ALM offering, DevSuite.
DevSuite comprises four modules: DevSpec, DevPlan, DevTrack, and DevTest. In addition, TechExcel offers Agile Studio, which comprises DevPlan and DevTrack, an offering customized for teams practicing Agile development. DevTest and DevTrack are bundled into a DevTest Studio offering aimed at QA teams. Overall, TechExcel’s solution provides tight integration among individual modules, offers repository support for unstructured data, and scales well to support a large number of concurrent users.
Recommendation: Consider
TechExcel’s impressive scores in the Technology dimensions that it covers earned the vendor a Consider rating. In Ovum’s view, TechExcel’s solution will be relevant to and merits close attention from mid-market enterprises and independent software vendors. The solution has been proven in many ISVs’ environments, particularly with PC and console games developers.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 59 This report is a licensed product and is not to be photocopied ThoughtWorks Studios
Figure 28: ThoughtWorks Studios ALM radars
Source: Ovum O V U M
ThoughtWorks Studios is a new entrant to the Ovum ALM Decision Matrix, having entered the ALM market with a comprehensive ALM solution. ThoughtWorks Studios offers an Agile ALM solution that not only covers Agile planning, project management, and reporting, but also provides tool support for delivery management in Agile environments. ThoughtWorks logged above-par scores in most of the Technology categories that it covers, namely project management, delivery management, requirements definition, and enterprise Agile. In high-level Technology dimensions such as breadth of lifecycle discipline coverage, solution architecture, collaboration, and internal
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 60 This report is a licensed product and is not to be photocopied tool interoperability, ThoughtWorks’ scores are close to, if not better than the average across vendors.
ThoughtWorks lost points in the requirements management dimension as the vendor does not yet support key features such as requirements-based test case generation and visual requirements modeling. Overall, ThoughtWorks Studios placed ninth on the Technology scale, ahead of the other three vendors with the Explore rating (Atlassian, CollabNet, and Rally Software).
ThoughtWorks Studios’ ALM offering comprises:
Mingle – Agile project management, requirements management, release planning, defect and issue tracking, and collaboration
Go – configuration management, build management, and release management
Twist – test management and functional testing
For this Decision Matrix, ThoughtWorks Studios was not rated by enough customers for Ovum to aggregate and present statistically significant Customer Sentiment scores.
On the Market Impact scale, a wide gulf exists between ThoughtWorks Studios and the market leaders, and it is unlikely that ThoughtWorks Studios will cover the gap in the near term. However, The vendor has executed its marketing strategy well, and been successful in capturing mindshare in this expanding market. This is due in no small measure to the strength of ThoughtWorks Studios’ Agile ALM offering, which was developed from scratch to support Agile development practices. Ovum expects that the vendor will continue on the growth path and be at the forefront of the challenger league by the time Ovum publishes the next edition of the ALM Decision Matrix. Furthermore, ThoughtWorks Studios has been acquiring customers in the highly competitive ALM market at a rapid rate, showing how traditional ALM vendors have been slow to deliver on the Agile front.
Recommendation: Explore
ThoughtWorks Studios’ moderate score on the Technology scale have led us to assign an Explore rating. However, Ovum believes ThoughtWorks Studios’ offering is a comprehensive ALM solution for organizations practicing Agile, and merits special attention from such enterprises. The solution will be especially relevant to enterprises that have failed or hit roadblocks in their Agile initiative, as its tools can be adopted incrementally with demonstrable value at each step.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 61 This report is a licensed product and is not to be photocopied APPENDIX
Ovum ratings Shortlist: This category represents the leading solutions, which Ovum believes are worthy of a place on most technology selection shortlists. The vendor has established a commanding market position with a product that is widely accepted as best of breed.
Consider: The vendors in this category have a good market positioning and are selling and marketing the product well. The products offer competitive functionality and good price-performance proposition, and should be considered as part of the technology selection.
Explore: Solutions in this category have less-broad applicability, and may have limitations in terms of the product’s functionality or the vendor’s execution capability. However, they will still be suitable to meet specific requirements, and may be worth exploring as part of the technology selection.
Methodology Market analysis: An analysis of vendors’ financial performance and impact on the market.
Customer survey: A survey of over 115 IT managers and CIOs.
Technology analysis: Ovum’s assessment of vendors’ technology based on the Ovum ALM Features Matrix.
Extended methodology Ovum assesses ALM vendors in three core areas: Technology, Market Impact, and Customer Sentiment. Each of the three dimensions consists of 5–12 specific criteria, and together they present a comprehensive view of various vendor offerings in the ALM space.
Technology
Ovum analysts score each vendor across 16 major Technology criteria on a normalized scale of 0–10. These scores are arrived at by assessing the completeness of the feature set and functionality across each of these areas. The first six are related to generic suite architecture, while the next ten are specific to the ALM component being assessed. The 16 rating areas used for assessing the vendors in the ALM market are:
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 62 This report is a licensed product and is not to be photocopied Breadth of lifecycle discipline coverage
Solution architecture
Internal tool interoperability
External tool interoperability
Collaboration
Architecture modeling support
Application project portfolio management
Project management
Requirements definition
Requirements management
Change and configuration management
Software assurance integrations
Quality management
Enterprise Agile
Application security
Delivery management
For further details see the Ovum’s Technology Evaluation and Comparison report, Software Lifecycle Management 2011.
Customer Sentiment
Ovum surveyed over 115 customers of the application lifecycle management vendors profiled in this report across the Americas, EMEA, and Asia-Pacific to understand their views on the vendor of their choice. The surveyed enterprises were also asked to rate the technology vendors that they have worked with in the past. Ovum provides an average rating in each of the following categories:
Product quality – perception of the quality of the vendor’s products
Customer support – the quality of the vendor’s business/technical support offerings
Service capabilities – the quality of a vendor’s particular service offerings (consulting, integration, maintenance, management)
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 63 This report is a licensed product and is not to be photocopied Vertical specialization – the extent to which the vendor offers industry-specific solutions and expertise
Portfolio depth – the CIO/IT manager’s perception of the depth of the vendor’s product portfolio
Service levels – the quality of a vendor’s service-level agreements (SLAs) and its ability to meet them
Financial stability – how financially stable the enterprise believes the vendor is
Client engagement – the effectiveness of the vendor’s sales force and the CIO/IT manager’s perception of its channel to market
Product integration – the customer organization’s perception of internal tool interoperability and third-party integrations that the solution offers
Product usage – how often the vendor’s products are used in the organization
Ovum now invites vendors to supply customer references, which we then contact independently to complete the survey. This is particularly helpful to the smaller vendors; however, we do not provide a Customer Sentiment score when vendor-supplied references account for more than 50% of the customer completions.
Market Impact
Ovum analysts use data collected through primary and secondary research to determine a vendor’s global market impact. Market impact is measured across six categories, each of which has a maximum score of 10:
Revenues – Each vendor’s global ALM revenues are calculated as a percentage of the market leader's. This percentage is then represented on a normalized scale of 0–10.
Revenue growth – Each vendor’s revenue growth estimate for the next 12 months is calculated as a percentage of the growth rate of the fastest-growing company in the market. The percentage is then multiplied by 10 and rounded up to the nearest integer.
Geographical penetration – Ovum determines each vendor’s revenues in three regions: the Americas; Europe, the Middle East, and Africa (EMEA); and Asia-Pacific. These revenues are calculated as a percentage of the market leader's revenues in each region, multiplied by 10, and then rounded to the nearest integer. The vendor’s overall geographical reach score is the average of these three values.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 64 This report is a licensed product and is not to be photocopied Recognition – Survey respondents are asked to select ALM vendors that they are aware of. The percentage of the vendor recognition is then divided by the highest percentage of the vendor recognition, multiplied by 10, and rounded up to the nearest integer.
Vertical penetration – Ovum determines percentage of revenues that vendors earn from 10 verticals: communications and media, energy and utilities, financial services, government and education, healthcare, manufacturing, professional services, retail and wholesale, transportation and logistics, and others. A vendor gets a score of 1 if its revenues from a vertical are not less than 10% of its overall ALM revenues; otherwise, the vertical score is 0. The final vertical score for any vendor is the sum of its scores across all 10 verticals.
Size-band coverage – Ovum determines each vendor’s revenues in three company size bands: large enterprises (over 5,000 employees), medium-sized enterprises (1,000–4,999 employees), and small enterprises (fewer than 1,000 employees). These revenues are calculated as a percentage of the revenues of the market leader in each region, multiplied by 10, and then rounded up to the nearest integer. The vendor’s overall company size band score is the average of these three values.
Further reading Software Lifecycle Management 2011, OI00068-004
Authors
Chandranshu Singh, Senior Analyst, Ovum (IT Solutions) [email protected]
Tony Baer, Senior Analyst, Ovum (IT Solutions) [email protected]
Michael Azoff, Principal Analyst, Ovum (IT Solutions) [email protected]
Ovum Consulting We hope that the analysis in this report will help you make informed and imaginative business decisions. If you have further requirements, Ovum’s consulting team may be able to help you. For more information about Ovum’s consulting capabilities, please contact us directly at [email protected].
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 65 This report is a licensed product and is not to be photocopied Disclaimer All Rights Reserved.
No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission of the publisher, Ovum (a subsidiary company of Datamonitor plc).
The facts of this report are believed to be correct at the time of publication but cannot be guaranteed. Please note that the findings, conclusions and recommendations that Ovum delivers will be based on information gathered in good faith from both primary and secondary sources, whose accuracy we are not always in a position to guarantee. As such Ovum can accept no liability whatever for actions taken based on any information that may subsequently prove to be incorrect.
Decision Matrix: Selecting an Application Lifecycle Management Vendor © Ovum (Published 03/2011) Page 66 This report is a licensed product and is not to be photocopied