Where Does Selenium Fit in an Enterprise Quality Strategy?

Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 1

“Selenium is the de facto standard for front-end test automation of modern web technologies due to the flexible and powerful browser automation capabilities of WebDriver…It’s a sophisticated tool that isn’t easy to learn. It requires your team to have programming expertise and familiarity with object-oriented languages.”

- Sean Kenefick, Gartner, in “A Guidance Framework for Automating Your Front-End Web Testing Using Selenium”

“Currently, open-source solutions are mainly oriented toward developers and focused on web and mobile technologies, making them somewhat specialized and not appropriate for all testing needs.”

- Joachim Herschmann and Thomas Murphy, Gartner, in “Magic Quadrant for Software Test Automation”

There’s no doubt that Selenium has earned its spot as the de facto standard for web browser automation. Selenium is well-suited for web UI development teams where testing is conducted by developers or testers who are well-versed in a programming / scripting language. Yet, Selenium is not a test automation panacea. When you move beyond this sweet spot and try to scale Selenium across a large enterprise, several hurdles emerge:

• Testing end-to-end business transactions

• Conquering the learning curve

• Managing brittle scripts

• Controlling the Total Cost of Testing This paper explores those challenges—and how to address them so that Selenium can play a key role in a broader enterprise-level quality strategy.

Testing end-to-end business transactions Selenium only automates web browser interactions—it cannot test end-to-end business transactions. Even Windows-based dialogs that open during a browser session cannot be automated by Selenium. And then there all those other technologies that are commonly involved in today’s highly-distributed end-to-end business transactions…

Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 2

If you want to automate end-to-end regression test scenarios that extend from the web UI to SAP, APIs, database, mainframes, etc., you will need to extend beyond Selenium’s core capabilities. Ideally, this is achieved via a broader test automation framework that embraces Selenium and WebDriver. This way, you gain the required breadth of technology support without having to sacrifice the benefits you were seeking with Selenium.

Conquering the learning curve Writing and maintaining Selenium frameworks and scripts might not be as challenging as working with production code, but it does require a considerable amount of technical know-how. Considering that manual testing still dominates over 80% of enterprise-level testing, it’s fair to assume that the average tester has limited test automation experience—and even less programming experience. A large gap needs to be bridged for these manual testers to become productive with Selenium test automation. Even more training and resources are required for them to approach the productivity level of programmers, who can easily wield Selenium as an instrument of power. Using Selenium for test automation requires an understanding of: • One of the Selenium-supported programming/scripting languages (e.g., Java, JavaScript, Python, Ruby, C#) • The WebDriver browser driver, browser driver proxies, language bindings, and Selenium Server

Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 3

• A build and dependency management system like Maven or Gradle • A test framework (e.g., JUnit, TestNG, NUnit) • The browser DOM structure • XPath and/or CSS locators This isn’t a significant hurdle for programmers—so it’s not surprising that they are comfortable working with Selenium. Productivity declines significantly with test scripters (technical testers who have already mastered test automation), then it hits rock bottom with manual testers.

Most enterprise QA teams are dominated by testers who have a high degree of business domain knowledge, but not the technical expertise needed to translate the test requirements into code or scripts. In these situations, a huge gap must be bridged to achieve success with Selenium. Either the QA staff must become (at least basic) programmers, or you must shift testing responsibilities to the programming staff. Both options are costly: the first involves a significant training or re-staffing investment, and the second either escalates your development costs or slows your speed of innovation.

Managing brittle scripts Many organizations vastly underestimate the amount of time and effort required to resolve false positives and update tests. False positives (from “flaky tests” or “brittle scripts”) are a common struggle among Selenium users. They frequently occur because:

Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 4

• A seemingly unrelated code change broke an XPath locator • The test data used in the initial run is now “expired” • Dynamic elements did not fully load before WebDriver tried to interact with them • The test timed out before the expected interaction had time to complete • Dependent components (e.g., a third-party API or service) were unavailable or unstable, or were not populated with appropriate data In addition to the work required to resolve false positives, more test maintenance is required to keep the scripts in sync with the evolving application. Selenium does offer ways to promote reuse (e.g., by implementing Page Object classes). However, this is considered an “advanced” use case even by Selenium standards. The vast majority of Selenium scripts use a considerable amount of duplicate code because that’s often the easiest path to automation. However, you pay the price for the initial copy/paste simplicity when a commonly-used control is renamed and hundreds of test scripts need to be updated. All of this leads to what is commonly known as the “maintenance trap”— where the level of test maintenance required starts to offset the time savings achieved by automation. While programmers might be able to rapidly resolve false positives and refactor existing tests for requirement changes, testers who are new to programming will usually require considerably more time and troubleshooting to get the automation back on track. Ultimately, the less technical your Selenium testing resources are, the higher the impact of the maintenance trap.

Managing the Total Cost of Testing No software is free—you always need to consider the total cost of ownership. With Selenium, the less technical your testing team is, the more it will ultimately cost you to achieve the expected test automation results. First, Selenium requires you to invest considerable resources in building a custom test framework. The exact resource commitment required varies based on your projects and priorities. However, for some idea of what this involves, consider that it recently took a large Australian bank approximately 500 person hours to build an effective test framework for one of their core applications. This is in line with anecdotal reports (in online testing forums) that it typically takes a single person 3 months to build a framework for a medium-complexity web application. Next, you need to consider the cost of the resources required to build and maintain test scripts with Selenium. For the reasons stated in the previous section, this is a costly endeavor—and the less technical your testers, the higher the costs will be. Based on data that we collected at over 125 customer sites (large enterprises across a variety of industries), building test cases with Selenium tends to require 4x more resources than with a commercial test automation tool. The same data set indicated that, on average, maintenance was 6X greater with Selenium than with a commercial tool.

Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 5

When you consider license costs plus the costs of creating and maintaining Selenium scripts, the total cost of ownership for Selenium vs. a commercial scriptless and frameworkless test automation platform such as Tricentis Tosca looks like this:

Granted, results will vary based on the complexity of your projects, your teams’ skill levels and how well the commercial tool meets your needs. However, in most situations, Selenium testing will ultimately cost more than testing with a commercial testing tool that’s faster and easier for the average tester to master.

Final thoughts Selenium has a sweet spot: developers and technical testers focused on testing web UIs. Selenium can play a role in your enterprise-level quality strategy. However, it’s not a silver bullet. An enterprise-level quality strategy also requires the coordinated orchestration of a broader set of software testing practices, as well as the ability to exercise end-to-end transactions beyond the browser. As Agile and DevOps continue to penetrate organizations, there is increased demand for closer collaboration between traditional business analysts / subject matter experts (SMEs) and technical resources. Selenium offers the technical resources a solution for testing web UIs. Yet, a comprehensive quality strategy requires more collaborative solutions that also accommodate the business SMEs. Reaching this balance is critical for achieving the goals driving the digital transformation initiative.

Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 6

About Tricentis Many leading Global 2000 companies have found that Tricentis’ approach is the catalyst needed to make Selenium/WebDriver test automation a feasible part of their broader enterprise quality strategy. With Tricentis Tosca, a comprehensive Continuous Testing Platform, any tester can rapidly create robust scriptless tests that leverage WebDriver. Additionally, any existing Selenium scripts can be integrated into the platform without modification, or they can be automatically migrated into scriptless tests for greater stability and easier maintenance. By supporting both technical and business-focused testing approaches in parallel, we foster collaboration between business SMEs and technical users—which in turn boosts team productivity and delivers better results from the test automation initiative. The Tricentis platform significantly increases test stability and minimizes test maintenance by applying advanced test automation in concert with supporting Continuous Testing capabilities. For example: • Risk-based test case design helps the team focus their resources on the smallest set of tests that deliver the greatest risk coverage • Test data generation and management provides a continuous supply of valid test data • Service virtualization ensures that tests run reliably despite unavailable or unstable dependencies Most importantly, the same scriptless testing approach and supporting Continuous Testing capabilities extend beyond the Web UI. From SAP, to APIs, to Java/.NET, to mainframes, databases, and mobile, testers can rapidly define automation using Tricentis Tosca’s intuitive interface, as well as popular open source testing tools (SoapUI, Appium, etc.). This promotes the rapid, collaborative creation of test scenarios that effectively validate how application changes impact end-to-end business transactions. Such insight is essential for accelerating the software delivery process without compromising the end user experience. We would be happy to discuss what test automation strategies could provide your organization the greatest value given your specific objectives, environment, and priorities. Just reach out to your Tricentis representative or contact us at www.tricentis.com.

Get the analysts’ perspective on test automation Download the new Gartner Magic Quadrant for Software Test Automation

About Tricentis

Tricentis is recognized as having the industry’s most innovative software testing technologies—transforming testing from a roadblock to a catalyst for innovation. Our Continuous Testing platform is architected to simplify testing of modern enterprise applications. Through risk-based testing, scriptless end-to-end test automation, and the industry’s most extensive technology support, Tricentis breaks through the barriers experienced with conventional software testing methods. Tricentis has earned “leader” status in all three top analyst reports. This honor is based on our proven track record of helping 600+ Global 2000 companies achieve test automation rates of over 90%—increasing risk coverage while accelerating testing to keep pace with Agile and DevOps.