Where Does Selenium Fit in an Enterprise Quality Strategy?

Where Does Selenium Fit in an Enterprise Quality Strategy? Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 1 “Selenium is the de facto standard for front-end test automation of modern web technologies due to the flexible and powerful browser automation capabilities of WebDriver…It’s a sophisticated tool that isn’t easy to learn. It requires your team to have programming expertise and familiarity with object-oriented languages.” - Sean Kenefick, Gartner, in “A Guidance Framework for Automating Your Front-End Web Testing Using Selenium” “Currently, open-source solutions are mainly oriented toward developers and focused on web and mobile technologies, making them somewhat specialized and not appropriate for all testing needs.” - Joachim Herschmann and Thomas Murphy, Gartner, in “Magic Quadrant for Software Test Automation” There’s no doubt that Selenium has earned its spot as the de facto standard for web browser automation. Selenium is well-suited for web UI development teams where testing is conducted by developers or testers who are well-versed in a programming / scripting language. Yet, Selenium is not a test automation panacea. When you move beyond this sweet spot and try to scale Selenium across a large enterprise, several hurdles emerge: • Testing end-to-end business transactions • Conquering the learning curve • Managing brittle scripts • Controlling the Total Cost of Testing This paper explores those challenges—and how to address them so that Selenium can play a key role in a broader enterprise-level quality strategy. Testing end-to-end business transactions Selenium only automates web browser interactions—it cannot test end-to-end business transactions. Even Windows-based dialogs that open during a browser session cannot be automated by Selenium. And then there all those other technologies that are commonly involved in today’s highly-distributed end-to-end business transactions… www.tricentis.com © 2018 Tricentis GmbH. All Rights Reserved v Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 2 If you want to automate end-to-end regression test scenarios that extend from the web UI to SAP, APIs, database, mainframes, etc., you will need to extend beyond Selenium’s core capabilities. Ideally, this is achieved via a broader test automation framework that embraces Selenium and WebDriver. This way, you gain the required breadth of technology support without having to sacrifice the benefits you were seeking with Selenium. Conquering the learning curve Writing and maintaining Selenium frameworks and scripts might not be as challenging as working with production code, but it does require a considerable amount of technical know-how. Considering that manual testing still dominates over 80% of enterprise-level testing, it’s fair to assume that the average tester has limited test automation experience—and even less programming experience. A large gap needs to be bridged for these manual testers to become productive with Selenium test automation. Even more training and resources are required for them to approach the productivity level of programmers, who can easily wield Selenium as an instrument of power. Using Selenium for test automation requires an understanding of: • One of the Selenium-supported programming/scripting languages (e.g., Java, JavaScript, Python, Ruby, C#) • The WebDriver browser driver, browser driver proxies, language bindings, and Selenium Server www.tricentis.com © 2018 Tricentis GmbH. All Rights Reserved v Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 3 • A build and dependency management system like Maven or Gradle • A test framework (e.g., JUnit, TestNG, NUnit) • The browser DOM structure • XPath and/or CSS locators This isn’t a significant hurdle for programmers—so it’s not surprising that they are comfortable working with Selenium. Productivity declines significantly with test scripters (technical testers who have already mastered test automation), then it hits rock bottom with manual testers. Most enterprise QA teams are dominated by testers who have a high degree of business domain knowledge, but not the technical expertise needed to translate the test requirements into code or scripts. In these situations, a huge gap must be bridged to achieve success with Selenium. Either the QA staff must become (at least basic) programmers, or you must shift testing responsibilities to the programming staff. Both options are costly: the first involves a significant training or re-staffing investment, and the second either escalates your development costs or slows your speed of innovation. Managing brittle scripts Many organizations vastly underestimate the amount of time and effort required to resolve false positives and update tests. False positives (from “flaky tests” or “brittle scripts”) are a common struggle among Selenium users. They frequently occur because: www.tricentis.com © 2018 Tricentis GmbH. All Rights Reserved v Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 4 • A seemingly unrelated code change broke an XPath locator • The test data used in the initial run is now “expired” • Dynamic elements did not fully load before WebDriver tried to interact with them • The test timed out before the expected interaction had time to complete • Dependent components (e.g., a third-party API or service) were unavailable or unstable, or were not populated with appropriate data In addition to the work required to resolve false positives, more test maintenance is required to keep the scripts in sync with the evolving application. Selenium does offer ways to promote reuse (e.g., by implementing Page Object classes). However, this is considered an “advanced” use case even by Selenium standards. The vast majority of Selenium scripts use a considerable amount of duplicate code because that’s often the easiest path to automation. However, you pay the price for the initial copy/paste simplicity when a commonly-used control is renamed and hundreds of test scripts need to be updated. All of this leads to what is commonly known as the “maintenance trap”— where the level of test maintenance required starts to offset the time savings achieved by automation. While programmers might be able to rapidly resolve false positives and refactor existing tests for requirement changes, testers who are new to programming will usually require considerably more time and troubleshooting to get the automation back on track. Ultimately, the less technical your Selenium testing resources are, the higher the impact of the maintenance trap. Managing the Total Cost of Testing No software is free—you always need to consider the total cost of ownership. With Selenium, the less technical your testing team is, the more it will ultimately cost you to achieve the expected test automation results. First, Selenium requires you to invest considerable resources in building a custom test framework. The exact resource commitment required varies based on your projects and priorities. However, for some idea of what this involves, consider that it recently took a large Australian bank approximately 500 person hours to build an effective test framework for one of their core applications. This is in line with anecdotal reports (in online testing forums) that it typically takes a single person 3 months to build a framework for a medium-complexity web application. Next, you need to consider the cost of the resources required to build and maintain test scripts with Selenium. For the reasons stated in the previous section, this is a costly endeavor—and the less technical your testers, the higher the costs will be. Based on data that we collected at over 125 customer sites (large enterprises across a variety of industries), building test cases with Selenium tends to require 4x more resources than with a commercial test automation tool. The same data set indicated that, on average, maintenance was 6X greater with Selenium than with a commercial tool. www.tricentis.com © 2018 Tricentis GmbH. All Rights Reserved v Tricentis | Where Does Selenium Fit in an Enterprise Quality Strategy | 5 When you consider license costs plus the costs of creating and maintaining Selenium scripts, the total cost of ownership for Selenium vs. a commercial scriptless and frameworkless test automation platform such as Tricentis Tosca looks like this: Granted, results will vary based on the complexity of your projects, your teams’ skill levels and how well the commercial tool meets your needs. However, in most situations, Selenium testing will ultimately cost more than testing with a commercial testing tool that’s faster and easier for the average tester to master. Final thoughts Selenium has a sweet spot: developers and technical testers focused on testing web UIs. Selenium can play a role in your enterprise-level quality strategy. However, it’s not a silver bullet. An enterprise-level quality strategy also requires the coordinated orchestration of a broader set of software testing practices, as well as the ability to exercise end-to-end transactions beyond the browser. As Agile and DevOps continue to penetrate organizations, there is increased demand for closer collaboration between traditional business analysts / subject matter experts (SMEs) and technical resources. Selenium offers the technical resources a solution for testing web UIs. Yet, a comprehensive quality strategy requires more collaborative solutions that also accommodate the business SMEs. Reaching this balance is critical for achieving

Load more