Titus Fortner - Crafting a Test Framework
Total Page:16
File Type:pdf, Size:1020Kb
Titus Fortner - Crafting a Test Framework Titus Fortner: Hello, everyone. My name is Titus Fortner and I'm excited to get to talk to you today about some things in test automation that I'm especially passionate about. Just a quick background about me, writing software is my third career. I started out in the navy as a nuclear engineer on a ballistic missile submarine. I do not recommend to that career path to anyone who enjoys sunlight or regular sleeping hours. When I left the military, I worked in a semiconductor manufacturing company for a few years. I could at least see the sky, but it still wasn't something I was especially excited about. I decided to pursue something that I thought I could be especially excited about and looked into a career as a software engineer. As with many of us, I think I just happened into the testing focus. On my first day at my first job as a software developer at a small startup, our manager came to us and said, "Hey, I need someone to figure out this Selenium thing." I said, "Sure, I'll figure it out," and I've been pretty happy with that decision ever since. I've since worked at five different companies as a software engineer and test. About a year ago, I joined Sauce Labs as a solution architect. It means I get to help various clients with their testing. One of my primary tasks is to provide automation framework assessments for our customers. This is all to say that I've seen many different approaches to frameworks from many different problem sets. I've gotten to try many different approaches and see what works and what doesn't work and why. I should also say that in addition to the things that I've gotten paid for I'm very active in the open source community. I've been a core contributor to the Selenium Project for two and a half years now, primarily as the maintainer of the Ruby Bindings. Also have been one of the lead developers on the Watir Project for the last three years and I've written or contributed to at least a dozen other open source projects in the testing space. In this presentation I'm going to talk about the important considerations in crafting your test framework. In the course of working with various colleagues and just being loud and opinionated in general, I found myself in many different disagreements on what constitutes a best practice in this industry. As part of discussing framework components, I'm going to bring up 10 issues that smart people disagree on and give you insight into the things you need to know to make the right decisions for your company. Let me start by clarifying what I mean by test framework. I have a blog article on my website discussing overloaded or ambiguous terms that annoy me. Often times, people use test framework to refer to just the test runner. I'm using it to mean the entire collection of things that are important for maintaining a large reliable test suite. Speaking of last maintainable test suites, let's get one thing clear upfront with our first issue record and playback tools or specifically the Selenium IDE. Selenium IDE is build as a way to easily record and playback tests https://automationguild.com using a Firefox extension. If you are listening to this talk do not be fooled. This is not the tool you're looking for. In the past year, the IDE has stopped working with the latest versions of Firefox. An effort is underway to create a new implementation of the Selenium IDE to provide a record and playback solution specifically supported by the Selenium Project. I'm frustrated by the time and energy this is garnering. The devs pushing for this argue that it is useful for quickly automating repetitive tasks or for quickly reproducing bugs to attach to bug reports. They further argue that it's the responsibility of the user to not abuse a given tool. I see this as somewhat disingenuous. Essentially, the Selenium project is putting its name on only two products. The first is Web Driver where we expect users to be skilled developers in order to use it. The second option is the IDE where anyone who can install a plugin and manually interact with a site can use it. So which one are manual testers going to turn to when a manager tells them to go learn Selenium to automate their tests? The IDE has been referred to as training wheels with the idea that users can start using IDE and then learn the skills that they need in order to write web driver tests. Unfortunately, I too often see these training wheels welded on pretty quickly. Further, the kinds of bicycles that you would use training wheels with they're never going to be the ones that you use to compete. If you want an effective test suite, starting out with the Selenium IDE is almost never going to be a good idea. My big desire for focusing on talks like these is to make it easier for people to get started writing maintainable test suites. If we consider writing straight web driver code as a difficulty of 10 and the IDE as a difficulty of one, how do we create a solution at a level four? How do we help people learn the basic tools that they need in order to be able to quickly start using something that works? These are the seven components that make up a successful test framework. Let's walkthrough each of these and describe what they are and what their issues are. Wrappers and helpers, this is the important code. This is what makes writing tests easier by providing a higher level interface, grouping actions are always taken together and handling potential before and after conditions. In Ruby, this is handled by open source add-on libraries like Watir or Capybara. As the maintainer of the Watir code I could do an entire talk on Watir but for this presentation I'll stick to just the highlights. The most important thing for this and really for any framework is automatically waiting for elements to become visible or interactable, an automatic re-looking up of stale elements. Most maintainable frameworks do this in some fashion but often this code is repeated inconsistently throughout the tests instead of being abstracted to just one place. Watir provides higher level handling for frames, so users don't need to explicitly switch into or out of browsing contexts Titus Fortner - Crafting a Test Framework Page 2 of 9 and for interacting with Windows by URL, title or index. Elements get lazy loaded so they could be defined before they are used. Elements are initialized by a specific element type so their attributes and actions are encapsulated to be directly applicable. Watir also provides a bunch of extra ways to locate elements. You want the parent of a div with attribute x present, an app in class name y but not class name z, it's true they want to put together a hash that makes it obvious what is being located. Also you can match anything by either string or regular expression. Finding unique elements when you don't control your application code without resorting to hard to read CSS or Xpath is much easier. One of the main objections I've heard often from other Selenium core contributors is that the kind of abstractions found in Watir, Capybara, Selenide, Web Driver IO especially things like automated waits and automatic re-looking up of things, remove too much insight from what you get using Selenium commands directly. There is this notion that it is harder to write maintainable tests without understanding the mechanics of Selenium itself. I could not disagree more. A common complaint I hear from experienced developers is that UI tests take too long to write and from less experienced developers that it is too difficult. I think these are valid complaints and we as a community need to work to address them. When testing a website it is more important to know that the features working as intended than it is to understand how or why the site was implemented the way it was. I shouldn't need to know what the last element to load on a page is or what CSS transition needs to happen before an element gets displayed. The feature needs me to click an element. It doesn't make sense for it to be clicked before it's displayed. It doesn't make sense to click it if it isn't enabled yet. Selenium takes the correct approach of being a do what I say tool. A good framework builds on conventions and expectations to allow your test code to do what you mean. It is more important that you can quickly and easily ensure your site's functionality is working than that the implementation details of the app are happening in some specific manner. Data modeling is handling the data that represents objects in the UI. There are three approaches to using data. The first is Grab and Hope. The site has existing users with existing reservations. Grab the first one. Hope it's in a state you need it to be in to test your feature then run your test with it. Obviously, there are a lot of unknowns that can cause a test to fail.