Sauce Labs Documentation

November 2015, v1.0 Copyright Sauce Labs 2015

1. Quick Start ...... 5 1.1 Instant C# Tests with Sauce Labs ...... 6 1.2 Instant JavaScript Unit Testing with Sauce Labs ...... 9 1.3 Instant Java Tests with Sauce Labs ...... 10 1.4 Instant Node.js Tests with Sauce Labs ...... 13 1.5 Instant PHP Tests with Sauce Labs ...... 19 1.6 Instant Python Tests with Sauce Labs ...... 24 1.7 Instant Ruby Tests with Sauce Labs ...... 29 2. Known Issues and Bugs ...... 32 3. Introducing Sauce Labs ...... 33 3.1 A Tour of the Sauce Labs Interface ...... 34 3.2 Sauce Labs FAQs ...... 36 3.3 Sauce Labs Basics ...... 38 3.4 Supported Browsers and Operating Systems ...... 39 4. Running Tests with Sauce Labs ...... 40 4.1 Tips and Best Practices for Running Tests with Sauce Labs ...... 42 4.1.1 Best Practices for Running Tests with Sauce Labs ...... 43 4.1.1.1 Best Practice: Avoid External Test Dependencies ...... 44 4.1.1.2 Best Practice: Avoid Dependencies between Tests to Run Tests in Parallel ...... 45 4.1.1.3 Best Practice: Don't Use Brittle Locators in Your Tests ...... 47 4.1.1.4 Best Practice: Have a Retry Strategy for Handling Flakes ...... 48 4.1.1.5 Best Practice: Keep Functional Tests Separate from Performance Tests ...... 49 4.1.1.6 Best Practice: Use Build IDs, Tags, and Names to Identify Your Tests ...... 50 4.1.1.7 Best Practice: Use Environment Variables for Authentication Credentials ...... 51 4.1.1.8 Best Practice: Use Explicit Waits ...... 53 4.1.1.9 Best Practice: Use the Latest Version of Selenium Client Bindings ...... 55 4.1.1.10 Best Practices: Use Small, Atomic, Autonomous Tests ...... 56 4.1.2 Tips for Lean, Speedy Tests with Sauce Labs ...... 57 4.1.3 Handling Authentication ...... 59 4.1.3.1 Basic HTTP Authentication ...... 60 4.1.3.2 Injecting Cookies to Bypass Authentication Dialogs ...... 61 4.1.3.3 Running an AutoIt Script as a Pre-run Executable to Handle Windows Security Authentication Dialogs . . . . 62 4.2 Pre-Run Executables ...... 63 4.2.1 Setting Up Pre-Run Executables ...... 64 4.3 Test Configuration and Annotation ...... 65 4.3.1 Configuring Tests with the WebDriver API DesiredCapabilities ...... 66 4.3.2 Configuring Tests with the Sauce Labs REST API ...... 67 4.3.3 Configuring Tests with Selenium's JavaScript Executor ...... 69 4.3.4 Test Configuration Options ...... 70 4.3.5 Examples of Desired Capabilities for iWebDriver and Appium iOS Tests ...... 79 4.4 Manual Testing with Sauce Labs ...... 81 4.4.1 Running a Manual Testing Session ...... 82 4.4.2 Starting Manual Tests from Automated Tests ...... 83 4.4.3 Troubleshooting Manual Tests ...... 84 4.5 Automated Testing with Sauce Labs ...... 86 4.5.1 Troubleshooting Automated Tests ...... 87 4.5.2 Common Error Messages ...... 89 4.6 Mobile Testing with Sauce Labs ...... 92 4.6.1 Mobile Testing with Appium ...... 93 4.6.2 Support and Requirements for Mobile Testing ...... 95 4.6.2.1 Supported Android Emulators ...... 96 4.6.2.2 Supported Mobile Operating Systems ...... 98 4.6.2.3 Requirements for Testing Mobile Native and Hybrid Applications ...... 99 4.6.3 Manual Testing for Mobile Apps ...... 100 4.6.3.1 Getting to the JavaScript Console for Manual iOS Browser Tests ...... 101 4.6.4 Running Emulator and Simulator Mobile Tests ...... 102 4.6.5 Types of Mobile Tests ...... 103 4.6.6 FAQs for Mobile Testing ...... 104 4.6.7 Example Appium Mobile Testing Scripts ...... 106 4.6.7.1 Android Example Scripts Written for Mobile Testing on Sauce Labs ...... 107 4.6.7.1.1 Example Java Script for Testing Android Mobile Applications on Sauce Labs ...... 108 4.6.7.1.2 Example Node.js Script for Testing Android Mobile Applications on Sauce Labs ...... 110 4.6.7.1.3 Example Python Script for Testing Android Mobile Applications on Sauce ...... 113 4.6.7.1.4 Example Ruby Script for Testing Android Mobile Applications on Sauce Labs ...... 116 4.6.7.2 iOS Example Scripts for Mobile Testing on Sauce ...... 119 4.6.7.2.1 Example Java Script for Testing iOS Mobile Applications on Sauce Labs ...... 120 4.6.7.2.2 Example Node.js Script for Testing iOS Mobile Applications on Sauce ...... 123 4.6.7.2.3 Example PHP Script for Testing iOS Mobile Applications on Sauce Labs ...... 126 4.6.7.2.4 Example Python Script for Testing iOS Mobile Applications on Sauce Labs ...... 128 4.6.7.2.5 Example Ruby Script for Testing iOS Mobile Applications on Sauce Labs ...... 129 4.7 Running Tests in Parallel with Sauce Labs ...... 132 4.7.1 Running Tests in Parallel with C# ...... 133

Page 2 Copyright Sauce Labs 2015

4.7.1.1 Running C# Tests in Parallel with Gallio and MBUnit ...... 134 4.7.1.2 Running C# Tests in Parallel with PNUnit ...... 139 4.7.2 Running Tests in Parallel with Java ...... 145 4.7.2.1 Running Java Tests in Parallel with JUnit ...... 146 4.7.2.2 Running Java Tests in Parallel with TestNG ...... 150 4.7.3 Running Tests in Parallel with PHP ...... 151 4.7.3.1 Running PHP Tests in Parallel with PHPUnit and Paratest ...... 152 4.7.4 Running Tests in Parallel with Python ...... 153 4.7.4.1 Running Python Tests in Parallel with nose ...... 154 4.7.4.2 Running Python Tests in Parallel with pytest ...... 156 4.7.5 Running Tests in Parallel with Ruby ...... 159 4.7.5.1 Running Ruby Tests in Parallel with RSpec ...... 160 4.7.6 Troubleshooting Parallel Tests on Sauce Labs ...... 163 4.8 Viewing and Managing Sauce Labs Test Results ...... 166 4.8.1 Viewing Screenshots, Commands, Logs, and Metadata for Sauce Test Results ...... 167 4.8.2 Using Status Badges and the Browser Matrix Widget to Monitor Test Results ...... 168 4.8.3 Sharing the Results of Sauce Labs Tests ...... 170 4.8.4 Embedding Test Results in HTML Pages ...... 171 4.8.5 Building Links to Test Results ...... 172 4.8.6 Managing Archived Test and Build Results ...... 173 4.8.6.1 Searching for Test Results and Builds on Your Archive Page ...... 174 4.8.6.1.1 Search Fields and Operators ...... 175 4.8.6.2 Using Favorites to Save Your Searches ...... 176 4.8.6.3 Deleting Test and Build Results ...... 177 4.8.6.4 Changing the Layout of Your Archives Page ...... 178 4.9 Using Sauce Storage for Test Assets ...... 179 4.9.1 Uploading Assets to Sauce Storage Using C# ...... 181 5. Using Sauce Connect for Testing Behind the Firewall or on Localhost ...... 182 5.1 Sauce Connect in the Network ...... 183 5.1.1 Sauce Connect Setup and Teardown Process ...... 184 5.1.2 Sauce Connect and Security ...... 186 5.1.2.1 Sauce Connect Certificate Handling ...... 187 5.1.3 Sauce Connect Network Configurations ...... 188 5.1.3.1 Improving Sauce Connect Network Performance ...... 189 5.1.3.2 Basic Sauce Connect Network Configuration ...... 190 5.1.3.3 Dysfunctional Sauce Connect Network Configurations ...... 191 5.1.3.4 Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration ...... 192 5.2 Sauce Connect Setup and Configuration ...... 193 5.2.1 System Requirements for Sauce Connect ...... 194 5.2.2 Setting Up Sauce Connect ...... 195 5.2.3 Monitoring Sauce Connect with a Service Management Tool ...... 196 5.2.4 Using Multiple Sauce Connect Tunnels ...... 201 5.2.5 Sauce Connect Proxy Configuration ...... 203 5.3 Sauce Connect Command Line Reference ...... 205 5.4 Sauce Connect FAQS ...... 207 5.5 Troubleshooting Sauce Connect ...... 209 5.6 Sauce Connect Best Practices ...... 211 6. Using Sauce Labs with Continuous Integration Platforms ...... 212 6.1 Setting Up CI Platform and Sauce Labs Integrations with Sauce Plugins ...... 213 6.1.1 Setting Up Sauce for Visual Studio Online PREVIEW ...... 214 6.1.2 Setting Up Sauce Labs with Bamboo ...... 215 6.1.2.1 Installing and Configuring the Sauce Labs Plugin for Bamboo ...... 216 6.1.2.2 Configuring Bamboo for a Java Project with Sauce Labs ...... 217 6.1.2.3 Configuring Bamboo for a Python Project with Sauce Labs ...... 219 6.1.2.4 Referencing Environment Variables for Bamboo Jobs ...... 221 6.1.2.5 Outputting the Bamboo Session ID to stdout ...... 222 6.1.3 Setting Up Sauce Labs with Jenkins ...... 223 6.1.3.1 Installing and Configuring the Sauce OnDemand Plugin for Jenkins ...... 224 6.1.3.2 Configuring Sauce Connect with the Jenkins Plugin ...... 225 6.1.3.3 Setting Desired Capabilities for Jenkins Projects ...... 227 6.1.3.3.1 Environment Variables Used by the Jenkins Plugin ...... 229 6.1.3.4 Running Parallel Tests with Jenkins ...... 230 6.1.3.4.1 Configuring Jenkins Matrix Projects with Sauce ...... 231 6.1.3.4.2 Setting Up Parameterized Builds for Jenkins and Sauce ...... 232 6.1.3.5 Setting Up Reporting between Sauce Labs and Jenkins ...... 233 6.1.4 Setting Up Sauce Labs with TeamCity ...... 234 6.1.4.1 Installing and Configuring the Sauce OnDemand Plugin for TeamCity ...... 235 6.1.4.2 Configuring TeamCity for a Java Project with Sauce Labs ...... 236 6.1.4.3 Referencing Environment Variables for TeamCity Jobs ...... 237 6.1.4.4 Outputting the TeamCity Session ID to stdout ...... 238 6.2 Other CI Platform Integrations with Sauce Labs ...... 239 7. Managing Sauce Labs Accounts and Teams ...... 240

Page 3 Copyright Sauce Labs 2015

7.1 Team Management Plans ...... 241 7.1.1 Canceling Your Subscription Plan ...... 242 7.1.2 Updating Your Plan Billing Information ...... 243 7.1.3 Upgrading Your Subscription Plan ...... 244 7.1.4 Viewing Plan Details ...... 245 7.1.5 Viewing Plan Usage Details for Your Accounts ...... 246 7.2 Access Configuration ...... 247 7.2.1 Setting Your Domain ...... 248 7.2.2 Requiring Sauce Connect for Your Domain ...... 249 7.2.3 Using Single Sign-On with Your Sauce Labs Domain ...... 250 7.2.3.1 SSO Integrations Supported by Sauce Labs ...... 251 7.2.3.2 Metadata for Single Sign-On ...... 252 7.2.3.3 Basic Single Sign-On Configuration for Sauce Labs ...... 253 7.2.3.4 Requiring SSO for Account Access ...... 254 7.2.3.5 Using Just-in-Time Provisioning for Users with Single Sign-On ...... 255 7.3 Managing Team Members and Accounts ...... 256 7.3.1 Adding Users ...... 257 7.3.2 Deleting Users ...... 258 7.3.3 Enabling Your Subaccounts to Add Users to Your Account ...... 259 7.3.4 Managing User Info and Accounts ...... 260 7.3.5 Resetting User Access Tokens ...... 261 7.3.6 User Account Types ...... 262 7.3.7 Viewing and Managing Your Account Information ...... 263 7.3.8 Viewing Users Associated with Your Account ...... 264 8. The Sauce Labs REST API ...... 265 8.1 Accessing the API for Windows Users ...... 267 8.2 Account Methods ...... 268 8.3 Bug Methods ...... 270 8.4 Information Methods ...... 271 8.5 JavaScript Unit Testing Methods ...... 272 8.6 Job Methods ...... 273 8.7 Temporary Storage Methods ...... 277 8.8 Test Activity and Usage Methods ...... 278 8.9 Tunnel Methods ...... 280

Page 4 Copyright Sauce Labs 2015

Quick Start

These topics are intended to provide you with a quick start overview of how you can set up testing in the Sauce Labs cloud using several popular scripting languages.

Instant C# Tests with Sauce Labs Instant JavaScript Unit Testing with Sauce Labs Instant Java Tests with Sauce Labs Instant Node.js Tests with Sauce Labs Instant PHP Tests with Sauce Labs Instant Python Tests with Sauce Labs Instant Ruby Tests with Sauce Labs

Page 5 Copyright Sauce Labs 2015

Instant C# Tests with Sauce Labs

Sauce Labs is a cloud platform for executing automated and manual mobile and web tests. Sauce Labs supports running automated tests with Selenium WebDriver (for web applications) and Appium (for native and mobile web applications). This topic describes the basics you need to get up and running with your C# tests on Sauce.

Prerequisites Quick Start Code Sample GuineaPigTests.cs Constants.cs Running Local Tests Running Tests in Parallel Reporting on Test Results

Prerequisites

Before getting started, you should review the Best Practices for Running Tests with Sauce Labs.

You need to have these components installed to set up testing on Sauce with C#.NET:

Visual Studio Selenium WebDriver Selenium DLLs for .NET installed and referenced by your project

Quick Start

Configuring Selenium tests to run on Sauce Labs is simple. The basic change is to switch from using a local Selenium driver to using a remote driver pointed at ondemand.saucelabs.com, specifying your Sauce Labs account credentials and desired browser configuration.

Local C# Testing Example

IWebDriver driver = new FirefoxDriver();

Remote C# Testing on Sauce Example

IWebDriver driver; DesiredCapabilities desiredCapability = DesiredCapabilities.Firefox()// set the desired browser desiredCapability.SetCapability("platform", "Windows 7"); // operating system to use driver = new RemoteWebDriver(new Uri("http://YOUR_USERNAME:[email protected]:80/wd/hub");

Code Sample

GuineaPigTests.cs

This example verifies the title, a link, and the presence of a user agent element on the page https://saucelabs.com/test/guinea-pig. It connects to Sauce Labs, run commands to remote control the target browser, and reports the results. It also includes the code for running tests in parallel and reporting the results to your Sauce Labs dashboard.

You can use the Platform Configurator to specify the desired capabilities for any browser/platform combination you want for your test.

Constants.cs

Use this class to set your Sauce Labs account name and access key. You can find these in the User Profile menu of your Sauce Labs

Page 6 Copyright Sauce Labs 2015

dashboard.

Hardcoding v. Using Environment Variables for Authentication You could also hardcode your authentication credentials in the GuineaPigsTest.cs file in these lines:

desiredCapabilites.SetCapability("username", Constants.SAUCE_LABS_ACCOUNT_NAME); // supply sauce labs username desiredCapabilites.SetCapability("accessKey", Constants.SAUCE_LABS_ACCOUNT_KEY); // supply sauce labs account key

Where you would substitute your account name and access key for Constants.SAUCE_LABS_ACCOUNT_NAME and Constants.SAU CE_LABS_ACCOUNT_KEY. However, as a best practice we recommend using environment variables stored in a local file or system for authentication, as this improves the security of your tests, and enables other members of your team to run tests on your account by referencing the authentication credentials.

Constants.cs Expand source namespace SauceLabs.SeleniumExamples

{ ///

contains constants used by the tests such as the user name and password for the sauce labs

internal static class Constants { ///

name of the sauce labs account that will be used

internal const string SAUCE_LABS_ACCOUNT_NAME = "Your Account Name"; ///

account key for the sauce labs account that will be used

internal const string SAUCE_LABS_ACCOUNT_KEY = "Your Access Key";

// NOTE: // To change the maximum number of parallel tests edit DegreeOfParallelism in AssemblyInfo.cs

} }

Running Local Tests

Developing apps on localhost is quick and efficient. The drawback is that localhost is not a publicly-accessible address on the Internet, so the browsers in the Sauce Labs cloud can't load and test your app. The solution is to use Sauce Connect. Sauce Connect uses a secure tunnel protocol that gives specific Sauce machines access to your local network. You can also use it to test applications that are inside your corporate firewall. Sauce Connect is not required to run tests on Sauce Labs, only in situations where the application or website you want to test is on your local machine or behind a firewall.

Running Tests in Parallel

Now that you’re running tests on Sauce, you may wonder how you can run your tests faster. One way to increase speed is by running tests in parallel across multiple virtual machines.

You can run your tests in parallel at two levels, and you can run your tests in parallel across multiple browsers. For example, if you have 10 tests and want to run on five browsers, this would be parallelism of five. You can also run tests across browsers and each test in parallel. Using our previous example this would be more like 50 parallel tests. Doing this requires that your tests are written in a way that they do not collide with one another. For more on this see the Selenium WebDriver - Running Your Tests in Parallel blog.

Before you start running tests in parallel, you should review the Best Practices for Running Tests with Sauce Labs, especially the topics on avoidi ng external test dependencies and avoiding dependencies between tests.

Page 7 Copyright Sauce Labs 2015

Parallel Tests and Concurrency Limits The number of tests you can run in parallel is determined by the concurrency limit associated with your account. You can check this in you Sauce Labs dashboard under Concurrent VMs.

Check out the topics under Running Tests in Parallel with C# for code examples and recommendations on running tests in parallel with C#.

Reporting on Test Results

Selenium is a protocol focused on automating the actions of a browser, and as such, it doesn't encompass the notions of a "test" that "passes" or "fails." Because of this, you need to include a section in your code that explicitly gets the status of the test, and then sends it to the Sauce Labs dashboard.

C# Code for Reporting Test Results to the Sauce Labs Dashboard Expand source [TearDown] public void Cleanup() { // Gets the status of the current test bool passed = TestContext.CurrentContext.Result.Status == TestStatus.Passed; try { // Logs the result to Sauce Labs ((IJavaScriptExecutor)driver).ExecuteScript("sauce:job-result=" + (passed ? "passed" : "failed")); } finally { // Terminates the remote webdriver session driver.Quit(); }

}

Page 8 Copyright Sauce Labs 2015

Instant JavaScript Unit Testing with Sauce Labs

If you already have JavaScript unit tests set up with a testing framework, or you have custom tests, it's easy to get up and running with Sauce.

You can use our JavaScript Unit Testing Methods to run your tests and report the results to your Sauce Labs dashboard You can use the Grunt-Saucelabs grunt task to run Jasmine, Mocha, YUI, QUnit, and custom tests on Sauce. Each framework/test suite has its own task, which in addition to setting up your test to run on Sauce, also opens a Sauce Connect tunnel so you can test applications and websites that are on localhost or located behind a firewall. The topics in Setting Up the Reporting Code for Your JavaScript Unit Tests explain how to set up your testing frameworks to report the results to your Sauce Labs dashboard

Page 9 Copyright Sauce Labs 2015

Instant Java Tests with Sauce Labs

Sauce Labs is a cloud platform for executing automated and manual mobile and web tests. Sauce Labs supports running automated tests with Selenium WebDriver (for web applications) and Appium (for native and mobile web applications). This topic covers the basic information you need to get your Java tests up and running on Sauce.

Prerequisites Quick Start Code Sample Running Local Tests Running Tests in Parallel Reporting on Test Results

Prerequisites

Before getting started, you should review the Best Practices for Running Tests with Sauce Labs.

You need to have these components installed to set up testing on Sauce with Java.

You must have JDK 1.6 or higher installed You need to have the selenium-java client for your operating system installed. You can download it from http://www.seleniumhq.org/down load/, under the section Selenium Client & WebDriver Language Bindings. You should install the Java Helper Library, which will automatically update your Sauce Labs dashboard with information like whether tests have passed or failed, output the Sauce Session ID to stdout for parsing by the Sauce Jenkins and Bamboo plugins, and which provide s a com.saucelabs.common.SauceOnDemandAuthentication class, which handles obtaining the Sauce OnDemand user name and access key from environment variables or the filesystem.

Quick Start

Configuring your existing Java-based Selenium tests to run on Sauce Labs is simple. The basic change is just to switch from using a local Selenium driver, to using a remote driver pointed at ondemand.saucelabs.com, specifying your Sauce Labs account credentials and desired browser configuration.

Local Testing Java Example

WebDriver driver = new FirefoxDriver();

Java Testing on Sauce Labs Example

DesiredCapabilities caps = DesiredCapabilities.firefox(); caps.setCapability("platform", "Windows 7"); caps.setCapability("version", "38.0"); WebDriver driver = new RemoteWebDriver(new URL("http://YOUR_USERNAME:[email protected]:80/wd/hub"),

Code Sample

This code example illustrates setting up a simple Java test to find the title of a page hosted by Sauce Labs.

Page 10 Copyright Sauce Labs 2015

Example Java Code for Testing with Sauce Labs Expand source import org.openqa.selenium.WebDriver; import org.openqa.selenium.remote.DesiredCapabilities; import org.openqa.selenium.remote.RemoteWebDriver;

import java.net.URL;

public class SampleSauceTest {

public static final String USERNAME = "YOUR_USERNAME"; public static final String ACCESS_KEY = "YOUR_ACCESS_KEY"; public static final String URL = "http://" + USERNAME + ":" + ACCESS_KEY + "@ondemand.saucelabs.com:80/wd/hub";

public static void main(String[] args) throws Exception {

DesiredCapabilities caps = DesiredCapabilities.chrome(); caps.setCapability("platform", "Windows XP"); caps.setCapability("version", "43.0");

WebDriver driver = new RemoteWebDriver(new URL(URL), caps);

/** * Goes to Sauce Lab's guinea-pig page and prints title */

driver.get("https://saucelabs.com/test/guinea-pig"); System.out.println("title of page is: " + driver.getTitle());

driver.quit(); } }

Running Local Tests

Running Tests in Parallel

Now that you’re running tests on Sauce, you may wonder how you can run your tests faster. One way to increase speed is by running tests in parallel across multiple virtual machines.

Page 11 Copyright Sauce Labs 2015

Most Java users use one of two popular third party testing frameworks: TestNG or Junit. These links are for two example projects written in each. They are designed to run in parallel. You can clone them and add your own test cases if you want:

1. https://github.com/ndmanvar/SeleniumJavaJunit 2. https://github.com/ndmanvar/SeleniumJavaTestNG

Running Tests in Parallel and Across Multiple Browsers Tests can be run in parallel at two levels: you can run your tests in parallel,and you can run your tests in parallel across multiple browsers. For example, if you have 10 tests and run them serially on five browsers, this would be parallelism of five. You can also run tests across browsers and each test in parallel. Using our previous example, this would be 50 parallel tests (10 tests, five browsers). This requires that your tests are written in a way that they do not collide with one another. For more on this see Selenium WebDriver - R unning Your Tests in Parallel blog.

For more information, see the topics under Running Tests in Parallel with Java.

Reporting on Test Results

The Java Helper Library will automatically send test names and pass/fail results to your Sauce Labs dashboard.

You should also check out the topic Viewing and Managing Sauce Labs Test Results for more suggestions on how to improve test reporting and using build numbers for your continuous integration platform.

Page 12 Copyright Sauce Labs 2015

Instant Node.js Tests with Sauce Labs

This tutorial will provide you with step-by-step information to get your node.js tests up and running on Sauce. This tutorial includes setting up Web DriverIO as the Node.js client for Selenium 2.0, and Jasmine as the Behavior Driven Development testing framework. While there are many options for setting up your Node.js toolchain, this tutorial focuses on these tools as the fastest way for you to get up and running with your Node.js tests. WebDriverIO will also work with Mocha or Cucumber as your testing framework.

Prerequisites Quick Start Running the Sample Test Code Sample Running Local Tests Running Tests in Parallel Reporting on Test Results

Prerequisites

Before you get started, you should review the

The example script assumes you have set your Sauce Labs access credentials as environment variables You will need to have Node.js v.0.11 or higher already installed You will need to install both WebDriverIO and Jasmine

$ npm install webdriverio jasmine

Once you've installed WebDriverIO, you can use the command line configuration utility to set up your test configuration

$ ./node_modules/.bin/wdio config

Quick Start

Configuring Selenium tests to run on Sauce Labs is simple. The basic change is to switch from using a local Selenium driver, to using a remote driver pointed at ondemand.saucelabs.com. If you're using the WebdriverIO testing utility for Node.js, all you have to do is provide your Sauce Labs user credentials, and it will automatically connect you. Note that this example assumes you have set your Sauce Labs authentication credentials as environment variables, a recommended best practice for testing on Sauce. This code sample shows how to set up a very simple standalone JavaScript test script that will authenticate you to Sauce Labs, and then test against Chrome to find the title of Google's home page.

standalone.chrome.js Code Example Expand source var client = require('webdriverio').remote({ user: process.env.SAUCE_USERNAME, key: process.env.SAUCE_ACCESS_KEY, desiredCapabilities: { browserName: 'chrome' } });

client .init() .url('http://google.com') .getTitle().then(console.log) .end();

Page 13 Copyright Sauce Labs 2015

Running the Sample Test

Once you have everything set up, you can run an example test by executing the WebDriverIO config file. This will spin up three browsers running in parallel.

$ ./node_modules/.bin/wdio wdio.conf.js

Code Sample

This code example was generated through the WebDriverIO configuration utility and defines the desired capabilities for the test, while the test.s pec.js code specifies the test behavior.

WebDriverIO Test Configuration Example wdio.conf.js Expand source exports.config = { // // ======// Service Providers // ======// WebdriverIO supports Sauce Labs, Browserstack and Testing Bot (other cloud providers // should work too though). These services define specific user and key (or access key) // values you need to put in here in order to connect to these services. // user: process.env.SAUCE_USERNAME, key: process.env.SAUCE_ACCESS_KEY,

// // If you are using Sauce Labs, WebdriverIO takes care to update the job information // once the test is done. This option is set to `true` by default. // updateJob: true,

// // ======// Specify Test Files // ======// Define which test specs should run. The pattern is relative to the directory // from which `wdio` was called. Notice that, if you are calling `wdio` from an // NPM script (see https://docs.npmjs.com/cli/run-script) then the current working // directory is where your package.json resides, so `wdio` will be called from there. // specs: [ './*.spec.js' ], // // ======// Capabilities // ======

// Define your capabilities here. WebdriverIO can run multiple capabilties at the same // time. Depending on the number of capabilities, WebdriverIO launches several

Page 14 Copyright Sauce Labs 2015 test // sessions. Within your capabilities you can overwrite the spec and exclude option in // order to group specific specs to a specific capability. // // If you have trouble getting all important capabilities together, check out the // Sauce Labs platform configurator - a great tool to configure your capabilities: // https://docs.saucelabs.com/reference/platforms-configurator // capabilities: [{ browserName: 'firefox', version: 37, name: 'Firefox Selenium tests', build: process.env.BUILD_NUMBER },{ browserName: 'chrome', version: 43, name: 'Chrome Selenium tests', build: process.env.BUILD_NUMBER },{ browserName: 'safari', version: 6, name: 'Safari Selenium tests', build: process.env.BUILD_NUMBER }], // // ======// Test Configurations // ======// Define all options that are relevant for the WebdriverIO instance here // // Level of logging verbosity. logLevel: 'silent', // // Enables colors for log output. coloredLogs: true, // // Saves a screenshot to a given path if a command fails. screenshotPath: './errorShots/',

// // Set a base URL in order to shorten url command calls. If your url parameter starts // with "/", the base url gets prepended. baseUrl: 'http://nodejs.org',

// // Default timeout for all waitForXXX commands. waitforTimeout: 10000,

// // Framework you want to run your specs with. // The following are supported: mocha, jasmine and cucumber // see also: http://webdriver.io/guide/testrunner/frameworks.html // // Make sure you have the node package for the specific framework installed before running // any tests. If not please install the following package: // Mocha: `$ npm install mocha`

Page 15 Copyright Sauce Labs 2015

// Jasmine: `$ npm install jasmine` // Cucumber: `$ npm install cucumber` framework: 'jasmine',

// // Test reporter for stdout. // The following are supported: dot (default), spec and xunit // see also: http://webdriver.io/guide/testrunner/reporters.html reporter: 'spec',

// // Options to be passed to Mocha. // See the full list at http://mochajs.org/ jasmineNodeOpts: {

Page 16 Copyright Sauce Labs 2015

defaultTimeoutInterval: 10000 } };

Jasmine test.spec.js Example Expand source describe('mocha spec examples', function() {

it('should get home page', function* () { yield browser.url('/'); expect(yield browser.getTitle()).toBe('Node.js'); expect(yield browser.getText('#home-intro')).toContain('JavaScript runtime'); });

it('should go to the doc page', function* () { yield browser.click('=API Docs'); expect(yield browser.getTitle()).toContain('Manual'); });

it('should return to the home page', function* () { yield browser.click('=V8'); expect(yield browser.getText('#apicontent h1')).toContain('V8'); }); });

Running Local Tests

If you want to use Sauce Connect, you need to set host to localhost and the port to 4445, as shown in this example script.

Sauce Connect and Node.js Example Expand source var client = require('webdriverio').remote({ user: process.env.SAUCE_USERNAME, key: process.env.SAUCE_ACCESS_KEY, host: 'localhost', port: 4445, desiredCapabilities: { browserName: 'chrome' } });

client .init() .url('http://localhost') .getTitle().then(console.log) .end();

Page 17 Copyright Sauce Labs 2015

Running Tests in Parallel

Now that you’re running tests on Sauce, you may wonder how you can run your tests faster. One way to increase speed is by running tests in parallel across multiple virtual machines.

Before you start running tests in parallel, you should review the , especially the topics on avoiding external test dependencies and avoiding dependencies between tests.

With WebDriverIO you can define multiple Capabilities to run tests in parallel, as shown in the wdio.conf.js example script.

Reporting on Test Results

Selenium doesn't have a built-in reporting mechanism to tell you whether a particular test passed or failed, only whether it did or didn't complete. Fortunately, WebDriverIO includes several reporting mechanisms, including an Xunit Reporter that will help you integrate your test reports with yo ur continuous integration server.

Page 18 Copyright Sauce Labs 2015

Instant PHP Tests with Sauce Labs

Sauce Labs is a cloud platform for executing automated and manual mobile and web tests. Sauce Labs supports running automated tests with Selenium WebDriver (for web applications) and Appium (for native and mobile web applications). In this topic you'll see how to get up and running with PHP and Sauce.

Requirements and Set Up Quick Start Code Example Running Local Tests Running Tests in Parallel Reporting Test Results

Requirements and Set Up

Before you get started, you should review the Best Practices for Running Tests with Sauce Labsprecut.

You need to have these components installed to set up testing on Sauce with PHP:

You must have PHP set up on your system before you start testing. Mac OS X comes with PHP installed, as long as you're running version 5.3 or higher, you're ready to go! If you're a Windows user, you can find instructions for setting up PHP on Windows at PHP.net. Windows users must also enable the PHP curl library and OpenSSL support to get the best performance from your PHP setup. For more information, see Installing Extensions for Windows on the php.net website as well as these setup topics: Enabling the PHP curl Library for Windows Setting Up curl/OpenSSL Support for PHP on Windows We also recommend using Sausage as your PHP testing framework. Sausage contains classes and libraries that are designed to work with the Sauce Labs API, and it provides a convenient user interface and other features for managing your test results. It also includes a demo test, WebDriverDemo.php, that you can run to see how Sauce works. For more information about how to set up and use Sausage, see Setting Up Sausage for Windows and Setting Up Sausage for OS X and Linux.

Quick Start

You can set up any PHP test you've already written to run on Sauce, regardless of the testing framework it uses. All you need to do is change the test from running locally, to running on the Sauce cloud by defining the sauce URL, your authentication credentials, and a set of desired capabilities for the test.

This example test suite directly subclasses PHPUnit_Extensions_Selenium2TestCase, and thereby uses a locally-running Firefox browser.

Page 19 Copyright Sauce Labs 2015

PHP Test Running Locally Expand source

public static $browsers = array( array( 'browserName' => 'firefox', ) ) );

public function testTitle() { $this->assertContains("I am a page title", $this->title()); }

public function testLink() { $link = $this->byId('i am a link'); $link->click(); $this->assertContains("I am another page title", $this->title()); }

}

And this example shows how you could edit the existing test to run on Sauce. Note that it assumes your authentication credentials are set up as PHP constants, as recommended in our topic Best Practice: Use Environment Variables for Authentication Credentials. You can use the Platform Configurator to specify the desired capabilities for any browser/platform combination you want.

Page 20 Copyright Sauce Labs 2015

PHP Test Running on the Sauce Cloud Expand source 'firefox', 'host' => SAUCE_HOST, 'port' => 80, 'desiredCapabilities' => array( 'version' => '15', 'platform' => 'Windows 2012' ) ) );

public function testTitle() { $this->assertContains("I am a page title", $this->title()); }

public function testLink() { $link = $this->byId('i am a link'); $link->click(); $this->assertContains("I am another page title", $this->title()); }

}

Code Example

In this code example for PHP, the test runs a check to make sure that clicking a link brings you to the expected page. While simple, this example illustrates everything you need to run an automated test on Sauce Labs. First, it checks for your Sauce authentication credentials, which are set as environmental variables, and then sets the platform, browser, version, and other capabilities to use in the test.

Page 21 Copyright Sauce Labs 2015

WebDriverDemo.php Expand source

require_once 'vendor/autoload.php';

define('SAUCE_HOST', SAUCE_USERNAME.':'.SAUCE_ACCESS_KEY.'@ondemand.saucelabs.com');

class WebTest extends PHPUnit_Extensions_Selenium2TestCase { protected $start_url = 'http://saucelabs.com/test/guinea-pig';

public static $browsers = array( array( 'browserName' => 'firefox', 'host' => SAUCE_HOST, 'port' => 80, 'desiredCapabilities' => array( 'version' => '15', 'platform'=> 'Windows 2012' ) ) );

protected function setUp() { $this->setBrowserUrl(''); }

public function testTitle() { $this->url($this->start_url); $this->assertContains("I am a page title", $this->title()); } } ?>

Running Local Tests

Running Tests in Parallel

Now that you’re running tests on Sauce, you may wonder how you can run your tests faster. One way to increase speed is by running tests in parallel across multiple virtual machines.

Page 22 Copyright Sauce Labs 2015

Check out the topics under Running Tests in Parallel with PHP for more information.

Reporting Test Results

Sausage includes a number of features that make it easier for Sauce to report on your test results.

By default, Sauce Labs doesn't know how to display the name of your test. Sausage comes up with a good name (TestClass::testFunction ) and reports it with your test so it's easy to find on your dashboard. Similarly, Sauce has no way to know if a particular test passed or failed. Sausage catches any failed assertions and reports the status of the test to Sauce after it's complete. Upon test failure Sausage will generate an authorized link to the failed job report on the Sauce Labs website, to facilitate reporting to people who need to know the details of the test. The job remains private (unless you change the status yourself), but others can follow the link without needing to log in with your credentials. Check out Sharing the Results of Sauce Labs Tests for more information.

You should also follow our recommended best practice of adding build numbers, tags, and other identifying information to your tests so you can easily find and manage them in your test results and archives pages, and associate tests with build numbers in your continuous integration pipeline.

Page 23 Copyright Sauce Labs 2015

Instant Python Tests with Sauce Labs

Sauce Labs is a cloud platform for executing automated and manual mobile and web tests. Sauce Labs supports running automated tests with Selenium WebDriver (for web applications) and Appium (for native and mobile web applications). In this topic you'll see how to get your Python tests up and running on Sauce.

Prerequisites Quick Start Code Sample Running Local Tests Running Tests in Parallel Reporting on Test Results

Prerequisites

Before getting started, you should read the Best Practices for Running Tests with Sauce Labs.

You will need to install the Selenium WebDriver client driver to your local Python environment You can either download the driver from the link, or use pip to install it.

pip install selenium

You should also install the Sauce Python client, which provides features for reporting job information to the Sauce Labs dashboard.

pip install sauceclient

Quick Start

It's very easy to get your existing Python test scripts up and running on Sauce.

If you wanted to run Selenium locally, you might initiate a driver for the browser that you want to test on like so:

Running a Local Python Test

driver = webdriver.Firefox()

If you wanted to run on Sauce, you would instead use webdriver.Remote(), and then pass it two paramaters: command_executor, which points to the Sauce cloud and uses your Sauce Labs authentication to log in, and desired_capabilties, which specifies the browsers and operating systems to run the tests against.

Page 24 Copyright Sauce Labs 2015

Running a Python Test Remotely on Sauce

# this is how you set up a test to run on Sauce Labs desired_cap = { 'platform': "Mac OS X 10.9", 'browserName': "chrome", 'version': "31", } driver = webdriver.Remote(

command_executor='http://YOUR_SAUCE_USERNAME:[email protected] om:80/wd/hub', desired_capabilities=desired_cap)

You can use the Platform Configurator to specify the desired capabilities for any browser/platform combination you want.

Code Sample

This simple Python test script tests the Google front page. Despite its simplicity, it contains everything you need to know in order to run an automated test on Sauce Labs.

Page 25 Copyright Sauce Labs 2015

Python on Sauce Code Example Expand source from selenium import webdriver from selenium.webdriver.common.keys import Keys from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

# This is the only code you need to edit in your existing scripts. # The command_executor tells the test to run on Sauce, while the desired_capabilties # parameter tells us which browsers and OS to spin up. desired_cap = { 'platform': "Mac OS X 10.9", 'browserName': "chrome", 'version': "31", } driver = webdriver.Remote(

command_executor='http://philg:[email protected] .com:80/wd/hub', desired_capabilities=desired_cap)

# This is your test logic. You can add multiple tests here. driver.implicitly_wait(10) driver.get("http://www.google.com") if not "Google" in driver.title: raise Exception("Unable to load google page!") elem = driver.find_element_by_name("q") elem.send_keys("Sauce Labs") elem.submit() print driver.title

# This is where you tell Sauce Labs to stop running tests on your behalf. # It's important so that you aren't billed after your test finishes. driver.quit()

This code sample is more complex, and relies on some libraries that you may need to install to make it work.

A More Complex Python Code Example Expand source import os import sys import httplib import base64 import json import new import unittest import sauceclient from selenium import webdriver from sauceclient import SauceClient # it's best to remove the hardcoded defaults and always get these values # from environment variables USERNAME = os.environ.get('SAUCE_USERNAME', _U_) ACCESS_KEY = os.environ.get('SAUCE_ACCESS_KEY', _K_) sauce = SauceClient(USERNAME, ACCESS_KEY) browsers = [{"platform": "Mac OS X 10.9", "browserName": "chrome",

Page 26 Copyright Sauce Labs 2015

"version": "31"}, {"platform": "Windows 8.1", "browserName": "internet explorer", "version": "11"}] def on_platforms(platforms): def decorator(base_class): module = sys.modules[base_class.__module__].__dict__ for i, platform in enumerate(platforms): d = dict(base_class.__dict__) d['desired_capabilities'] = platform name = "%s_%s" % (base_class.__name__, i + 1) module[name] = new.classobj(name, (base_class,), d) return decorator

@on_platforms(browsers) class SauceSampleTest(unittest.TestCase): def setUp(self): self.desired_capabilities['name'] = self.id() sauce_url = "http://%s:%[email protected]:80/wd/hub" self.driver = webdriver.Remote( desired_capabilities=self.desired_capabilities, command_executor=sauce_url % (USERNAME, ACCESS_KEY) ) self.driver.implicitly_wait(30) def test_sauce(self): self.driver.get('http://saucelabs.com/test/guinea-pig') assert "I am a page title - Sauce Labs" in self.driver.title comments = self.driver.find_element_by_id('comments') comments.send_keys('Hello! I am some example comments.' ' I should be in the page after submitting the form') self.driver.find_element_by_id('submit').click() commented = self.driver.find_element_by_id('your_comments') assert ('Your comments: Hello! I am some example comments.' ' I should be in the page after submitting the form' in commented.text) body = self.driver.find_element_by_xpath('//body') assert 'I am some other page content' not in body.text self.driver.find_elements_by_link_text('i am a link')[0].click() body = self.driver.find_element_by_xpath('//body') assert 'I am some other page content' in body.text def tearDown(self): print("Link to your job: https://saucelabs.com/jobs/%s" % self.driver.session_id) try: if sys.exc_info() == (None, None, None): sauce.jobs.update_job(self.driver.session_id, passed=True) else:

Page 27 Copyright Sauce Labs 2015

sauce.jobs.update_job(self.driver.session_id, passed=False) finally: self.driver.quit()

Running Local Tests

Running Tests in Parallel

Now that you’re running tests on Sauce, you may wonder how you can run your tests faster. One way to increase speed is by running tests in parallel across multiple virtual machines.

Reporting on Test Results

"Wait," you might be asking, "My test says 'Complete' but what happens if it fails?"

Unfortunately, Sauce has no way to determine whether your test passed or failed automatically, since it is determined entirely by your business logic. You can, however, tell Sauce about the results of our tests automatically using the Sauce python client and adding these lines to your test.

# this authenticates you from sauceclient import SauceClient sauce_client = SauceClient("YOUR_SAUCE_USERNAME", "YOUR_SAUCE_ACCESSKEY")

# this belongs in your test logic sauce_client.jobs.update_job(driver.session_id, passed=True)

Page 28 Copyright Sauce Labs 2015

Instant Ruby Tests with Sauce Labs

Prerequisites Quick Start Code Sample Running Local Tests Running Tests in Parallel Reporting on Test Results

Prerequisites

Before getting started, you should read the Best Practices for Running Tests with Sauce Labs.

Make sure you have the selenium-webdriver gem installed If you want to run tests in parallel you will need the parallel_tests gem You will also need to install the sauce_whisk gem to report the results of your test to your Sauce Labs dashboard

Quick Start

It's very easy to get your existing Ruby tests up and running on Sauce.

If you wanted to run a test on Selenium locally, you would initiate a driver for the browser you want to test against like this.

Local Ruby Test Example

driver = Selenium::WebDriver.for :firefox

To run an existing test on Sauce, all you need to change is your driver definition, and make sure that the sauce_endpoint variable includes your Sauce USERNAME and ACCESSKEY.

Remote Ruby Test on Sauce Example

driver = Selenium::WebDriver.for :remote, :url => sauce_endpoint, :desired_capabilities => caps sauce_endpoint = "http://YOUR_SAUCE_USERNAME:[email protected]:80/wd/hub"

Once you have your tests set up to run in the Sauce cloud, you need to define the platform, browser, and version you want the test to run against, which is where :desired_capabilites and caps come into play.

caps = { :platform => "Mac OS X 10.9", :browserName => "Chrome", :version => "31" }

You can manually enter the values you want for :platform, :browsername, and :version, or you can use the handy Sauce Labs Platform Configurator to generate the caps values for any combination of platform, browser, and browser version.

Code Sample

This code examples shows a very simple test to see if the title of Google's home page has changed, but it includes everything you need to set up and run an automated test on Sauce.

Page 29 Copyright Sauce Labs 2015

Example Ruby Code for Testing with Sauce Labs Expand source require "selenium/webdriver"

sauce_endpoint = "http://YOUR_SAUCE_USERNAME:[email protected]:80/wd/hub"

caps = { :platform => "Mac OS X 10.9", :browserName => "Chrome", :version => "31" }

driver = Selenium::WebDriver.for :remote, :url => sauce_endpoint, :desired_capabilities => caps driver.manage.timeouts.implicit_wait = 10 driver.navigate.to "http://www.google.com"

raise SystemError("Unable to load Google.") unless driver.title.include? "Google"

query = driver.find_element :name, "q" query.send_keys "Sauce Labs" query.submit

puts driver.title driver.quit

Running Local Tests

Running Tests in Parallel

Now that you’re running tests on Sauce, you may wonder how you can run your tests faster. One way to increase speed is by running tests in parallel across multiple virtual machines.

Check out the topics under Running Tests in Parallel with Ruby for code examples and instructions on using popular testing frameworks for running tests in parallel with Ruby.

Reporting on Test Results

Selenium is a protocol focused on automating the actions of a browser, and as such, it doesn't encompass the notions of a "test" that "passes" or

Page 30 Copyright Sauce Labs 2015

"fails." Sauce Labs lets you notify us of test status using our REST API. All you need is the ID Sauce Labs gave the job, and the status your test ended with. Then, you can use the Update Job method on the REST API to set the job's status.

The Job ID is the simplest part of the process. The ID assigned to each job by Sauce Labs is the same as the Session ID for the corresponding Selenium session, which you can pull off the driver like so:

job_id = driver.session_id

Using the REST API is best done with the sauce_whisk gem, which assumes you've set your Sauce username and access key as the SAUCE_US ERNAME and SAUCE_ACCESS_KEY environment variables.

This example shows how you would modify the example RSpec spec_helper.rb file to check the status of the test, and pass that to the REST API.

Modifying spec_helper.rb to Report Test Results to the Dashboard Using Expand source the sauce-whisk Gem require "sauce_whisk"

RSpec.configure do |config| config.around(:example, :run_on_sauce => true) do |example| @driver = SauceDriver.new_driver @job_id = @driver.session_id begin example.run ensure SauceWhisk::Jobs.change_status job_id, !example.exception.nil? @driver.quit end end end

That's it! Your tests will now be reporting their status to the Sauce Labs REST API.

Page 31 Copyright Sauce Labs 2015

Known Issues and Bugs

Key Summary

WEB-2302 Not able to launch the Android Emulators with Google API in a Manual Session

WEB-2300 Large log.json File Causes Test Details Page to Load Slow

WEB-2298 The new Manual UI does not work in IE lower than 11

WEB-2290 Build Doesn't Match Results Between Parent & Sub Account

WEB-2282 Subaccounts with Admin Privileges can't set the Parent account as a parent when creating new users

WEB-2268 Users don't see a complete list of their subaccounts.

WEB-2261 Credentials don't get propagated to SC copy command

WEB-2251 Manual configurator broken on mobile screens

WEB-2239 Number of tests in builds is not correct

WEB-2223 Next Renewal Date on Billing Page is incorrect

WEB-2170 Manual Launcher: Amazon Kindle Fire Emumlator Android 2.3 combo is not working

WEB-2146 500 Internal Error when creating a sub-account with existing email address

WEB-2054 Parent account can't delete sub-account's job

WEB-1880 Scrolling over commands on test details page doesn't work in a specific case

WEB-1863 This build says '1 failed' but no test in the list is marked as failed

WEB-1833 Videos Freeze

WEB-1774 Interactions with manual sessions fail if using computers with touch screen

WEB-1784 Manual sessions go blank and throw access error randomly

WEB-1770 Users being kicked out of their sessions on every deploy

WEB-1622 Sauce Connect outputs HTML when encountering JSON parse error

WEB-1338 Bad test data in YUI js unit tests cause job page not to load

WEB-1035 Manual Testing does not work using Windows 8 or Windows 10 Chrome or IE, very slow on FF

SC-213 Closing a Sauce Connect Windows tunnel does not close the tunnel in the UI

MOBILE-600 Emulators fail to start when the url to open has special characters

MOBILE-545 System Time Changes During Test

MOBILE-534 Live screencast doesn't load right away on test detials page / needs a few refreshes

MOBILE-488 Cannot launch tests with Android 2.3 (Manual & Automated)

MOBILE-364 VM fails to start for Samsung Galaxy S3 Emulator with Android 4.1

MOBILE-344 35.2 Mb app is causing Insufficient Space errors

MOBILE-307 net::ERR_PROXY_CONNECTION_FAILED on Android 4.4 5.0 when using Sauce Connect

MOBILE-305 Android 5.0 Mobile sessions can't resolve DNS entries, initially

Page 32 Copyright Sauce Labs 2015

Introducing Sauce Labs

Welcome to Sauce Labs! Topics in this section are intended to provide you with an introduction to some of the things you can do with Sauce, an overview of what you'll find in the Sauce Labs interface, answers to some frequently asked questions, and a listing of the browsers and operating systems you can use with our Web interface.

A Tour of the Sauce Labs Interface Sauce Labs FAQs Sauce Labs Basics Supported Browsers and Operating Systems

Page 33 Copyright Sauce Labs 2015

A Tour of the Sauce Labs Interface The Sauce Labs web interface provides a convenient location for managing your account and teams, checking the results of recent Dashboard User Interface Legend and archived tests and builds, managing Sauce Connect tunnels, and running manual tests. UI Element Description

1 Click the Tunnels tab to set up Dashboard User Interface and view your active Sauce Connect tunnels.

2 In the Archives tab you'll find a complete list of the manual tests and the automated tests and builds you've run on Sauce Labs. Check out the topics in M anaging Archived Test and Build Results for more information.

3 The number of concurrent VMs allowed for your account. Check out the topics in Runnin g Tests in Parallel with Sauce Labs for information on running concurrent tests.

4 The number of testing hours Account Profile User Interface allowed for your account.

When you open the Account Profile drawer, you will see links for 5 The Account Profile drawer managing your account information such as username and opens to provide you with links password, managing teams and sub-accounts, and user settings for managing your account and such as your Sauce Labs access key. team, as described in the Acco unt Profile User Interface.

6 In the Automated Builds tab you can see the results for your most recent automated builds.

7 In the Automated Tests tab you can see the results for your most recent automated tests.

8 In the Manaual Tests tab you can see the results for your most recent manual tests.

Account Profile User Interface

UI Element Description

1 In the Team Management pag e you'll find usage information for your accounts and sub accounts, as well as links to add users to your account, set up a Sauce Connect tunnel, and enable single sign-on (SSO). Check out the topics under Managing Sauce Labs Accounts and Teams for more information.

Page 34 Copyright Sauce Labs 2015

2 In the My Account page you'll find usage information for your specific account, your Sauce Labs access key, Sauce Connect tunnels associated with your account, and users associated with your account.

3 In the User Settings page you can edit your username and password, email address, and regenerate your access key.

Page 35 Copyright Sauce Labs 2015

Sauce Labs FAQs

How does billing work? What do you charge per minute? Will my firewall block your service? Can I use Sauce if my tool is behind basic HTTP Auth? My automated tests run serially. Can I still use Sauce? How do I run my automated tests in parallel? Can I use Sauce for load testing my app? How is Sauce different from Selenium Grid? Can I reuse a session within multiple tests?

How does billing work? What do you charge per minute?

For every test you run, the actual test time will be slightly shorter than the overall job duration, or what is billed by Sauce. This is because after the test finishes, our test servers remain fully dedicated to your account as it processes the video and screenshots, and uploads all artifacts to your account page.

It's important to point out that this time is necessary only on our servers. If you are running multiple tests at once, they will be assigned to new machines so there shouldn't be any impact on the overall test suite execution time. What OS and browser combinations do you support?

We support Firefox, Internet Explorer, Google Chrome, Safari, and Opera on Windows, Linux and Mac. Check out our complete list of browser and operating system platforms for details.

Will my firewall block your service?

No. By using Sauce Connect, you can connect to our service securely without opening any ports in your firewall.

Sauce Connect creates a reliable, encrypted connection between your firewalled server and ours, and eliminates the need to whitelist IP addresses (since all requests come from the machine in which Sauce Connect is running).

Can I use Sauce if my tool is behind basic HTTP Auth?

For HTTP Authentication, we behave as Selenium does, leveraging the use of the RFC 1738 specification, which you enter within the url. So you just have to change the url in your tests to include the Auth info. Here's an example: http://username:[email protected]/secretarea/. You can also read more about this here. This should work with all our browsers, but if you find any obstacles with your site, please let us know.

Note: If you don't yet have any HTTP Auth setup and all you want to do is have Sauce test your internal app, we strongly recommend using Sauc e Connect instead.

My automated tests run serially. Can I still use Sauce?

Yes. With Sauce, you'll benefit from our many browser configurations and features such as live video recording.

How do I run my automated tests in parallel?

Check out blog posts on Running Selenium tests in parallel, as well as the topics under Running Tests in Parallel with Sauce Labs

Can I use Sauce for load testing my app?

Sauce is meant to be used for browser-based functional testing (also sometimes called "acceptance testing" or "UI testing"). And even though parallelization inherently causes load, the features other load testing services provide are probably better for your needs. We're friends with and fans of Neustar (formally BrowserMob) and BlazeMeter.

How is Sauce different from Selenium Grid?

Check out our Sauce vs. Local doc to see some differences.

Can I reuse a session within multiple tests?

Page 36 Copyright Sauce Labs 2015

One of our main criteria for a mature set of tests is "test independence." If your tests are completely independent from each other, the complete test suite will be ready for scaling. And believe us, scaling is never too far off. Once you reach a certain amount of tests, the time needed to run them serially will exceed the practical limit and:

1. Your team will stop running the tests as regularly as needed because it takes too long 2. The tests will start eating more and more of the development time

This can also depend on the context. In case it's really needed, we suggest preventing the test framework on your side from calling the start() and stop() commands between tests.

Page 37 Copyright Sauce Labs 2015

Sauce Labs Basics

Sauce Labs is your one-stop shop for manual and automated functional testing of your web and mobile applications. Adding Sauce to your application development process is the secret to speeding up your development process, and delivering robust applications to your users.

Your Own Personal Selenium Grid Mobile Application Testing on Emulators and Real Devices Made Easy with Appium No Public Access, No Problem Save Time by Running Tests in Parallel Build Your Continuous Integration and Continuous Deployment Pipeline Many Hands Make Light Work

Your Own Personal Selenium Grid

Automated testing with Sauce is based on Selenium WebDriver, which provides the simple yet highly useful ability to automate browsers. At the most basic level, Sauce Labs provides a Selenium Grid that enables you to run multiple tests for multiple browser configurations from the same test scripts. For each test that you run on Sauce, we spin up a pristine virtual machine (VM) to match the desired capabilities of your test (such as the operating system, browser, and browser version). You can write your test scripts in any one of several popular languages, from Java to Ruby, and then use them to run tests from your local machine on the browsers in the Sauce Labs testing cloud. All you need to do is provide your Sauce Labs authentication credentials and the desired capabilities for your test, along with the path to the site or application you want to test, and Sauce Labs does the rest! Our Quick Start topics provide information on how to get your existing tests up and running on Sauce, while our Tutorials provi de more step-by-step instructions on setting up a complete testing toolchain with example tests that you can run.

Mobile Application Testing on Emulators and Real Devices Made Easy with Appium

Appium is an open-source tool for automating native, mobile web, and hybrid applications on iOS and Android platforms. Like Selenium, it works with the language of your choice. Sauce also offers you the choice of testing on emulators or real devices, because while functional tests on emulators might work out most of your bugs, you can't know for sure that you've squashed all of them out until you run the real thing.

No Public Access, No Problem

Is the application you want to test on your local machine, or sequestered behind a firewall? You can still test it by using Sauce Connect. Sauce Connect uses a secure tunnel protocol that provides specific Sauce machines with access to your local network.

Want to test your application "in the wild," but you aren't ready to make it publicly available yet? You can use Sauce Storage to load the assets you want to test into our cloud, and then point your tests to their location. After seven days the assets are cleared out, and only you and other users associated with your account can access them while they're available. There's no extra charge, and you get to see how your applications behave over a network connection without having to commit to a hosting solution just for the purpose of testing.

Save Time by Running Tests in Parallel

The great advantage in having access to a Selenium grid is being able to run multiple tests for multiple browser configurations at the same time, vastly speeding up the overall testing process. The number of tests you can run in parallel is set by the concurrency limits of your service plan, all you have to do configure your test framework as described in the section Running Tests in Parallel with Sauce Labs.

Build Your Continuous Integration and Continuous Deployment Pipeline

Continous Integration (CI) and Continuous Deployment (CD) are key components of DevOps and Agile software development, enabling rapid, iterative development of applications. Automated testing is an integral part of the CI and CD build process that cuts down on overall cycle time, and provides the means to incorporate new features, functionality, and bug fixes on a daily and even hourly basis. You can incorporate Sauce Labs automated testing into your CI and CD toolchain with the Sauce OnDemand plugin for popular CI platforms such as Bamboo, TeamCity, Jenkins, Travis CI, and Circle CI, as described in the section Using Sauce Labs with Continuous Integration Platforms.

Many Hands Make Light Work

Do you have multiple team members working on your project, and you want each of them to be able to integrate and test their code? You can set up each of them with a subaccount and allocate VMs to them based on the concurrency limits of your plan. You can even use your already-established single sign-on (SSO) mechanism to authenticate users to your Sauce Labs account, as described in Managing Sauce Labs Accounts and Teams.

Page 38 Copyright Sauce Labs 2015

Supported Browsers and Operating Systems

Sauce Labs supports using web and mobile browsers with the web interface at two levels.

Full Support - all browsers at this level will have identical behavior when accessing the Sauce Labs web interface Functional Support - browsers at this level will be fully functional with the web interface, but may have differences in the way content is displayed

Chrome Firefox Safari IE iOS Android Safari Browser

Full Support Latest, Latest - 1 Latest, Latest - 1 Latest, Latest -1 (6.2+) Older than 2 years (11+ | Latest Latest Win 8)

Functional Older than 6 months Older than 6 months Older than 2 years 9+ iOS 7.1+ Android 5.0+ Only (40+) (40+) (5.1+)

Page 39 Copyright Sauce Labs 2015

Running Tests with Sauce Labs

Manual testing, automated testing, mobile app testing, running tests in parallel - so many ways to test your sites and applications with Sauce Labs! Topics in this section cover all the possibilities of testing with Sauce , including using Sauce Connect to test sites and apps on localhost or behind a firewall, using Sauce Storage for test assets, and the various settings you can use to configure your tests.

Tips and Best Practices for Running Tests with Sauce Labs Best Practices for Running Tests with Sauce Labs Best Practice: Avoid External Test Dependencies Best Practice: Avoid Dependencies between Tests to Run Tests in Parallel Best Practice: Don't Use Brittle Locators in Your Tests Best Practice: Have a Retry Strategy for Handling Flakes Best Practice: Keep Functional Tests Separate from Performance Tests Best Practice: Use Build IDs, Tags, and Names to Identify Your Tests Best Practice: Use Environment Variables for Authentication Credentials Best Practice: Use Explicit Waits Best Practice: Use the Latest Version of Selenium Client Bindings Best Practices: Use Small, Atomic, Autonomous Tests Tips for Lean, Speedy Tests with Sauce Labs Handling Authentication Basic HTTP Authentication Injecting Cookies to Bypass Authentication Dialogs Running an AutoIt Script as a Pre-run Executable to Handle Windows Security Authentication Dialogs Pre-Run Executables Setting Up Pre-Run Executables Test Configuration and Annotation Configuring Tests with the WebDriver API DesiredCapabilities Configuring Tests with the Sauce Labs REST API Configuring Tests with Selenium's JavaScript Executor Test Configuration Options Examples of Desired Capabilities for iWebDriver and Appium iOS Tests Manual Testing with Sauce Labs Running a Manual Testing Session Starting Manual Tests from Automated Tests Troubleshooting Manual Tests Automated Testing with Sauce Labs Troubleshooting Automated Tests Common Error Messages Mobile Testing with Sauce Labs Mobile Testing with Appium Support and Requirements for Mobile Testing Supported Android Emulators Supported Mobile Operating Systems Requirements for Testing Mobile Native and Hybrid Applications Manual Testing for Mobile Apps Getting to the JavaScript Console for Manual iOS Browser Tests Running Emulator and Simulator Mobile Tests Types of Mobile Tests FAQs for Mobile Testing Example Appium Mobile Testing Scripts Android Example Scripts Written for Mobile Testing on Sauce Labs Example Java Script for Testing Android Mobile Applications on Sauce Labs Example Node.js Script for Testing Android Mobile Applications on Sauce Labs Example Python Script for Testing Android Mobile Applications on Sauce Example Ruby Script for Testing Android Mobile Applications on Sauce Labs iOS Example Scripts for Mobile Testing on Sauce Example Java Script for Testing iOS Mobile Applications on Sauce Labs Example Node.js Script for Testing iOS Mobile Applications on Sauce Example PHP Script for Testing iOS Mobile Applications on Sauce Labs Example Python Script for Testing iOS Mobile Applications on Sauce Labs Example Ruby Script for Testing iOS Mobile Applications on Sauce Labs Running Tests in Parallel with Sauce Labs Running Tests in Parallel with C# Running C# Tests in Parallel with Gallio and MBUnit Running C# Tests in Parallel with PNUnit Running Tests in Parallel with Java Running Java Tests in Parallel with JUnit Running Java Tests in Parallel with TestNG Running Tests in Parallel with PHP Running PHP Tests in Parallel with PHPUnit and Paratest Running Tests in Parallel with Python

Page 40 Copyright Sauce Labs 2015

Running Python Tests in Parallel with nose Running Python Tests in Parallel with pytest Running Tests in Parallel with Ruby Running Ruby Tests in Parallel with RSpec Troubleshooting Parallel Tests on Sauce Labs Viewing and Managing Sauce Labs Test Results Viewing Screenshots, Commands, Logs, and Metadata for Sauce Test Results Using Status Badges and the Browser Matrix Widget to Monitor Test Results Sharing the Results of Sauce Labs Tests Embedding Test Results in HTML Pages Building Links to Test Results Managing Archived Test and Build Results Searching for Test Results and Builds on Your Archive Page Search Fields and Operators Using Favorites to Save Your Searches Deleting Test and Build Results Changing the Layout of Your Archives Page Using Sauce Storage for Test Assets Uploading Assets to Sauce Storage Using C#

Page 41 Copyright Sauce Labs 2015

Tips and Best Practices for Running Tests with Sauce Labs

These topics cover tips and best practices for automated Selenium and Appium testing, and for running tests in the Sauce Labs browser cloud.

Best Practices for Running Tests with Sauce Labs Tips for Lean, Speedy Tests with Sauce Labs Handling Authentication

Page 42 Copyright Sauce Labs 2015

Best Practices for Running Tests with Sauce Labs

These topics describe general best practices for web and mobile application testing, as well as best practices for testing with Sauce Labs.

Best Practice: Avoid External Test Dependencies Best Practice: Avoid Dependencies between Tests to Run Tests in Parallel Best Practice: Don't Use Brittle Locators in Your Tests Best Practice: Have a Retry Strategy for Handling Flakes Best Practice: Keep Functional Tests Separate from Performance Tests Best Practice: Use Build IDs, Tags, and Names to Identify Your Tests Best Practice: Use Environment Variables for Authentication Credentials Best Practice: Use Explicit Waits Best Practice: Use the Latest Version of Selenium Client Bindings Best Practices: Use Small, Atomic, Autonomous Tests

Page 43 Copyright Sauce Labs 2015

Best Practice: Avoid External Test Dependencies

Use Setup and Teardown

If there are are "prerequisite" tasks that need to be taken care of before your test runs, you should include a setup section in your script that executes them before the actual testing begins. For example, you may need to to log in to the application, or dismiss an introductory dialog that pops up before getting into the application functionality that you want to test. Similarly, if there are "post requisite" tasks that need to occur, like closing the browser, logging out, or terminating the remote session, you should have a teardown section that takes care of them for you.

Don't Hard Code Dependencies on External Accounts or Data

Development and testing environments can change significantly in the time between the writing of you test scripts and when they run, especially if you have a standard set of tests that you run as part of your overall testing cycle. For this reason, you should avoid building into your scripts any hard coded dependencies on specific accounts or data, Instead, use API requests to dynamically provide the external inputs you need for your tests.

Page 44 Copyright Sauce Labs 2015

Best Practice: Avoid Dependencies between Tests to Run Tests in Parallel

Dependencies between tests prevent tests from being able to run in parallel. And running tests in parallel is by far the best way to speed up the execution of your entire test suite. It's much easier to add a virtual machine than to try to figure out how to squeeze out another second of performance from a single test.

What are dependencies? Imagine a test suite with two tests:

Java Example of Test Dependencies Expand source @Test public void testLogin() { // do some stuff to trigger a login assertEquals("My Logged In Page", driver.getTitle()); }

@Test public void testUserOnlyFunctionality() { driver.findElement(By.id("userOnlyButton")).click(); assertEquals("Result of clicking userOnlyButton", driver.findElement(By.id("some_result"))); }

PHP Example of Test Dependencies Expand source assertEquals("My Logged In Page", $this->title()); }

function testUserOnlyFunctionality() { $this->byId('userOnlyButton')->click(); $this->assertTextPresent("Result of clicking userOnlyButton"); }

In both of these examples, testLogin() triggers the browser to log in and asserts that the login was successful. The second test clicks a button on the logged-in page and asserts that a certain result occurred.

This test suite works fine as long as the tests run in order. But second test makes an assumption that you are already logged in, which creates a dependency on the first test. If these tests run at the same time, or if the second one runs before the first one, the browser's cookies will not yet allow Selenium to access the logged-in page, and the second test fails. You can get rid of this dependency by making sure that each test can run independently independently of the others, as shown in these examples.

Page 45 Copyright Sauce Labs 2015

Java Example of Corrected Test Dependency Expand source @Test public void doLogin() { //do some stuff to trigger a login assertEquals("My Logged In Page", driver.getTitle()); }

@Test public void testLogin() { this.doLogin(); }

@Test public void testUserOnlyFunctionality() { this.doLogin(); driver.findElement(By.id("userOnlyButton")).click(); assertEquals("Result of clicking userOnlyButton", driver.findElement(By.id("some_result"))); }

PHP Example of Corrected Test Dependency Expand source assertEquals("My Logged In Page", $this->title()); }

function testLogin() { $this->doLogin(); }

function testUserOnlyFunctionality() { $this->doLogin(); $this->byId('userOnlyButton')->click(); $this->assertTextPresent("Result of clicking userOnlyButton"); }

The main point is that it is dangerous to assume any state when developing tests for your app. Instead, you should find ways to quickly generate desired states for individual tests. In the example, this is accomplished with the doLogin() function, which generates a logged-in state instead of assuming it. You might even want to develop an API for the development and test versions of your app that provides URL shortcuts that generate common states. For example, a URL that's only available in test that creates a random user account and logs it in automatically.

Page 46 Copyright Sauce Labs 2015

Best Practice: Don't Use Brittle Locators in Your Tests

WebDriver provides a number of locator strategies for accessing elements on a webpage. It's tempting to use complex XPath expressions like // body/div/div/*[@class="someClass"] or CSS selectors like #content .wrapper .main. While these might work when you are developing your tests, they will almost certainly break when you make unrelated refactoring changes to your HTML output.

Instead, use sensible semantics for CSS IDs and form element names, and try to restrict yourself to using these semantic identifiers. For example, in Java you could designate elements with $this->byId() or $this->byName() or, in the example of PHP, you could use $this-> byId() or $this->byName() . This makes it much less likely that you'll inadvertently break your page by shuffling around some lines of code.

Page 47 Copyright Sauce Labs 2015

Best Practice: Have a Retry Strategy for Handling Flakes

There will always be flaky tests, and tests that once breezed through with no problem can fail for what seems like no reason. The trick is figuring out whether a test that fails does so because it found a real problem in your app functionality, or because there was an issue with the test itself.

The best way to handle this problem is log your failing tests into a database and then analyze them. Even tests that fail intermittently with no apparent cause may turn out to have a pattern when you are able to analyze them in detail and as a larger data set. If this is beyond the scope of your testing setup, the next best strategy is to log your failing cases into a log file that records the browser, version, and operating system for those tests, and then retry those tests. If they continue to fail after a second or third retry, chances are that the issue is with the functionality you're testing, rather than the test itself. This isn't a total solution for dealing with flakes, but it should help you get closer to the source of the problem.

Page 48 Copyright Sauce Labs 2015

Best Practice: Keep Functional Tests Separate from Performance Tests It's very important to maintain a distinction between functional tests and performance tests to make sure that you understand what your test results should be, and not have performance tests fail because they not have been designed with system capabilities and the desired outputs in mind. You should use Sauce Labs exclusively for functional testing, and for being able to integrate your functional testing into your continuous delivery pipeline.

Functional tests should, as the name indicates, test some functionality or feature of your application. The output of these tests should be either a simple "pass" or "fail" - either your functionality worked as expected, or it didn't.

Performance tests, in contrast, should gauge and output performance metrics. For example, can your application server handle a particular load, and does it behave as expected when you push it to its limit? These types of tests are better undertaken with a testing infrastructure that has been specifically developed for performance testing, so all baseline performance metrics are well established and understood before you start the test.

By maintaining the distinction between functional and performance tests, and the different outputs that you expect from them, you should be able to more precisely design your tests to uncover the specific kinds of issues that you need to address to make your app more robust under any conditions.

Page 49 Copyright Sauce Labs 2015

Best Practice: Use Build IDs, Tags, and Names to Identify Your Tests

When you set the desired capabilities for your test, you can also add identifying information such as a name, tags, and build numbers that you can then use to filter results in your test results or Web Archive page, or to identify builds within your continuous integration pipeline.

Build, Tags, Name Example for Java Expand source DesiredCapabilities caps = DesiredCapabilities.firefox(); caps.setCapability("platform", "Windows XP"); caps.setCapability("version", "37.0"); caps.setCapability("name", "Web Driver demo Test"); caps.setCapability("tags", "Tag1"); caps.setCapability("build", "build-1234"); WebDriver driver = new RemoteWebDriver( new URL("http://YOUR_USERNAME:[email protected]:80/wd/hub"), caps);

Python Example Build Expand source desired_cap = { 'platform': "Mac OS X 10.9", 'browserName': "chrome", 'version': "31", 'build': "build-1234", }

Python Example Tags Expand source desired_cap = { 'platform': "Mac OS X 10.9", 'browserName': "chrome", 'version': "31", 'build': "build-1234", 'tags': [ "tag1", "tag2", "tag3" ]

}

Page 50 Copyright Sauce Labs 2015

Best Practice: Use Environment Variables for Authentication Credentials As a best practice, we recommend setting your Sauce Labs authentication credentials as environment variables that can be referenced from within your tests, as shown in this code example. This provides an extra layer of security for your tests, and also enables other members of your development and testing team to write tests that will authenticate against a single account.

Setting Environment Variables for Mac OS X/Linux 1. In Terminal, enter vi .bash_profile, and then press Enter. 2. Type i to insert text. 3. Enter these lines:

export SAUCE_USERNAME="your Sauce username" export SAUCE_ACCESS_KEY="your sauce access key"

4. Press Escape. 5. Hold Shift and press Z twice (z z) to save and quit.

Setting Environment Variables for Authentication Credentials on Windows 1. Click Start on the task bar. 2. For Search programs and fields, enter Environment Variables. 3. Click Edit the environment variables. This will open the System Properties dialog. 4. Click Environment Variables. This will open the Environment Variables dialog. 5. In the System variables section, click New. This will open the New System Variable dialog. 6. For Variable name, enter SAUCE_USERNAME. 7. For Variable value, enter your Sauce username. 8. Click OK. 9. Repeat 4 - 8 to set up the SAUCE_ACCESS_KEY.

Java Example of Using Environment Variables for Authentication

public static final String USERNAME = System.getenv("SAUCE_USERNAME"); public static final String ACCESS_KEY = System.getenv("SAUCE_ACCESS_KEY"); public static final String URL = "http://" + USERNAME + ":" + ACCESS_KEY + "@ondemand.saucelabs.com:80/wd/hub";

PHP Example of Using Environment Variables for Authentication

define('SAUCE_HOST', SAUCE_USERNAME.':'.SAUCE_ACCESS_KEY.'@ondemand.saucelabs.com');

Python Example of Using Environment Variables for Authentication

driver = webdriver.Remote(

command_executor='http://YOUR_USERNAME:[email protected]:80/wd/ hub', desired_capabilities=desired_cap)

Ruby Example of Using Environment Variables for Authentication

Page 51 Copyright Sauce Labs 2015

username = ENV["SAUCE_USERNAME"] access_key = ENV["SAUCE_ACCESS_KEY"] remote_server_url = "http://#{username}:#{access_key}@ondemand.saucelabs.com:80/wd/hub"

C# Example of Using Environment Variables for Authentication

using System; //Importing the System library

internal readonly static string SAUCE_LABS_ACCOUNT_NAME = System.Environment.GetEnvironmentVariable("SAUCE_USERNAME"); internal readonly static string SAUCE_LABS_ACCOUNT_KEY = System.Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY");

Page 52 Copyright Sauce Labs 2015

Best Practice: Use Explicit Waits

There are many situations in which your test script may run ahead of the website or application you're testing, resulting in timeouts and a failing test. For example, you may have a dynamic content element that, after a user clicks on it, a loading appears for five seconds. If your script isn't written in such a way as to account for that five second load time, it may fail because the next interactive element isn't available yet.

The general advice from the Selenium community on how to handle this is to use explicit waits. While you could also use implicit waits, an implicit wait only waits for the appearance of certain elements on the page, while an explicit wait can be set to wait for broader conditions. Selenium guru Dave Haeffner provides an excellent example of why you should use explicit waits on his Elemental Selenium blog.

These code samples, from the SeleniumHQ documentation on explicit and implicit waits, shows how you would use an explicit wait. In their words, this sample shows how you would use an explicit wait that "waits up to 10 seconds before throwing a TimeoutException, or, if it finds the element, will return it in 0 - 10 seconds. WebDriverWait by default calls the ExpectedCondition every 500 milliseconds until it returns successfully. A successful return for ExpectedCondition type is Boolean return true, or a not null return value for all other ExpectedCondi tion types."

Python Example of an Explicit Wait from SeleniumHQ.org

from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0 from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0

ff = webdriver.Firefox() ff.get("http://somedomain/url_that_delays_loading") try: element = WebDriverWait(ff, 10).until(EC.presence_of_element_located((By.ID, "myDynamicElement"))) finally: ff.quit()

Java Example of an Explicit Wait from SeleniumHQ.org

WebDriver driver = new FirefoxDriver(); driver.get("http://somedomain/url_that_delays_loading"); WebElement myDynamicElement = (new WebDriverWait(driver, 10)) .until(ExpectedConditions.presenceOfElementLocated(By.id("myDynamicElement")));

C# Example of an Explicit Wait from SeleniumHQ.org

IWebDriver driver = new FirefoxDriver(); driver.Url = "http://somedomain/url_that_delays_loading"; WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10)); IWebElement myDynamicElement = wait.Until((d) => { return d.FindElement(By.Id("someDynamicElement")); });

Ruby Example of an Explicit Wait from SeleniumHQ.org

Page 53 Copyright Sauce Labs 2015

require 'selenium-webdriver' driver = Selenium::WebDriver.for :firefox driver.get "http://somedomain/url_that_delays_loading" wait = Selenium::WebDriver::Wait.new(:timeout => 10) # seconds begin element = wait.until { driver.find_element(:id => "some-dynamic-element") } ensure driver.quit end

Page 54 Copyright Sauce Labs 2015

Best Practice: Use the Latest Version of Selenium Client Bindings

The Selenium Project is always working to improve the functionality and performance of its client drivers for supported languages like Java, C#, Ruby, Python, and JavaScript, so you should always be using the latest version of the driver for your particular scripting language. You can find these on the Downloads page of the SeleniumHQ website, under Selenium Client & WebDriver Language Bindings.

Page 55 Copyright Sauce Labs 2015

Best Practices: Use Small, Atomic, Autonomous Tests

When a test fails, the most important thing is knowing what went wrong so you can easily come up with a fix. The best way to know what went wrong is to keep three words in mind when designing your tests: Small, Atomic, and Autonomous.

Small

Small refers to the idea that your tests should be short and succinct. If you have a test suite of 100 tests running concurrently on 100 VMs, then the time it will take to run the entire suite will be determined by the longest/slowest test case. Keeping your tests small ensures that your suite will run efficiently and provide you with results faster.

Atomic

An atomic test is one that focuses on testing a single feature, and which makes clear exactly what it is that you're testing. If the test fails, then you should also have a very clear idea of what needs to be fixed.

Autonomous

An autonomous test is one that runs completely independently of other tests, and is not dependent on the results of one test to run successfully. In addition, an autonomous test should use its own data to test against, and not create potential conflicts with other tests over the same data.

Page 56 Copyright Sauce Labs 2015

Tips for Lean, Speedy Tests with Sauce Labs

It's a reality that tests run on Sauce Labs will always have some latency, compared to running locally, due to there being an Internet in the middle between your tests and our browsers. So, rather than running in the same machine, each Selenium request is sent through the wire to our VMs and receives a response back through the same channels. These are some other common factors that contribute to test duration, and a few tips for using your minutes efficiently.

Identify Time Sinks If Using Sauce Connect . . . Optimize Your Test Scripts Integrate intelligently with Sauce

Identify Time Sinks

If you're testing against a local application server, you will see some additional latency due to the distance from our browsers to your server.

The same applies if you're using Sauce Connect to test against a firewalled server. When you're running Connect, it sends all requests from Sauce's browsers back through the machine running Sauce-Connect.jar, such that they appear to originate from that machine. This means that all data sent between the Sauce cloud and the site you're testing must travel first to your local machine, then back out to the application under test or to the Sauce cloud.

While Connect also includes features that help to reclaim this time - such as caching static site resources in our cloud - this added security comes with some extra running time.

If you're using Capybara, be aware that this integration testing tool is especially chatty when formulating tests (and the same may go for other wrappers that run on top of the Selenium library). For example, entering a value in an input field can spawn five Selenium commands: verifying that the element is displayed, getting its name, determining its type, clearing it, and then entering the text. While these steps help make tests robust and informative, if you're writing scripts by hand, they are usually not necessary.

Mobile browsers (iOS and Android) require a little extra time to fire up, as they launch first the device emulator, then a browser within it.

If Using Sauce Connect . . .

Reduce the unnecessary traffic that goes through the Sauce Connect tunnel, this in turn will help the test run faster. Here are some suggestions to accomplish this on your end:

1. In your test, determine the URLs that are publicly available over the internet and list their domains with the -D, --direct-domains flag when you start the Sauce Connect tunnel. The -D, --direct-domains flag takes a comma-separated list of domains which will be relayed directly through the internet, instead of through the tunnel.

2. Determine those resources in your application that are not necessary for your test verifications (for example, images or advertisements). List the domains for these resources using the -F, --fast-fail-regexps flag when you start the Sauce Connect tunnel. The -F, - -fast-fail-regexps flag takes a comma-separated list of domains and any requests matching one of these will be dropped instantly, not going through the tunnel. This will allow your application to load faster.

You can find information about all the flags that you can use with Sauce Connect in the topic Sauce Connect Command Line Reference.

Optimize Your Test Scripts

The main thing you can do to decrease latency is to break your test down into small, atomic, autonomous tests. Our best practices topic goes into more detail, but this will approach has several benefits for decreasing overall testing time.

That said, there are a few things that should help bring this down. We mainly recommend breaking the test down into smaller chunks, because:

1. You will be able to run independent tests in parallel, decreasing your build time. 2. It will make your tests more robust, since independent sections are tested without a pre-existing application state from earlier actions. 3. If you're using Sauce Connect, you have use of a proxy that intelligently caches static resources, so that later tests don't have to re-load those from scratch.

A couple of other tips:

CSS locators are generally faster than XPATH locators, especially if you're running tests in Internet Explorer. (Also, locating by an

Page 57 Copyright Sauce Labs 2015

element's unique ID is usually the most stable method.) Cut down on unneeded chatter. If your test contains an abundance of GET requests (get text, get displayed, etc.), this will contribute disproportionately to the test duration. Each command takes time to transmit to our cloud and pass back a result; since these steps execute so quickly, the time required to send the command through the Internet is proportionally larger, causing the test duration to inflate more than usual. Reduce repetition: Check your test logs for duplicated requests and determine whether this is necessary.

Our Best Practices for Running Tests with Sauce Labs provide several tips for optimizing your test scripts.

Integrate intelligently with Sauce

Besides these script-dependent factors, there are a few additional Sauce processes that happen on a per-job basis, aside from the test, that add some extra time. These include:

Automatic screenshots and video recording: We capture an automatic screenshot after each page-changing command, and record video of each test by default. These are post-processed and uploaded after the test is over. The time this takes is added to the test, as it means we have to keep that machine fully dedicated until this is finished. These options are turned on by default, and can be disabled in your script as described in the topic Test Configuration Options.

Browser start time. The time it takes browsers to start varies with browsers and versions. This is inherent to the complexity of browser testing and how Selenium handles each browser boot, so this can't be eliminated. This happens before the test starts and is added to the test time as well under the same principles.

We consider 3x local runtime to be acceptable, with a couple of caveats:

When using Connect, some additional length is expected (due to the time required to relay each request and response between our browsers and your machine running the .jar) If your tests are very short, the processing on each end also contributes time in a greater proportion relative to the test - since these times will not reduce below a few seconds.

That said, if your tests exceed the 3x metric, or if they suddenly start running much more slowly than previously, feel free to reach out to us at hel [email protected] and we'll do our best to assist you.

Page 58 Copyright Sauce Labs 2015

Handling Authentication

Dealing with authentication or security dialogs when testing a site or application can be tricky business. These topics present some options you can use to deal with various kinds of authentication challenges.

Basic HTTP Authentication Injecting Cookies to Bypass Authentication Dialogs Running an AutoIt Script as a Pre-run Executable to Handle Windows Security Authentication Dialogs

Page 59 Copyright Sauce Labs 2015

Basic HTTP Authentication

Basic HTTP authentication is when you supply a username and password in the URL, such as http://test:[email protected]/password-ok.php. This request sends the credentials in the standard HTTP "Authorization" header.

Browser support for basic authentication is limited.

Browser HTTP Authentication Support

Chrome Supported

Firefox Supported, but Firefox will display a prompt asking you to confirm

Safari Unsupported. See the support KB article How Can I use HTTP basic auth in Safari for an example of how you can use a pre-run executable to handle this situation.

Internet Unsupported - https://support.microsoft.com/en-us/kb/834489 Explorer

Because browser support for basic HTTP authentication is limited, we recommend either Injecting Cookies to Bypass Authentication Dialogs or R unning an AutoIt Script as a Pre-run Executable to Handle Windows Security Authentication Dialogs as better solutions for handling authentication.

Page 60 Copyright Sauce Labs 2015

Injecting Cookies to Bypass Authentication Dialogs Authentication dialogs present a challenge for automated testing with Selenium, because there's no way for Selenium to interact with them. A solution is to use Selenium to inject cookies that will let you bypass authentication by setting an authenticated state for the application or site you're testing. With this solution, you may need to make a change in the source code of the application so that the cookie is acknowledged, but your tests will be able to run without the need for user authentication credentials. The basic process for using cookie injection with testing on Sauce would be:

1. Launch your browser on Sauce Labs. 2. Inject cookies via Selenium. 3. Go to the site or application you want to test. 4. Run your tests.

You must be on the same domain that the cookie is valid for in order for this to work.

If the home page of the site you want to test takes a long time ago, you can try accessing a smaller page, like the 404 page, for the purpose of injecting the cookie before you access the home page.

This code example demonstrates how you would set the cookies, but you can find additional examples for Java, Python, Ruby, and Perl on the official Selenium Commands and Operations page.

# prereq: have sauce username and accesskey set as environment variables # i.e. export SAUCE_USERNAME=YOUR_USERNAME # export SAUCE_ACCESS_KEY=YOUR_ACCESS_KEY

require 'selenium-webdriver'

url = "http://#{ENV['SAUCE_USERNAME']}:#{ENV['SAUCE_ACCESS_KEY']}@ondemand.saucelabs.com:80/ wd/hub".strip

browser = Selenium::WebDriver.for(:remote, :url => url, :desired_capabilities => { :browserName => 'firefox', :version => '40', :platform => 'Windows 7' })

browser.manage.add_cookie({ :name => 'CookieName', :value => 'CookieValue', :path => '/', :secure => false

})

puts "Printing out cookies : #{browser.manage.all_cookies}"

# go to URL / app where authentication was needed / prompted. # i.e. # browser.get 'http://yourwebsite.com' # ... browser.quit

Page 61 Copyright Sauce Labs 2015

Running an AutoIt Script as a Pre-run Executable to Handle Windows Security Authentication Dialogs

When using Sauce Connect to run local tests on a Windows machine, you may encounter an Integrated Windows Authentication (IWA) dialog, also known as NTLM or Domain authentication. This is because the machine that Sauce Connect is running on is used to look up the host where the site or application under test is located. If the host has IWA enabled, the authentication request will be passed back through Sauce Connect to the Sauce Labs browser. Because there is no registry key available on our Windows virtual machines to disable this authentication, the solution is to create an AutoIT script to respond to the dialog, and run it as a pre-run executable in advance of running the test.

The AutoIT Script

You can use the AutoIT Script editor to write and save your script.

The script for handling the IWA dialog should look like this:

WinWaitActive(“Windows Security”) Send(“Username”) Send(“{TAB}”) Send(“mysupersecretpassword”) Send(“{ENTER}”)

For Username and mysupersecretpassword, enter the authentication credentials required by the Windows host.

When you save the script, it will be an .au3 file, and you will need to compile it as an .exe file. Once compiled, you can use it as a pre-run executable with this API call:

"prerun": { "executable": "http://url.to/your/executable.exe", "args": [ "--silent", "-a", "-q" ], "background": true }

64 v. 32-bit Version of AutoIT The 64bit version of AutoIT works on IE11, and not on IE9. The 32bit version works with both browser versions.

Page 62 Copyright Sauce Labs 2015

Pre-Run Executables

In the real world, there's no such thing as a "stock" version of any particular browser, because users can easily modify them with add-ons, change settings, and set profiles with specific functionality. There are also situations in which browser or virtual machine configurations can cause issues with your tests. For these reasons, you may want to make changes to the browser or VM configuration before your test runs. For example, you might want to test against a security-conscious user profile that includes using private windows, blocking all pop-ups, and disabling JavaScript, or you may want to change settings that could raise dialogs and cause your tests to fail.

There are different ways to deal with these situations, depending on the type of browser you're testing against. Firefox includes a Profiles feature t hat lets you manage everything from passwords to security settings to site-specific settings, like which sites are allowed to display pop-ups. The S elenium driver for Firefox includes a webdriver.firefox.profile system property that lets you select which profile you want to use in your testing. With Chrome, there is a long list of switches that you can set to change browser behavior, which you can pass as arguments to the Chrome WebDriver. Unfortunately, neither Safari or Internet Explorer have a built-in way to edit user settings, which is where pre-run executables come into play.

A pre-run executable is simply a script that you can run prior to a test to change settings for Safari, Internet Explorer, or any other browser, or to configure the virtual machine that your tests will run on. You can see an example of a pre-run executable for virtual machine configuration in the Sauce Labs Support KB article Edit the Domain Name System (DNS) within the Sauce Labs Virtual Machine, which edits the local hosts file on the Sauce Labs VM so that when the driver tries to access http://www.saucelabs.com, it will be re-directed to www.google.com. You can see an example of a pre-run executable for Safari in the Support KB article How Can I Use HTTP Basic Authentication in Safari? In this case, Safari will normally load a Warning page when you try to access a site that uses HTTP basic authentication, which will cause your test to fail. The solution is to use a pre-run executable to change the setting that warns about potentially fraudulent websites. An example of a pre-run executable for Internet Explorer would be changing the registry key MaxScriptStatements to a very high number to try and prevent the dialog Stop unresponsive script? from appearing.

You can configure your Sauce testing jobs to use pre-run executables using the DesiredCapabilities of the WebDriver API, as described in both Test Configuration Options and Setting Up Pre-Run Executables.

Setting Up Pre-Run Executables

Page 63 Copyright Sauce Labs 2015

Setting Up Pre-Run Executables

A pre-run executable is a script that you can use to change browser settings or virtual machine configuration prior to running your tests. This topic describes the basic steps involved in setting up pre-run executables using the DesiredCapabilities of the WebDriver API. You can find more information about the prerun setting for job configuration in the topic Test Configuration Options.

1. Write your script to reflect the changes you want to make to the browser or virtual machine configuration. This example script disables the Warning that Safari pops up when using basic HTTP authentication to access a website.

disable_fraud.sh

#!/bin/bash defaults write com.apple.Safari WarnAboutFraudulentWebsites false

2. Put your script in a location where Sauce can access it. For example, your script could be stored in GitHub or in Sauce Storage. 3. In your test script, set the prerun desired capability to point to the location of your pre-run executable. This example illustrates setting the prerun desired capability to point to myscriptstorage.com, where the disable_fraud.sh scrip t used as an example in step 1 is located.

desired_capabilities['prerun'] = { 'executable':'http://myscriptstorage.com/disable_fraud.sh', 'background': 'false' }

4. Run your test!

Page 64 Copyright Sauce Labs 2015

Test Configuration and Annotation

When running tests in the Sauce Labs cloud, you have a number of options for how you want to configure your tests, and for adding annotations to them such as a build name. These topics describe three methods for configuring and annotating tests - through the Selenium and Appium Cap abilities object, through the Sauce Labs REST API, and though the Selenium JavaScript executor - as well as the the most common options for test configuration and annotation in Selenium and Appium, along with specialized configuration parameters for running tests on Sauce.

Configuring Tests with the WebDriver API DesiredCapabilities Configuring Tests with the Sauce Labs REST API Configuring Tests with Selenium's JavaScript Executor Test Configuration Options Examples of Desired Capabilities for iWebDriver and Appium iOS Tests

Page 65 Copyright Sauce Labs 2015

Configuring Tests with the WebDriver API DesiredCapabilities

If you're using the WebDriver API for your Selenium and Appium tests, you can can use the DesiredCapabilities object to configure your test option. Any key-value pair described in the REST API topics can be set through this hash-like object, as well as the configuration and annotation options listed in Test Configuration Options. You can find more information about RemoteDriver and the DesiredCapabilities o bject on Selenium's Remote WebDriver wiki. You can also use the Platforms Configurator to set many of the DesiredCapabilities for your jobs in the syntax of your testing language.

Related Links

SauceREST Java Client on GitHub Python Example for Job Annotation via the REST API on GitHub Ruby Example for Job Annotation via the REST API on GitHub

Page 68 Copyright Sauce Labs 2015

Configuring Tests with Selenium's JavaScript Executor

You can use the Sauce REST API to set some test information after the test has run, but you can also set some information using Selenium’s JavaScript executor during the test.

Python Example Java Example Options

Python Example

driver.execute_script("sauce:job-name=My test")

Java Example

((JavascriptExecutor)driver).executeScript("sauce:job-name=My test");

Options

Option Example Description

Pass/Fail "sauce:job-result=passed" Sets the pass/fail status of the job. Valid options are passed, failed, true, and f Status alse. True means passed and false means failed.

Job Name "sauce:job-name=My awesome Sets the job name job"

Tags "sauce:job-tags=tag1,tag2,tag3" Sets the job tags in a comma-separated list.

Build "sauce:job-build=mybuild123" Sets the job’s build name. Name

Start/Stop "sauce: stop network" Stops/re-starts the VM’s network connection. A space must be included between sa the VM uce: and stop or start. Connection "sauce: start network"

Mac OS X only

Break "sauce: break" Puts a Sauce breakpoint in the test. Test execution will pause at this point, waiting Point for manual control by clicking in the test’s live video. A space must be included between sauce: and break. See Manual Testing for Mobile Apps for more information on how to use this annotation.

Context "sauce:context=This line Logs the given line in the job’s Selenium commands list. No spaces can be between appears in the command list as sauce: and context. 'info'"

Job Info "sauce: Sets one or more job information fields to the values sent in the JSON-formatted job-info={'build':'mybuild', dictionary. 'name':'my test name', 'public':'team'}"

Page 69 Copyright Sauce Labs 2015

Test Configuration Options

You can use the Platform Configurator to get the correct configuration of testing options for your choice of Appium or Selenium tests in your favorite scripting language. The examples in this topic are for Java. You can find out more about Selenium testing options in the DesiredCapabilities page of the SeleniumHQ wiki You can find out more about Appium testing options in the Appium Server Capabilities page of the Appium.io website.

Selenium-Specific Options Required Selenium Test Configuration Settings Other Selenium Options Selenium Version Appium-Specific Options Required Appium Test Configuration Settings Other Appium Options General Options Auto Accept Alerts Test Annotation Timeouts Sauce Testing Options Optional Sauce Testing Features

Selenium-Specific Options

Required Selenium Test Configuration Settings Browser Name Browser Version Platform

Setting Description Key Value Type Example

The name of the browser test against. Options are: browserName string "browserName": "firefox"

Browser Name android chrome firefox htmlunit internet explorer iPhone iPad opera safari

The version of the browser you want to use in your test. version string "version", "45.0" Browser Version

Which platform the browser should be running on. Options are: platform string "platform", "OS X 10.9"

Platform windows xp vista mac linux unix android

Other Selenium Options

Option Description Key Value Example Type

Allows you to choose the version of Selenium you want to use seleniumVersion string "seleniumVersion":"2.46.0" for your test. Selenium Version

Appium-Specific Options

Page 70 Copyright Sauce Labs 2015

Required Appium Test Configuration Settings Browser Name Device Name Platform Version Platform Name Application Path

Setting Description Key Value Example Notes Type

The mobile web browser that will be automated in browserName string "browserName", "Safari" If you're testing a mobile the simulator, emulator or device. native application or a mobile Browser hybrid application, the value Name for this capability should be an empty string. Check out T ypes of Mobile Tests for more information.

The name of the simulator, emulator, or device deviceName string "deviceName","Google Nexus 7 HD For an Android emulator test you want to use in the test. Emulator" you can request a generic Device Android emulator by using Name the option "deviceName":" Android Emulator". If you want to use an Android emulator that looks and feels like a specific Android phone or tablet, for example a Google Nexus 7 HD Emulator or a Samsung Galaxy S4, then instead of " deviceName":"Android Emulator", you need to specify the exact Android emulator skin to use, for example "deviceName":"S amsung Galaxy S4 Emulator".

Each Android emulator skin will have a different configuration depending on the phone or tablet that it emulates. For example, all the skins have different resolutions, screen dimensions, pixel densities, memory, etc. You can use the Platforms Configurator to get a list of the available Android emulator skins for the various Android emulator versions. Supported Android Emulators describes the qualities for each type of emulator that Sauce Labs supports.

The mobile operating system version that you platformVersion string "platformVersion","9.1" want to use in your test. Platform Version

The mobile operating system platform you want to platformName string "platformName", "iOS" use in your test. Platform Name

The path to a .ipa, .apk or .zip file containing app string "app","sauce-storage:my_app.zip" This capability is only for the app to test. This could be the location of your testing mobile native or Application app in the Temporary Sauce Storage, for example, mobile hybrid applications. T Path sauce-storage:myapp.zip, or the URL to a his capability is not required remote location where your app is located, for for Android if you specify the example http://myappurl.zip/. appPackage and appActiv For mobile and ity capabilities. Hybrid application testing only. See Types of Mobile Tests for more information

Other Appium Options Appium Version Device Orientation

Page 71 Copyright Sauce Labs 2015

Automation Engine Auto Accept Alerts Application Package Android Activity

Option Description Key Value Example Notes

The version of the Appium appiumVersion string It's better to driver you want to use. If not specify the latest Appium specified the test will run Appium version, Version against the default Appium which is the one version. suggested by the Platforms Configurator, unless you have a reason for testing against some other version.

The orientation in which the deviceOrientation string "deviceOrientation","portrait" simulator/device will be Device rendered. Options are: Orientation portrait landscape.

The automation engine that automationName string "automationName","Selendroid" will be used. Options are: Automation Engine Appium Selendroid.

The default is Appium.

You can set this option to autoAcceptAlerts boolean "autoAcceptAlerts": true automatically accept privacy Auto access alerts related to Accept accessing contacts, photos, Alerts or other privacy-protected iOS9 applications. For iOS9 Only

The Java package of the appPackage string "appPackage","com.example.android.myApp Appium Android app you want to run. , com.android.settings" automatically Application determines the Package package to For Android launch, you only Only need to use this desired capability if you want to specify a package different than the default one.

The name for the Android appActivity string "appActivity",".MainActivity" This capability activity you want to launch needs to be Android from your package. preceded by a . Activity (dot). For For Android example, .Main Only Activity instea d of MainActiv ity.

Appium automatically determines the activity to launch, you only need to use this desired capability if you want to specify an activity different than the default one.

General Options

Page 72 Copyright Sauce Labs 2015

These options can be set for both Selenium and Appium Tests.

Option Description Key Value Example Type

Setting this option will automatically accept any unexpected browser alerts that come autoAcceptAlerts boolean "autoAcceptAlerts": up during your test, such as when Safari pops up the alert "Safari would like to use your true Auto current location (Don't Allow | Allow)." Accept Alerts

Test Annotation

You can add these annotations to your tests to make them easier to track and identify. Test Names Build Numbers Tagging Pass/Fail Status Custom Data

Option Description Key Value Example Type

Used to record test names for jobs and make it easier to find name string "name": "my example name" individual tests Test Names

Used to associate jobs with a build number or app version, which is build string "build": "build-1234" then displayed on both the Dashboard and Archives view Build Numbers

User-defined tags for grouping and filtering jobs in the Dashboard tags list "tags": ["tag1","tag2","tag3"] and Archives view Tagging

Selenium and Appium handle sending commands to control a passed boolean "passed": "true" browser or app, but don't report to the server whether a test passed Pass/Fail or failed. To record pass/fail status on Sauce, set the passed flag Status on the job. Since you can't know in advance whether a test passed or failed, this flag can't be set in the initial configuration.

User-defined custom data that will accept any valid JSON object, customData object "customData": {"release": "1.0", limited to 64KB in size. "commit": Custom "0k392a9dkjr", Data "staging": true, "execution_number": 5, "server": "test.customer.com"}

Timeouts Maximum Test Duration Command Timeout Idle Test Timeout

Option Description Key Value Example Type

As a safety measure to prevent broken tests from running indefinitely, Sauce limits maxDuration integer "maxDuration":1800 the duration of tests to 30 minutes by default. You can adjust this limit on per-job Maximum basis. The value of this setting is given in seconds. The maximum test duration Test value allowed is 10800 seconds. Duration

Page 73 Copyright Sauce Labs 2015

As a safety measure to prevent Selenium crashes from making your tests run commandTimeout integer "commandTimeout":300 indefinitely, Sauce limits how long Selenium can take to run a command in our Command browsers. This is set to 300 seconds by default. The value of this setting is given Timeout in seconds. The maximum command timeout value allowed is 600 seconds.

As a safety measure to prevent tests from running too long after something has idleTimeout integer "idleTimeout":90 gone wrong, Sauce limits how long a browser can wait for a test to send a new Idle Test command. This is set to 90 seconds by default and limited to a maximum value of Timeout 1000 seconds. You can adjust this limit on a per-job basis. The value of this setting is given in seconds.

Sauce Testing Options Version (Browser) Pre-run Executables Identified Tunnels Specifying the Screen Resolution Custom Time Zones Chrome Driver Version Internet Explorer Driver Version Avoiding the Selenium Proxy Job Visibility

Option Description Key Value Example Notes Type

If this capability is null, an empty string, or omitted altogether, the latest version string or "version": "35" version of the browser will be used automatically. integer Version (Browser)

You can provide a URL to an executable file, which will be downloaded prerun "prerun": { "executable": "http: If a single string is and executed to configure the VM before the test starts. For faster //url.to/your/executable.exe", sent as the prerun Pre-run performance, you may want to upload the executable to temporary Sauce (primary key) "args": [ "--silent", "-a", "-q" capability rather than storage. This capability takes a JSON object with four main keys. Check ], "background": true, a JSON object, this Executables out the topics under Pre-Run Executables for more information. "timeout": 120 } string is considered to be the URL to the executable, and the Running AutoIt Scripts executable launches If you want to run an AutoIt script during your test, compile with background se it as an exe, send it using this capability, and set backgro t to false. und to true to allow AutoIt to continue running throughout the full duration of your test.

Using Multiple Pre-Run Executables If you need to send multiple pre-run executables, the best way is to bundle them into a single executable file, such as a self-extracting zip file.

Interacting with the Android Emulator If you need to use Android tools to tweak the emulator before your test starts (such as adb), the ANDROID_HOME environment variable will be set before your executable is invoked. For example, you can access adb via $ANDROID _HOME/platform-tools/adb.

The URL to the executable you want to run before your browser session executable starts. (secondary key)

A list of the command line parameters that you want the executable to args receive. (secondary key)

A boolean that defines whether Sauce should wait for this executable to background finish before your browser session starts. If background isn't set or is set to false, Sauce will wait for up to 90 seconds for the executable to (secondary key) finish. At that point, the browser will start and your test will proceed.

The number of seconds Sauce will wait for your executable to finish timeout before your browser session starts. If timeout isn't set, Sauce will wait for up to 90 seconds for the executable to finish. timeout is capped at 360 (secondary key) seconds and won't apply if background is set to true.

If an identified tunnel is started using Sauce Connect, your jobs can tunnelIdentifier string "tunnelIdentifier": "MyTunnel01" choose to proxy through it using this set of keys with the right identifier. Identified Tunnels

Page 74 Copyright Sauce Labs 2015

This setting specifies which screen resolution should be used during the screenResolution string "screenResolution": "1280x1024" Valid values for test session. This feature is available in Windows XP, Windows 7 (except Windows XP and Specifying Windows 7 with IE 9), Windows 8, Windows 8.1, and OS X 10.8. We do Windows 7 not yet offer specific resolutions for Windows 10, OS X 10.9, OS X 10.10, the Screen OS X 10.11, Linux, or mobile platforms. 800x600 Resolution 1024x768 1052x864 1152x864 1280x800 1280x960 1280x1024 1400x1050 1440x900 1600x1200 1680x1050 1920x1200 2560x1600

Valid values for OS X 10.8

1024x768 1152x864 1152x900 1280x800 1280x1024 1376x1032 1400x1050 1600x1200 1680x1050 1920x1200

Valid values for Windows 8 and 8.1

800x600 1024x768 1280x1024

Desktop Test VMs can be configured with custom time zones. This timeZone string "timeZone": "Pacific" feature should work on all operating systems, however time zones on "timeZone": "Honolulu" Custom Windows VMs are approximate. They will default to the time zone that the "timeZone": "Alaska" provided location falls into. You can find a complete list of timezones on "timeZone": "New_York" Time Zones Wikipedia. Underscores should be replaced with spaces. Sauce takes only location names (not their paths), as shown in the example below.

Sauce Labs supports the ChromeDriver version 1 series (i.e. 26.0.1383 chromedriverVersion string "chromedriverVersion": "2.15" Supported Chrome .0) and the version 2 series (i.e. 2.15). The default version of Drivers Chrome ChromeDriver when no value is specified depends on the version of Chrome. 21.0.1180.0 Driver 23.0.1240.0 Version Chrome 28 and below: Chromedriver 26.0.1383.0 26.0.1383.0 Chrome 29 - 30: Chromedriver 2.4 0.6 Chrome 31 - 32: Chromedriver 2.8 0.7 Chrome 33 - 36: Chromedriver 2.10 0.8 Chrome 37 - 39: Chromedriver 2.11 0.9 Chrome 40 - 44: Chromedriver 2.15 2.0 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15

Page 75 Copyright Sauce Labs 2015

Sauce Labs supports launching 64-bit IE on our 64-bit VMs: Windows 7, iedriverVersion string "iedriverVersion": "2.46.0" Supported IE Windows 8, and Windows 8.1. This provides a workaround for two known Drivers Internet Selenium issues: 2.21.1 Explorer Using a 32 bit driver on a 64 bit operating system causes Selenium's 2.21.2 Driver screenshot feature to only capture the part of the page currently visible in 2.24.0 the browser viewport Selenium Issue 5876. 2.25.3 Version 2.26.0 Using a 64 bit driver on a 64 bit operating system causes text entry to be 2.28.0 extremely slow Selenium Issue 5516. 2.29.0 2.30.1 2.31.0 2.32.2 2.33.0 2.34.0 2.35.0 2.35.1 2.35.2 2.35.3 2.36.0 2.37.0 2.38.0 2.39.0 2.40.0 2.41.0 2.42.0 2.43.0 2.44.0 2.45.0 2.46.0 x64_2.29.0 x64_2.39.0 x64_2.40.0 x64_2.41.0 x64_2.42.0 x64_2.43.0 x64_2.44.0 x64_2.45.0 x64_2.46.0

By default, Sauce routes traffic from some WebDriver browsers (Internet avoidProxy boolean "avoidProxy": true Explorer and Safari) through the Selenium HTTP proxy server so that Avoiding HTTPS connections with self-signed certificates work everywhere. The Selenium proxy server can cause problems for some users. If that's the the case for you, you can configure Sauce to avoid using the proxy server Selenium and have browsers communicate directly with your servers. Proxy Don't Need the Selenium Proxy with Firefox or Google Chrome Firefox and Google Chrome under WebDriver aren't affected by this flag as they handle invalid certificates automatically and there isn't a need to proxy through Selenium.

Incompatible with Sauce Connect This flag is incompatible with Sauce Connect.

Page 76 Copyright Sauce Labs 2015

Sauce Labs supports several test result visibility levels, which control who can view the test details. The visibility level for a test can be set manually Job from the test results page, but also programatically when starting a test or with our REST API. For more information about sharing test result, see Visibility the topics under Sharing the Results of Sauce Labs Tests.

Available visibility levels are:

Visibility Description Key

public Making your test public means that it is accessible to everyone, and may be listed on public web pages and indexed by search engines.

public If you want to share your job's result page and video, restricted but keep the logs only for you, you can certainly do so with public restricted visiblity mode. This visibility mode will hide the fancy job log as well as prohibit access to the raw Selenium log, so that anonymous users with the link will be able to watch the video and screen shots but won't be able to see what's being typed and done to get there.

share You can also decide to make your test sharable. Making your test sharable means that it is only accessible to people having valid link and it is not listed on publicly available pages on saucelabs.com or indexed by search engines.

team If you want to share your jobs with other team members (that were created as a sub-accounts of one parent account), you can use team visiblity mode. Making your test acessible by team means that it is only accessible to people under the same root account as you.

private If you don't want to share your test's result page and video with anyone, you should use private job visibility mode. This way, only you (the owner) will be able to view assets and test result page.

Optional Sauce Testing Features

By default, Sauce Labs captures screenshot and video of your tests. You can disable these and other optional test features. Disable video recording Disable step-by-step screenshots Disable log recording Enable HTML source capture Enable WebDriver's automatic screen shots

Option Description Key Value Example

By default, Sauce records a video of recordVideo boolean "recordVideo": false every test you run. This is generally Disable handy for debugging failing tests, as video well as having a visual confirmation recording that certain feature works (or still works!) However, there is an added wait time for screen recording during a test run.

This setting will let you discard videos recordScreenshots boolean "recordScreenshots": false for passing tests identified using the pa Disable ssed setting. This disables video step-by-step post-processing and uploading that screenshots may otherwise consume some extra time after your test is complete.

By default, Sauce creates a log of all recordLogs boolean "recordLogs": false the actions that you execute to create Disable log a report for the test run that lets you recording troubleshoot test failures more easily.

In the same way Sauce captures captureHtml boolean "captureHtml": true step-by-step screenshots, we can Enable capture the HTML source at each step HTML of a test. This feature is disabled by source default, but you can turn it on any time capture and find the HTML source captures on your job result page.

Page 77 Copyright Sauce Labs 2015

Selenium WebDriver captures webdriverRemoteQuietExceptions boolean "webdriverRemoteQuietExceptions": automatic screenshots for every server false Enable side failure, for example if an element WebDriver's is not found. Sauce disables this by automatic default to reduce network traffic during screen tests, resulting in a considerable performance improvement in most shots tests. You can enable this feature, but keep in mind that it may be detrimental to the performance of your jobs.

Page 78 Copyright Sauce Labs 2015

Examples of Desired Capabilities for iWebDriver and Appium iOS Tests

This topic describes configuration of desired capabilities for iOS iWebDriver and Appium tests on Sauce Labs. Check out Appium Capabilities for Mobile Tests on Sauce for a general description of mobile test desired capabilities, and Types of Mobile Tests for explanations of the difference between Mobile Web and Native mobile applications and how to test them with Sauce.

Using iWebdriver Mobile Web Application iPhone iPad Using Appium Mobile Web Application iPhone iPad Native Mobile Applications iPhone iPad Learn more about Appium and its desired capabilities: http://appium.io/slate/en/v1.2.0/?python#toc_56 Manual Test + Native Mobile Applications:

Using iWebdriver

You can use iWebdriver to drive the iOS simulator when testing Mobile Web Applications with Sauce Labs. If you want to test Native Mobile Application then you will need to use the Appium driver.

Mobile Web Application iPhone self.desired_capabilities = webdriver.DesiredCapabilities.IPHONE self.desired_capabilities['platform'] = "OS X 10.10" self.desired_capabilities['version'] = "6.0" self.desired_capabilities['browserName'] = 'iPhone' iPad self.desired_capabilities = webdriver.DesiredCapabilities.IPAD self.desired_capabilities['platform'] = "OS X 10.10" self.desired_capabilities['version'] = "6.0" self.desired_capabilities['browserName'] = 'iPad'

Using Appium

Appium is not supported for iOS version 6.0 and earlier. For these earlier versions of iOS you must use iWebdriver. For iOS version 6.1 and later you can use Appium. Appium will be the tool in the background that is starting and driving the simulator for your Sauce Labs test.

If in your test you omit the appiumVersion desired capability, your test will be running with our default Appium version (for example, Appium 0.18.2.). Sauce recommends that you specify one of the newer Appium versions (for example, Appium version 1.3.6 or 1.3.7) which provides a more extended API and fixes to known bugs.

Checking the Appium Version for Your Test 1. Log in to saucelabs.com. 2. Find and select the test that you ran using Appium to view the Test Details page. 3. Click the Metadata tab. 4. Look for the Logs row and select Appium Log. The first line should indicate the Appium version. For example, 2014-05-05T17:45:07.541Z - info: Welcome to Appium v1.3.6.

Mobile Web Application iPhone

Page 79 Copyright Sauce Labs 2015

self.desired_capabilities = {} self.desired_capabilities['platformName'] = 'iOS' self.desired_capabilities['platformVersion'] = '8.1' self.desired_capabilities['browserName'] = 'safari' self.desired_capabilities['deviceName'] = 'iPhone Simulator' self.desired_capabilities['appiumVersion'] = '1.3.6' iPad self.desired_capabilities = {} self.desired_capabilities['platformName'] = 'iOS' self.desired_capabilities['platformVersion'] = '8.1' self.desired_capabilities['browserName'] = 'safari' self.desired_capabilities['deviceName'] = 'iPad Simulator' self.desired_capabilities['appiumVersion'] = '1.3.6'

Native Mobile Applications iPhone self.desired_capabilities = {} self.desired_capabilities['platformName'] = 'iOS' self.desired_capabilities['platformVersion'] = ‘8.1' self.desired_capabilities['browserName'] = '' self.desired_capabilities['deviceName'] = 'iPhone Simulator' self.desired_capabilities['app'] = 'sauce-storage:myapp.zip' self.desired_capabilities['appiumVersion'] = '1.3.6' iPad self.desired_capabilities = {} self.desired_capabilities['platformName'] = 'iOS' self.desired_capabilities['platformVersion'] = ‘8.1' self.desired_capabilities['browserName'] = '' self.desired_capabilities['deviceName'] = 'iPhone Simulator' self.desired_capabilities['app'] = 'sauce-storage:myapp.zip' self.desired_capabilities['appiumVersion'] = '1.3.6'

Note that when testing a Native Mobile Application the value for the 'browserName' desired capability is simply an empty string (e.g ''). Also have in mind that you must add the desired capability 'appiumVersion'with a value of Appium version 1.0 or greater. If you need to use a remote app instead of an app that you previously loaded into sauce-storage then you can specify it like this: self.desired_capabilities['app'] = 'http://myappurl.zip'

--> Make sure that you app has internet permissions

Learn more about Appium and its desired capabilities: http://appium.io/slate/en/v1.2.0/?python#toc_56

Manual Test + Native Mobile Applications:

Currently, the only way that users can interact with Native Mobile Applications is through an automated tests. After the app is loaded through an automated test, you could take manual control of your Native Mobile Application by inserting a breakpoint. But overall the loading of the app has to happen through an automated test.

Here is some information about breakpoints: http://sauceio.com/index.php/2012/09/using-sauce-breakpoints-to-find-and-fix-flakey-tests/

Page 80 Copyright Sauce Labs 2015

Manual Testing with Sauce Labs

Automated testing is the fastest, most efficient way to undertake complex testing suites, but sometimes the devil is in the details, and you need to run a manual test. Manual testing, for example, is a great way for designers to debug CSS issues, or for documentation specialists to understand how a feature works. With Sauce, you can run manual tests that share great features with automated testing like recordings of your tests, or being able to use Sauce Connect to access sites and applications that are behind a firewall or on localhost, but have unique features of their own, like being able to invite other Sauce Labs users to observe your tests.

Running a Manual Testing Session Starting Manual Tests from Automated Tests Troubleshooting Manual Tests

Page 81 Copyright Sauce Labs 2015

Running a Manual Testing Session

1. Log in to Sauce Labs and select Manual Tests. 2. Click New Session. 3. Enter the URL of the application or website you want to test. If you use Sauce Connect to access your application, select the tunnel to use. 4. Select the browser or mobile operating system you want to test against. Click the History icon

to select browser/version/operating system/screen resolution combinations you've used in previous manual tests. 5. For Web browsers, select the browser version, operating system, and screen resolution that you want to test against. 6. Test assets such as screenshots are automatically saved. If you don't want to save them, clear the Save Screenshots, Logs, & Video o ption. 7. Click Start Session. You'll see a loading screen, and then the URL you entered will launch in a manual test window. At the top of the screen you will see a tab with the test URL, and a menu bar that contains information about the parameters of your test and the time remaining before your test session ends. You will also see icons to Stop the test, Share the session with other users, use the Clipboard, take Snapshots, and enter Full-screen mode.

Time Limit on Manual Tests All manual test sessions are limited to three hours.

8. Use your keyboard and mouse to test the functionality of your website or application.

Follow the Black Dot During your testing session your cursor will appear as a black dot. You can use it to navigate the screen and interact with interface elements in the same way as you would with a typical arrow cursor.

9. If you find a bug, click the Snapshot icon

to record it. This will save it to the Test Details page. 10. When you're finished with your test session, click Stop

. You can now download video of your test, and other test assets, on the Test Details page under the Metadata tab.

Running Tests in Parallel You can run multiple manual test sessions at the same time, with the number of tests limited by the concurrency allowance associated with your account. If you want to start additional sessions, click the + icon next to the tab containing the URL of your current test session. Follow the steps to set up the session, and then you can switch back and forth between the sessions by clicking on the URL tabs.

Inviting Others to Observe Your Test You can invite other users with Sauce Labs accounts to observe your test session by clicking the Share icon

. This will display a URL for the test that you can then send to other users.

Using the Clipboard You can use the remote clipboard to store and then copy text that you want to use in your tests. Click the Clipboard icon

, enter the text you want to store on the remote clipboard, and click Send. You'll see the text appear under the Remote Clipboard head er. To copy text from the remote clipboard back to your local clipboard, click the Copy icon

Page 82 Copyright Sauce Labs 2015

Starting Manual Tests from Automated Tests

There may be situations where you want to run a manual version of an automated test: for example, if your automated test found a bug that you want to investigate in more detail.

1. Navigate to the Test Details page for the automated test. 2. Click the Commands tab. 3. Below the screenshot in the right-hand pane, click Open Manual Session. This will launch a manual testing session. 4. Follow the steps for running a manual testing session.

Page 83 Copyright Sauce Labs 2015

Troubleshooting Manual Tests

Seeing a Security Error Message (Error #2048) Seeing a Black Screen / "Plugin Failure" Error Long Load Times or Timing Out All Links Open in New Tabs Job Does Not Load Want more?

Seeing a Security Error Message (Error #2048)

This error is displayed when the ports used by manual testing relies are being blocked by a firewall on your end. This may also be caused by running applications such as Avast! antivirus software.

These are the servers and ports used by manual testing. Please check with your IT organization that they are accessible.

If you are launching manual testing from Internet Explorer on your local machine: tv1.saucelabs.com:843 tv1.saucelabs.com:5901 saucelabs.com:843

If you are launching manual testing from any other locally-installed browser: charon.saucelabs.com:80

We recommend making all of these accessible if you plan on using several browsers locally.

Seeing a Black Screen / "Plugin Failure" Error

If your test shows a black screen after starting the virtual machine, you may need to reinstall Adobe Flash Player on your machine. This should only occur if you are using Internet Explorer to launch Sauce Manual Testing, which requires this software for our in-browser VNC player to function.

Long Load Times or Timing Out

We've streamlined our service to provide the best possible load times. If you are seeing slow manual testing sessions, check our status page, or get instant updates by following @sauceops on Twitter.

All Links Open in New Tabs

It's possible for the manual testing VNC client to have a modifier key "stuck" down, causing any clicked links to open in new tabs. This happens if the client loses focus while a key is held down -- for example, when using Alt-Tab to switch application windows. In this case, VNC never receives the keyUp event.

To prevent this from happening: every time you focus back on the manual testing window, first click in the middle of the page, then press and release all the modifier keys (like Alt, Control, Command, and Shift).

Job Does Not Load There are two common scenarios here:

Error message: "Uh oh! Some error occurred while connecting to the browser" The job seems to start, but you see only a white text box in the middle of a black screen.

Page 84 Copyright Sauce Labs 2015

Both of these indicate that your browser is having trouble displaying the VNC stream from the remote machine. Take the following steps to troubleshoot:

Check the video on Sauce: If the recorded video after the job shows a steady video stream, this indicates that the issue is in your computer or connection to Sauce. However, if the Sauce video shows the same issue, that indicates an issue in our service. In that case, send us the URL for the job page and a screenshot of the issue. Check that your browser is up to date: If you're on an older version, this may cause incompatibilities. Update your browser and try again. Check your firewall: make sure that your machine allows full access for the interactive stream over the required ports (https://saucelabs.c om/docs/manual#troubleshooting). Check that the Internet connection is stable: We recommend running Sauce tests from a machine with a wired Ethernet connection, to ensure a steady connection. If the connection flickers, this error could be thrown.

Want more?

If you'd like to dig deeper, feel free to check out our help forums or email us directly at [email protected].

Page 85 Copyright Sauce Labs 2015

Automated Testing with Sauce Labs

Prerequisites

Before you get started with testing on Sauce, you should check out the Automated Testing with Sauce Labs and the Automated Testing with Sauce Labs If you're interested in testing mobile applications, you should read the Automated Testing with Sauce Labs If you want to test applications that are behind a firewall or on localhost, you should read up on Sauce Connect, which enables you to create a secure tunnel between the location where your application resides and the Sauce Labs testing infrastructure If you aren't already, you should familiarize yourself with using Selenium for Web application testing, and Appium for mobile application testing

Running Automated Tests for Web and Mobile Applications

Automated testing of both Web and mobile applications on Sauce Labs boils down to a few basic steps:

1. Set yourself up with a Sauce Labs account. Check out the topics under Automated Testing with Sauce Labs if you're interested in setting up an account for multiple team members. 2. Update your existing tests to run on Sauce. The topics under Automated Testing with Sauce Labs include examples of how to do this. 3. Write new tests that include the desired capabilities of your test, in the language of your choice.

You can use the Platform Configurator to set the desired capabilities for both Selenium and Appium tests in your preferred language. You can also use the REST API or the WebDriver API to further configure your jobs with timeouts, annotations (a recommended best practice for managing your tests results, and for managing builds in your continuous integration pipeline), and other job functionality. Both the Automated Testing with Sauce Labs and Tutorial topics include code samples that illustrate how to write automated tests for Web applications in various languages, while the topics under Automated Testing with Sauce Labs illustrate how to write scripts for testing iOS and Android mobile applications.

1. Run your tests and watch the results appear on your Sauce Labs dashboard, complete with video, screenshots, and logs.

Once you've mastered the basics, you'll be ready to move on to more advanced functionality, like running your tests in parallel to speed up your testing process, and integrating your tests on Sauce into your continuous integration pipeline.

Page 86 Copyright Sauce Labs 2015

Troubleshooting Automated Tests

My tests can't connect to Sauce. What should I do? The video is missing, even though the test finished. What happened? Open Commands time out, even though I see the app loaded in the video. Why is this? My tests are taking too long to start. What should I do? Tests that failed on my end appear to have passed in Sauce. How did that happen? Your service is down. What should I do?

My tests can't connect to Sauce. What should I do?

A general problem some users face is when outgoing connections from private networks on port 4444 are blocked, causing the tests to not reach Sauce. As a fast solution to that problem, Sauce also listens at ondemand.saucelabs.com:80. Just use ondemand.saucelabs.com as the Selenium host and 80 as the Selenium port.

The video is missing, even though the test finished. What happened?

Missing videos are generally caused by the need to first post-process and upload the recorded video for it to be accessible to our users through the web interface. The time these tasks take will depend on the duration of the test (longer tests will produce longer videos). A reasonable estimation is that video processing and uploading consumes around 30% of total test time. This means that if your test takes 1 minute total (between browser start and shutdown), the video will take 20 seconds to upload to your account page. Additionally, if a test takes 10 minutes, it may take us up to 3 minutes to have the video ready.

Please let us know if you find videos are taking longer than that to process.

Open Commands time out, even though I see the app loaded in the video. Why is this?

This is generally caused by a connection gap or a problem with the application's server handling requests incorrectly. As a first step, you should proceed with a deep analysis of the network traffic. If you make it automated and run several tests at the same time, you will have higher chances of replicating the error.

Another good recommendation is to try out the captureNetworkTraffic command, which requires the Selenium instance to be started with the option captureNetworkTraffic=true and your test to use Firefox. This will let you pull the request info back out as JSON/XML/plain text. Then you can parse that content and find any problems.

My tests are taking too long to start. What should I do?

We're constantly working to making our resource allocation as slick as possible, but at certain times, when our service is under very high load, this could take longer than expected. Please check our status page to see if there's an ongoing issue and let us know if you find this happening too often.

Tests that failed on my end appear to have passed in Sauce. How did that happen?

Because of the client/server architecture that Selenium employs, there's no information about assertion results on the server side (which, in this case, is Sauce). Here's an example. If your test has a step for validating that the title of your AUT is "My Shiny WebApp's Title", all that Sauce sees is a request to get the title from the current page. Therefor, it will only return the result, without even knowing what was expected.

Your test: assertEquals(sel.getTitle(), "My Shiny WebApp's Title");

Sauce Labs: Command requested: getTitle() Result: Your Page's Title

Notice that we use the same criteria for other kinds of failures, such as a Selenium exception for trying to click on a non-existent element. The reality is that tests on your end could be coded in such a way that failures won't always end up as a failed job.

The good news is that you can let Sauce know what actually happened with your tests. Check out our Pass/Fail API to do it from within your Selenium tests.

Your service is down. What should I do?

We're constantly checking our service so we're likely already aware of an outage. Please check our status page and let us know if you don't see

Page 87 Copyright Sauce Labs 2015

anything reported there.

Page 88 Copyright Sauce Labs 2015

Common Error Messages

Sauce Labs Authentication Error User Terminated Timeout errors Test did not see a new command for 90 seconds. Timing out Selenium didn't complete your last command on time Test exceeded maximum duration after 1800 seconds ERROR user closed connection while waiting for command to complete UnreachableBrowserException: Error communicating with the remote browser. It may have died The connection with your vm was lost and your job can't complete. You won't be charged for these minutes The VM's disk has filled up. Selenium likely crashed as a result. Unsupported OS/browser/version combo The Sauce VMs failed to start the browser or device Client disconnected during getNewBrowserSession request Runaway execution. Please contact [email protected] for assistance UnreachableBrowserException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure Client disconnected during getNewBrowserSession request

Sauce Labs Authentication Error

A variation of the following error message will be shown by your test: Sauce Labs Authentication Error. You used username 'None' and access key 'None' to authenticate, which are not valid Sauce Labs credentials.

The following desired capabilities were received: ...

This indicates that your Sauce Labs username and access key aren't provided by your tests as expected. To address this, please follow the right tutorial for your programming language.

User Terminated

This message is used for tests manually interrupted using the Cancel or Breakpoint buttons in our website. Since both of these take control of the virtual machine immediately, test assets like screenshots, video, or logs that require additional execution time will not be collected and made available afterwards.

Timeout errors

Sauce Labs implements several types of timeouts for automated tests as safety nets to prevent unwanted consumption of minutes by runaway or broken tests.

You can set these to different values using the DesiredCapabilities in the setup of your test, as described in the Timeouts section of the Test Configuration and Annotation topic..

To better understand why these timeouts exist, let's look at the lifecycle of a Selenium command through the Sauce Labs service:

Here you can see the different types of interactions that happen in real time between the test script running in your own infrastructure, Sauce Labs' service and the Selenium or Appium server running inside our virtual machines.

Here are some common timeout errors you may find, what causes them and how to address them:

Test did not see a new command for 90 seconds. Timing out

This error is shown when Sauce Labs doesn't receive a new command from your Selenium script in more than 90 seconds (the timeout's default length.)

Using the previously shown graph as a reference, this would happen if step number 5 never happened. Without such a timeout, any tests that lack a proper session ending request (generally rendered as a call to driver.quit() or browser.stop()) will keep running forever, consuming all test minutes available in your account.

The most common cause for this is customers' test scripts crashing, getting forcefully interrupted or losing connectivity.

A less common but still possible cause is tests legitimately needing more than 90 seconds to send a new command to the browser. This happens most often when network or disk IO occurs in between selenium API calls in your tests (DB queries, local file reads or changes.)

Page 89 Copyright Sauce Labs 2015

If that's the case, our idleTimeout desired capability can be used to modify Sauce's wait time for further commands. Check our idleTimeout docs for more details on this.

Selenium didn't complete your last command on time

This error is shown when Sauce Labs doesn't receive a response from Selenium to your Script's last command in more than 5 minutes (the timeout's default length.)

Using the previously shown graph as a reference, this would happen if step number 3 never happened. Without such a timeout, any tests in which the Selenium, Appium server or browser crashes would keep running forever, consuming all test minutes available in your account.

The most common causes for this are unresponsive javascript in your application or a bug in Selenium/Appium.

A less common but still possible cause is Selenium or Appium legitimately needing more than 5 minutes to run your command. If that's the case, our commandTimeout desired capability can be used to have Sauce wait longer for your commands to complete in Selenium. Check our comma ndTimeout docs for more details on this.

Test exceeded maximum duration after 1800 seconds

This error is shown when Sauce Labs finds an automated test still running after 30 minutes (the timeout's default length.)

The most common cause for this is an infinite loop in your tests that keep sending commands without an end clause.

It's not rare to find cases where tests legitimately need more than 30 minutes to complete. If that's the case, our maxDuration desired capability can be used to have Sauce wait longer for your test to finish. Check ourmaxDuration docs for more details on this.

ERROR user closed connection while waiting for command to complete

Similar to our commandTimeout, this means that Selenium took a long time to run the command -- usually the act of loading a page -- and your test runner shut down the test before it might have timed out on Sauce's end. We recommend checking into why the page would take a long time to load (perhaps because of perpetually-loading items from another domain), and if need be, extending the related timeout setting in your test runner.

UnreachableBrowserException: Error communicating with the remote browser. It may have died

This error is thrown in Java when your test has timed out -- for example, due to the automatic 90-second idle timeout, which most often is caused by a dropped Internet connection. Then, the test script attempts to send another command to Sauce, but the job has already finished, so the VM is no longer available. Check your Internet connection, and try running your tests from a machine with a wired Ethernet connection.

The connection with your vm was lost and your job can't complete. You won't be charged for these minutes

This message occurs when our infrastructure loses communication with our vm and can't regain that connection after a reasonable time. If you only get this message rarely and randomly, it is probably a fluke on our end caused by an infrastructure blip.

However, if you are experiencing this error repeatedly for a specific test or set of tests there may be an issue on your end that is causing the failure. For example, if the error regularly appears after a specific selenium command there could be something wrong with the test that is causing selenium to crash. We have also seen issues with prerun executables: your script could be crashing the vm in any number of exciting ways, but the most common we've seen is consuming too much memory. We even had a situation once where a customer's script killed processes in a windows session, including the process we use to run jobs!

The VM's disk has filled up. Selenium likely crashed as a result.

Our VMs have virtual disks which, just like hardware disks, can fill up. We make sure that our virtual machines have at least 3G free when we start a job, but sometimes complex/long-running tests fill up the guest machine's allocated space. This causes selenium to crash, which ends your test.

The best thing you can do to avoid this failure is to break out your long tests into shorter tests and/or make sure that your tests are not filling up a lot of disk space on the VM.

Unsupported OS/browser/version combo

Page 90 Copyright Sauce Labs 2015

Check to make sure that the browser, version, and platform settings you're using are in our supported list of platforms.

Note that there's a separate list for Appium tests.

Relatedly, you may see: The requested combination of browser, version and OS is unsupported by the Selenium version requested and w ould lead to a test failure. Please set a different Selenium version, or just don't set it at all to default to a working version...

This error usually means that you've manually set a Selenium version that is too old to handle a more recent browser version. It should be resolved by choosing a newer version of Selenium, or omitting this setting altogether.

The Sauce VMs failed to start the browser or device

The twin sibling of "Unsupported version", this message means that something a little more unusual is off in your test setup. Usually, this means that you're specifying a Selenium version that isn't compatible with the browser/version/OS you've selected. (For example, you should not be setting this for any mobile tests.)

Try simply omitting that setting, and if you still see the issue, feel free to contact our support team with a description of the issue and a copy of your setup code.

Client disconnected during getNewBrowserSession request

This means that your test runner decided to end the job before it had fully initialized on Sauce's end. There are a few potential causes:

You're running too many tests at a time: Check the left sidebar on your Account page (https://saucelabs.com/account). It shows a "Parallel tests" number, which is the maximum number of tests you can run at a time, based on your subscription level. If your account can run 2 parallel tests, and you're launching 10, 8 will be "queued" until one of your tests finishes and a slot frees up. However, if this takes a long time, your test runner may choose to end the queued jobs after a few minutes instead of waiting. Just make sure you're launching a reasonable number of simultaneous tests for your account. High job wait times: Check our status page (http://status.saucelabs.com/) and/or follow @sauceops on Twitter for up-to-the-minute news about any issues within the service. If something causes demand for certain VMs to stack up, your jobs may be queued and (as above) terminated by your test runner.

Tests that end this way are never taken out of your minutes.

Runaway execution. Please contact [email protected] for assistance

This message means that an error has been detected in the VM or in Selenium, which caused the test to behave abnormally, and Sauce detected this and shut down the job. These are very rare and usually do not recur. If you do see more than one on the same test, let us know.

UnreachableBrowserException: Could not start a new session. Possible causes are invalid address of the remote server or browser start-up failure

This error is thrown on the client side, when Selenium isn't able to reach our testing servers to start a session. Try running this on the command line: telnet ondemand.saucelabs.com 80

If your machine is able to hit our servers, you should see: Trying 65.50.202.179... Connected to ondemand.saucelabs.com. Escape character is '^]'.

If you do receive this successful message, and still can't start a session, send that output to our support team and we'll troubleshoot further.

Otherwise, there's definitely something blocking the connection on your end. If so, you'll need to talk with your IT/network team to ensure that the required ports can be made available.

Client disconnected during getNewBrowserSession request

Page 91 Copyright Sauce Labs 2015

Mobile Testing with Sauce Labs

Need to test a mobile native, web, or hybrid application? Want to test on emulators, simulators, and real devices? Leave it to Sauce! Sauce fully supports testing of mobile apps and sites with Appium, Topics in this section will provide you with an overview of mobile testing with Appium on Sauce, support and requirements for mobile testing, how to set up your mobile tests, and examples of mobile testing scripts in a variety of flavors.

Mobile Testing with Appium Support and Requirements for Mobile Testing Supported Android Emulators Supported Mobile Operating Systems Requirements for Testing Mobile Native and Hybrid Applications Manual Testing for Mobile Apps Getting to the JavaScript Console for Manual iOS Browser Tests Running Emulator and Simulator Mobile Tests Types of Mobile Tests FAQs for Mobile Testing Example Appium Mobile Testing Scripts Android Example Scripts Written for Mobile Testing on Sauce Labs Example Java Script for Testing Android Mobile Applications on Sauce Labs Example Node.js Script for Testing Android Mobile Applications on Sauce Labs Example Python Script for Testing Android Mobile Applications on Sauce Example Ruby Script for Testing Android Mobile Applications on Sauce Labs iOS Example Scripts for Mobile Testing on Sauce Example Java Script for Testing iOS Mobile Applications on Sauce Labs Example Node.js Script for Testing iOS Mobile Applications on Sauce Example PHP Script for Testing iOS Mobile Applications on Sauce Labs Example Python Script for Testing iOS Mobile Applications on Sauce Labs Example Ruby Script for Testing iOS Mobile Applications on Sauce Labs

Page 92 Copyright Sauce Labs 2015

Mobile Testing with Appium

Appium is an open-source tool that can be used to automate your mobile applications. Just like the Selenium WebDriver - which is an open-source tool used to automate web browsers - Appium is an automation library used to drive your mobile applications, and even the web browser within the mobile simulator, emulator or real device.

Advantages of Using Appium

You can write tests against multiple mobile platforms using the same API You can write and run your tests using any language or test framework It's an open-source tool that you can easily contribute to

Advantages of Using Appium on Sauce Labs

You save the time it takes to set up the Appium server locally You don't have to install/configure the mobile emulators/simulators in your local environment You don't have to make any modifications to the source code of your application You can start scaling your tests instantly

For an in-depth explanation of what Appium is, how it works, and the various technologies behind it, check out the official Appium Documentation.

How Appium Works on Sauce Labs

The driver address in you test script is what points the test to the Sauce Labs cloud, for example http://SAUCE_USERNAME:SAUCE_ACCESS_K [email protected]:80/wd/hub. Once you execute the test using the test framework of your choice, Sauce Labs initializes its own Appium server using the specifications taken from your desired capabilities. What happens after that depends on the type of mobile platform you're testing against. iOS Tests

After the mobile application is initialized, the Appium server takes the driver commands in your test script, which are in a WebDriver JSON Wire Protocol format, and converts them into UIAutomation JavaScript commands that can be understood by Apple Instruments. These commands, once converted, are sent to your iOS mobile application via Apple Instruments, which executes the commands against your mobile application in the simulator/device.

The responses from your mobile application are received by Apple Instruments, sent to UIAutomation, and relayed to the Appium server in the Sauce Labs cloud. The Appium server then converts the UIAutomation JavaScript responses back into WebDriver JSON Wire Protocol format, and sends the JSON responses to your test script.

Android Tests

After the mobile application is initialized, the Appium server receives your test script commands and launches your mobile application in the emulator/device that you've specified. Appium then takes the driver commands in your test script, which are in a WebDriver JSON Wire Protocol f ormat, and converts them into UIAutomator Java commands. UIAutomator is the library provided by Google as part of the Android SDK, and is also the library that Appium uses to automate your Android mobile application tests.

In the case where you are connecting to the Selendroid automation backend, Appium simply proxies all requests to the Selendroid server running on the emulator/device.

The response from your mobile application are received by UIAutomator and relayed to the Appium server in the Sauce Labs cloud. The Appium server then converts the UIAutomator Java responses back into WebDriver JSON Wire Protocol format, and send the JSON responses back to your test script.

Appium hides all of this complexity from your test script. Your test script thinks it's communicating with your mobile application, but in reality it is communicating with Appium's implementation of the WebDriver API. The fact that Appium is running your mobile application in the appropriate simulator, emulator or device, and wrapping up all of the communications with your mobile application, including the command conversions, remains completely hidden, and your test script is none the wiser.

Appium Resources

For more information, feel free to visit the resources listed below:

Appium Introduction to Appium Concepts

Page 93 Copyright Sauce Labs 2015

Appium Docs Appium Discussion Group Appium on Github Selenium WebDriver JSON Wire Protocol Selenium Apple UI Automation Documentation Android UI Testing Documentation

Page 94 Copyright Sauce Labs 2015

Support and Requirements for Mobile Testing

Supported Android Emulators Supported Mobile Operating Systems Requirements for Testing Mobile Native and Hybrid Applications

Page 95 Copyright Sauce Labs 2015

Supported Android Emulators

Settings Applying to All Emulators

All emulators have 128MB SD cards configured as their external storage.

Emulator Specifics

Google Play Versions for Android Emulator Versions Google Play Services (com.google.android.gms) that is downloaded with Android 4.4 SDK: versionName=8.1.15 (2255478-030) Google Play Services (com.google.android.gms) that is downloaded with Android 5.0 SDK: versionName=8.1.15 (2255478-270) Google Play Services (com.google.android.gms) that is downloaded with Android 5.1 SDK: versionName=6.7.74 (1723905-470)

Name RAM Heap Size Data Partition Screen Diagonal Resolution Pixel DPI (dots per GoogleAPI Size Length Density inch) version*

DEFAULT: Generic 512 MB 2.3 to 4.2: 64MB 200MB 4.0" 480x800 HDPI 233 4.4/5.0 Phone 4.3 and above: 4.2 and 4.3: 32MB 300MB

Generic Tablet 512 MB 2.3-4.3: 64 MB 200MB 7.0" 800x1280 HDPI 216 4.4/5.0 4.4+: 32 MB 4.2 and 4.3: 300MB

Motorola Atrix HD 1024 64MB 200MB 4.5" 720x1280 XHDPI 326 N/A MB

Motorola Photon Q 1024 32MB 200MB 4.3" 540x960 HDPI 256 N/A MB

Motorola Razr Maxx HD 1024 64MB 200MB 4.7" 720x1280 XHDPI 312 N/A MB

Motorola Droid 4 1024 32MB 200MB 4.0" 540x960 HDPI 275 N/A MB

Motorola Droid Razr 1024 32MB 200MB 4.3" 540x960 HDPI 256 N/A MB

Amazon Kindle Fire 512 MB 16MB 200MB 7.0" 600x1024 MDPI 170 N/A

Amazon Kindle Fire HD 1907 32MB 200MB 8.9" 720x1220 *** MDPI 160 4.4/5.0 MB

Samsung Galaxy S 512 MB 32MB 200MB 4.0" 480x800 HDPI 233 N/A

Samsung Galaxy S2 1024 32MB 200MB 4.3" 480x800 HDPI 217 N/A MB

Samsung Galaxy S3 1900 64MB 200MB 4.8" 720x1280 XHDPI 306 4.4/5.0 MB

Samsung Galaxy S4 1900 64MB 200MB 5.0" 720x1280 *** XHDPI 293 4.4/5.0 MB

Samsung Galaxy Nexus 1024 64MB 200MB 4.6" 720x1280 XHDPI 320 4.4/5.0 MB

Samsung Galaxy Note 1024 64MB 200MB 5.3" 800x1280 XHDPI 285 N/A MB

Samsung Galaxy Note 1907 32MB 200MB 10.1" 800x1280 MDPI 150 N/A 10.1 MB

Samsung Galaxy Note 2 1907 32MB 200MB 5.5" 720x1280 HDPI 267 N/A MB

Samsung Galaxy Tab 2 1024 64MB 200MB 10.1" 800x1280 MDPI 150 N/A 10.1 MB

Samsung Galaxy Tab 3 1024 32MB 200MB 7.0" 600x1024 MDPI 170 N/A 7.0 MB

HTC One X 1024 64MB 200MB 4.7" 720x1280 XHDPI 312 N/A MB

HTC Evo 3D 1024 32MB 200MB 4.3" 540x960 HDPI 256 N/A MB

HTC Wildfire S 512 MB 16MB 200MB 3.2" 320x480 MDPI 180 N/A

LG Optimus 3D 512 MB 32MB 200MB 4.3" 480x800 TVDPI 217 N/A

Page 96 Copyright Sauce Labs 2015

Google Nexus 4 1907 64MB 200MB 4.7" 768x1280 XHDPI 318 4.4/5.0 MB

Google Nexus 7 C 1024 4.1 to 4.3: 64MB 200MB 7.0" 800x1280 TVDPI 216 4.4/5.0 MB 4.4: 32MB

Google Nexus 7 FHD 1907 4.3: 64MB 200MB 7.0" 800x1280 TVDPI 216 4.4/5.0 MB 4.4: 32MB

Sony Xperia X10 384 MB 32MB 200MB 4.0" 480x854 HDPI 245 N/A:

Page 97 Copyright Sauce Labs 2015

Supported Mobile Operating Systems

Sauce supports testing for these mobile operating systems.

Platform/Operating System Version Notes

iOS 6.1 and later

Android 4.4 and later For Mobile Application testing

Android 2.3, 4.0 and later for Mobile Native Application and Mobile Hybrid Application testing

Page 98 Copyright Sauce Labs 2015

Requirements for Testing Mobile Native and Hybrid Applications

Both iOS and Android have specific requirements for being able to run mobile native and hybrid application testing on Sauce. iOS Requirements

The mobile application must be compiled in debug mode The mobile application must be compiled for the simulator/device version of your choice The mobile application must be signed with a developer certificate (only if the application is signed) The mobile application must be hosted in a place that Sauce Labs can access, for example: A remote location, for example a GitHub Repository Your Temporary Sauce Storage

Android Requirements

The mobile application must be compiled for the emulator/device version of your choice The mobile application must have internet permissions The mobile application must be hosted in a place that Sauce Labs can access, for example: A remote location, for example a GitHub Repository Your Temporary Sauce Storage

Page 99 Copyright Sauce Labs 2015

Manual Testing for Mobile Apps

Sauce Labs does not currently support manual testing of iOS and Android apps. However, there is a workaround that uses automated sessions and JavaScript annotation, and relies on some experience with automated testing or scripting to start a session. For example, you need to know how to start a basic Appium test.

1. Upload your mobile app to Sauce Storage. 2. Create an automated test script with the desired capabilities for your test. 3. Have your script create a Sauce Labs job, then using the Javascript execution function of Selenium WebDriver, execute the script sauce : break. This will breakpoint your script on Sauce Labs. See Configuring Tests with Selenium's JavaScript Executor for more information on using sauce: break. 4. Exit your script without quitting the session. 5. Navigate to saucelabs.com, find the breakpointed session and open it. 6. Click on the video of the still-running test. You can now interact with the running emulator as though it were any other manual session.

Page 100 Copyright Sauce Labs 2015

Getting to the JavaScript Console for Manual iOS Browser Tests

Once you've got a manual session open for the iOS emulator you're after, you can inspect the JavaScript for the iOS Safari by following these instructions.

1. Click the manual session outside the boundary of the iOS simulator. The header should change to Finder. 2. Open the Desktop version of Safari. The version of Safari running on the Mac which is hosting the simulator. 3. Open the iOS Simulator again. 4. In the iOS version of Safari, open your site. 5. Go back to the Desktop version of Safari and open the Develop menu. 6. Select the type of simulator you're testing against. For example, iPad or iPhone. 7. Click yourpage.html. You should now see the same developer tools as though you were in Safari itself.

Page 101 Copyright Sauce Labs 2015

Running Emulator and Simulator Mobile Tests

Prerequisites

Before you get started with testing on Sauce, you should check out the Running Emulator and Simulator Mobile Tests and the Running Emulator and Simulator Mobile Tests If you're interested in testing mobile applications, you should read the Running Emulator and Simulator Mobile Tests If you want to test applications that are behind a firewall or on localhost, you should read up on Sauce Connect, which enables you to create a secure tunnel between the location where your application resides and the Sauce Labs testing infrastructure If you aren't already, you should familiarize yourself with using Selenium for Web application testing, and Appium for mobile application testing

Running Automated Tests for Web and Mobile Applications

Automated testing of both Web and mobile applications on Sauce Labs boils down to a few basic steps:

1. Set yourself up with a Sauce Labs account. Check out the topics under Running Emulator and Simulator Mobile Tests if you're interested in setting up an account for multiple team members. 2. Update your existing tests to run on Sauce. The topics under Running Emulator and Simulator Mobile Tests include examples of how to do this. 3. Write new tests that include the desired capabilities of your test, in the language of your choice.

You can use the Platform Configurator to set the desired capabilities for both Selenium and Appium tests in your preferred language. You can also use the REST API or the WebDriver API to further configure your jobs with timeouts, annotations (a recommended best practice for managing your tests results, and for managing builds in your continuous integration pipeline), and other job functionality. Both the Running Emulator and Simulator Mobile Tests and Tutorial topics include code samples that illustrate how to write automated tests for Web applications in various languages, while the topics under Running Emulator and Simulator Mobile Tests illustrate how to write scripts for testing iOS and Android mobile applications.

1. Run your tests and watch the results appear on your Sauce Labs dashboard, complete with video, screenshots, and logs.

Page 102 Copyright Sauce Labs 2015

Types of Mobile Tests

There are three kinds of mobile application tests you can run using Appium on Sauce: Mobile Native, Mobile Web, and Mobile Hybrid. The type of application you are testing will determine some of the Desired Capabilities you need to set for your test.

Mobile Native

This type of application is developed for an specific platform, for example, iOS or Android, using the native SDKs provided by the platform vendor, and distributed to users via the appropriate app store.

Mobile Web

A mobile web application is formally referred to as a mobile website. It can be accessed through a browser, for example Mobile Safari, Chrome, etc., in a mobile simulator/emulator or real device.

Mobile Hybrid

This type of application is part mobile native app and part mobile web app. Just like mobile native apps, you can find and download mobile hybrid apps using Apple’s App Store or the Google Play Store. A mobile hybrid app, however, looks like a mobile website that would be accessed through a browser, but in this case the browser is an embedded webview within the application that displays some HTML.

Page 103 Copyright Sauce Labs 2015

FAQs for Mobile Testing

How can I test Android tablets? How can I run manual tests for my mobile native app or mobile hybrid app? Sauce Labs doesn't support manual tests for mobile native app or mobile hybrid app tests. Check out Types of Mobile Tests for more information. What type of keyboard and buttons do the Android emulators have? How can I run Android tests without Appium? How can I run iOS tests without Appium? What mobile web browsers can I automate in the Android emulator? How can I test with mobile real devices instead of using the simulators or emulators?

How can I test Android tablets?

The best way to test on different Android emulators screen sizes is by using the different Android Emulator Skins . For instance, if you use our Pla tform Configurator you'll see the available skins for the different Android versions (e.g Google Nexus 7 HD, LG Nexus 4, Samsung Galaxy Nexus, Samsung Galaxy S3, etc). Some of these skins are tablets, for example the Google Nexus 7C is a tablet which has a very large resolution and very high density.

How can I run manual tests for my mobile native app or mobile hybrid app?

Sauce Labs doesn't support manual tests for mobile native app or mobile hybrid app tests. Check out Types of Mobile Tests for more information.

What type of keyboard and buttons do the Android emulators have?

Android Emulators have software buttons and a hardware keyboard. In a regular Android emulator the device buttons are software buttons displayed on the right size of the emulator. For the Android emulators with different skins (e.g Google Nexus 7 HD, LG Nexus 4, Samsung Galaxy Nexus, Samsung Galaxy S3, etc) the device buttons are also software buttons that are overplayed on top of the skin. For instance, if you hover the mouse around the edges of any of our Android emulators with an specified skin, a hover icon will appear and you should be able to find whatever buttons actually exist on the device that the skinned emulator is trying to emulate (e.g power button along the top, volume buttons along the edge, back/home buttons right below the screen, etc).

How can I run Android tests without Appium?

For older versions of Android Appium might not be supported. For instance, Appium is only supported in Android versions 4.4 or later for Mobile Web Application tests, and Android versions 2.3, 4.0 and later for Mobile Native Application and Mobile Hybrid Application tests.

For those versions in which Appium is not supported you can request an emulator driven by Webdriver + Selendroid. All you need to do is use our Platforms Configurator and select Selenium for the API instead of Appium.

In the Sauce Labs test you will notice that the top of the emulator says "AndroidDriver Webview App". In addition, you will notice that you will get a "Selenium Log" tab which has the output of the Selendroid driver.

With an emulator driven by Webdriver + Selendroid you will be able to test Mobile Web Application only. You should be able to select any Android emulator version from 4.0 to the latest version and any Android emulator skin (e.g "deviceName":"Samsung Galaxy Tab 3 Emulator").

How can I run iOS tests without Appium?

For older versions of iOS Appium might not be supported. For instance, Appium is supported in iOS versions 6.1 and later. For earlier versions of iOS the tool or driver used to drive your mobile applications automated test is called iWebdriver.

To obtain a simulator driven by iWebdriver use our Platforms Configurator and select Selenium for the API instead of Appium. With an emulator driven by iWebdriver you will be able to test Mobile Web Applicationonly. In addition, in the Sauce Labs test you will notice a "Selenium Log" tab which has the output of iWebdriver.

What mobile web browsers can I automate in the Android emulator?

Currently the only browser that can be automated in our Android emulators is the stock browser (i.e Browser). The Android stock browser is an Android flavor of 'chromium' which presumably implies that its behavior is closer to that of Google Chrome.

Page 104 Copyright Sauce Labs 2015

How can I test with mobile real devices instead of using the simulators or emulators?

The mobile real device cloud is a new feature that Sauce Labs is currently working on. For more information about this feature please directly email one of our sales team representatives ([email protected]).

Page 105 Copyright Sauce Labs 2015

Example Appium Mobile Testing Scripts

These are examples of scripts in several popular languages, for each major mobile platform, that were written for Appium mobile testing on Sauce Labs.

Android Example Scripts Written for Mobile Testing on Sauce Labs Example Java Script for Testing Android Mobile Applications on Sauce Labs Example Node.js Script for Testing Android Mobile Applications on Sauce Labs Example Python Script for Testing Android Mobile Applications on Sauce Example Ruby Script for Testing Android Mobile Applications on Sauce Labs iOS Example Scripts for Mobile Testing on Sauce Example Java Script for Testing iOS Mobile Applications on Sauce Labs Example Node.js Script for Testing iOS Mobile Applications on Sauce Example PHP Script for Testing iOS Mobile Applications on Sauce Labs Example Python Script for Testing iOS Mobile Applications on Sauce Labs Example Ruby Script for Testing iOS Mobile Applications on Sauce Labs

Page 106 Copyright Sauce Labs 2015

Android Example Scripts Written for Mobile Testing on Sauce Labs

Example Java Script for Testing Android Mobile Applications on Sauce Labs Example Node.js Script for Testing Android Mobile Applications on Sauce Labs Example Python Script for Testing Android Mobile Applications on Sauce Example Ruby Script for Testing Android Mobile Applications on Sauce Labs

Page 107 Copyright Sauce Labs 2015

Example Java Script for Testing Android Mobile Applications on Sauce Labs

You can also view this script on GitHub.

Example Java Script for Android Testing Expand source

Page 108 Copyright Sauce Labs 2015 package com.saucelabs.appium; import static org.junit.Assert.assertEquals; import io.appium.java_client.AppiumDriver; import io.appium.java_client.android.AndroidDriver; import java.io.File; import java.net.URL; import java.util.List; import org.junit.After; import org.junit.Before; import org.junit.Test; import org.openqa.selenium.By; import org.openqa.selenium.WebElement; import org.openqa.selenium.remote.DesiredCapabilities; public class AndroidTest {

private AppiumDriver driver;

@Before public void setUp() throws Exception { File classpathRoot = new File(System.getProperty("user.dir")); File appDir = new File(classpathRoot, "../../../apps/ApiDemos/bin"); File app = new File(appDir, "ApiDemos-debug.apk"); DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability("deviceName","Android Emulator"); capabilities.setCapability("platformVersion", "4.4"); capabilities.setCapability("app", app.getAbsolutePath()); capabilities.setCapability("appPackage", "io.appium.android.apis"); capabilities.setCapability("appActivity", ".ApiDemos"); driver = new AndroidDriver<>(new URL("http://127.0.0.1:4723/wd/hub"), capabilities); }

@After public void tearDown() throws Exception { driver.quit(); }

@Test public void apiDemo(){ WebElement el = driver.findElement(By.name("Animation")); assertEquals("Animation", el.getText()); el = driver.findElementByClassName("android.widget.TextView"); assertEquals("API Demos", el.getText()); el = driver.findElement(By.name("App")); el.click(); List els = driver.findElementsByClassName("android.widget.TextView"); assertEquals("Activity", els.get(2).getText()); } }

Page 109 Copyright Sauce Labs 2015

Example Node.js Script for Testing Android Mobile Applications on Sauce Labs

You can also view this script on GitHub.

Example PHP Script for Android Testing Expand source "use strict";

require("./helpers/setup");

var wd = require("wd"), _ = require('underscore'), serverConfigs = require('./helpers/appium-servers');

describe("android simple", function () { this.timeout(300000); var driver; var allPassed = true;

before(function () { var serverConfig = process.env.SAUCE ? serverConfigs.sauce : serverConfigs.local; driver = wd.promiseChainRemote(serverConfig); require("./helpers/logging").configure(driver);

var desired = process.env.SAUCE ? _.clone(require("./helpers/caps").android18) : _.clone(require("./helpers/caps").android19); desired.app = require("./helpers/apps").androidApiDemos; if (process.env.SAUCE) { desired.name = 'android - simple'; desired.tags = ['sample']; } return driver .init(desired) .setImplicitWaitTimeout(3000); });

after(function () { return driver .quit() .finally(function () { if (process.env.SAUCE) { return driver.sauceJobStatus(allPassed); } }); });

afterEach(function () { allPassed = allPassed && this.currentTest.state === 'passed'; });

it("should find an element", function () { return driver .elementByAccessibilityId('Graphics') .click() .elementByAccessibilityId('Arcs') .should.eventually.exist

Page 110 Copyright Sauce Labs 2015

.back() .elementByName('App') .should.eventually.exist .elementsByAndroidUIAutomator('new UiSelector().clickable(true)') .should.eventually.have.length(12) .elementsByAndroidUIAutomator('new UiSelector().enabled(true)') .should.eventually.have.length.above(20) .elementByXPath('//android.widget.TextView[@text=\'API Demos\']')

Page 111 Copyright Sauce Labs 2015

.should.exists; }); });

Page 112 Copyright Sauce Labs 2015

Example Python Script for Testing Android Mobile Applications on Sauce

You can also view this script on GitHub.

Example Python Script for Android Mobile Testing Expand source import os from time import sleep

import unittest

from appium import webdriver

# Returns abs path relative to this file and not cwd PATH = lambda p: os.path.abspath( os.path.join(os.path.dirname(__file__), p) )

class SimpleAndroidTests(unittest.TestCase): def setUp(self): desired_caps = {} desired_caps['platformName'] = 'Android' desired_caps['platformVersion'] = '4.2' desired_caps['deviceName'] = 'Android Emulator' desired_caps['app'] = PATH( '../../../sample-code/apps/ApiDemos/bin/ApiDemos-debug.apk' )

self.driver = webdriver.Remote('http://localhost:4723/wd/hub', desired_caps)

def tearDown(self): # end the session self.driver.quit()

def test_find_elements(self):

el = self.driver.find_element_by_accessibility_id('Graphics') el.click() el = self.driver.find_element_by_accessibility_id('Arcs') self.assertIsNotNone(el)

self.driver.back()

el = self.driver.find_element_by_accessibility_id("App") self.assertIsNotNone(el)

els = self.driver.find_elements_by_android_uiautomator("new UiSelector().clickable(true)") self.assertGreaterEqual(12, len(els))

self.driver.find_element_by_android_uiautomator('text("API Demos")')

def test_simple_actions(self): el = self.driver.find_element_by_accessibility_id('Graphics') el.click()

el = self.driver.find_element_by_accessibility_id('Arcs') el.click()

Page 113 Copyright Sauce Labs 2015

self.driver.find_element_by_android_uiautomator('new UiSelector().text("Graphics/Arcs")')

Page 114 Copyright Sauce Labs 2015 if __name__ == '__main__': suite = unittest.TestLoader().loadTestsFromTestCase(SimpleAndroidTests) unittest.TextTestRunner(verbosity=2).run(suite)

Page 115 Copyright Sauce Labs 2015

Example Ruby Script for Testing Android Mobile Applications on Sauce Labs

This example uses environment variables for your Sauce Labs authentication credentials, and the sauce-whisk gem.

You can also view this script on GitHub.

Example Ruby Script for Android Mobile Testing Expand source # This example automates a test of the Android example notepad app. # # To run this example, make sure you've run: # $ bundle install # # And set the environment variables: # SAUCE_USERNAME = your-sauce-username # SAUCE_ACCESS_KEY = your-sauce-key # Then just: # bundle exec ruby android_on_sauce.rb # # Of note compared to the iOS example, here we're giving the package and # activity, no OS and an empty browserName # # Of note compared to the non-sauce examples, you need to host your app # somewhere Sauce Labs' cloud can fetch it for your test.

require 'rubygems' require 'spec' require 'appium_lib' require 'sauce_whisk'

describe 'Notepad' do def desired_caps { caps: { :'appium-version' => '1.3.4', platformName: 'Android', platformVersion: '4.3', deviceName: 'Android Emulator', app: 'http://appium.s3.amazonaws.com/NotesList.apk', name: 'Ruby Appium Android example' }, appium_lib: { wait: 60 } } end

before do Appium::Driver.new(desired_caps).start_driver end

after do driver_quit end

it 'can create and save new notes' do find('New note').click first_textfield.type 'This is a new note, from Ruby'

Page 116 Copyright Sauce Labs 2015

find('Save').click

note_count = ids('android:id/text1').length note_count.must_equal 1 texts.last.text.must_equal 'This is a new note, from Ruby' end end passed = Minitest.run_specs({ :trace => [__FILE__] }).first

# Because WebDriver doesn't have the concept of test failure, use the Sauce # Labs REST API to record job success or failure user = ENV['SAUCE_USERNAME'] key = ENV['SAUCE_ACCESS_KEY'] if user && !user.empty? && key && !key.empty?

Page 117 Copyright Sauce Labs 2015

passed = passed.failures == 0 && passed.errors == 0 SauceWhisk::Jobs.change_status $driver.driver.session_id, passed end

Page 118 Copyright Sauce Labs 2015 iOS Example Scripts for Mobile Testing on Sauce

Example Java Script for Testing iOS Mobile Applications on Sauce Labs Example Node.js Script for Testing iOS Mobile Applications on Sauce Example PHP Script for Testing iOS Mobile Applications on Sauce Labs Example Python Script for Testing iOS Mobile Applications on Sauce Labs Example Ruby Script for Testing iOS Mobile Applications on Sauce Labs

Page 119 Copyright Sauce Labs 2015

Example Java Script for Testing iOS Mobile Applications on Sauce Labs

This example uses the JUnit testing framework and authenticates to Sauce Labs using credentials set as environment variables.

You can also view this example in GitHub.

Example Java Script for iOS Testing Expand source import com.saucelabs.common.SauceOnDemandAuthentication; import com.saucelabs.common.SauceOnDemandSessionIdProvider; import com.saucelabs.junit.SauceOnDemandTestWatcher; import io.appium.java_client.MobileElement; import io.appium.java_client.ios.IOSDriver; import io.appium.java_client.remote.MobileCapabilityType; import junit.framework.TestCase; import org.junit.After; import org.junit.Before; import org.junit.Rule; import org.junit.Test; import org.junit.rules.TestName; import org.openqa.selenium.remote.DesiredCapabilities;

import java.net.MalformedURLException; import java.net.URL; import java.util.concurrent.TimeUnit;

import static junit.framework.Assert.assertEquals; import static junit.framework.TestCase.assertTrue;

public class SimpleIOSSauceTests implements SauceOnDemandSessionIdProvider { final private String USERNAME = System.getenv("SAUCE_USERNAME"); final private String ACCESS_KEY = System.getenv("SAUCE_ACCESS_KEY"); private SauceOnDemandAuthentication authentication = new SauceOnDemandAuthentication(USERNAME, ACCESS_KEY);

private IOSDriver driver; private String sessionId;

@Rule public SauceOnDemandTestWatcher resultReportingTestWatcher = new SauceOnDemandTestWatcher(this, authentication); @Override public String getSessionId() { return sessionId; }

public @Rule TestName name = new TestName();

@Before public void setUp() throws MalformedURLException { DesiredCapabilities desiredCapabilities = new DesiredCapabilities(); desiredCapabilities.setCapability(MobileCapabilityType.PLATFORM_VERSION, "7.1"); desiredCapabilities.setCapability(MobileCapabilityType.DEVICE_NAME, "iPhone Simulator"); desiredCapabilities.setCapability(MobileCapabilityType.APP, "http://appium.s3.amazonaws.com/TestApp6.0.app.zip"); desiredCapabilities.setCapability("appiumVersion", "1.3.4"); desiredCapabilities.setCapability("name", name.getMethodName());

Page 120 Copyright Sauce Labs 2015

URL sauceUrl = new URL("http://" + authentication.getUsername() + ":"+ authentication.getAccessKey() + "@ondemand.saucelabs.com:80/wd/hub");

driver = new IOSDriver(sauceUrl, desiredCapabilities); driver.manage().timeouts().implicitlyWait(30, TimeUnit.SECONDS); sessionId = driver.getSessionId().toString(); }

@After public void tearDown() { System.out.println("Link to your job: https://saucelabs.com/jobs/" + this.getSessionId()); driver.quit(); }

@Test public void testUIComputation() {

// populate text fields with values MobileElement fieldOne = (MobileElement) driver.findElementByAccessibilityId("TextField1"); fieldOne.sendKeys("12");

MobileElement fieldTwo = (MobileElement) driver.findElementsByClassName("UIATextField").get(1); fieldTwo.sendKeys("8");

// they should be the same size, and the first should be above the second assertTrue(fieldOne.getLocation().getY() < fieldTwo.getLocation().getY()); assertEquals(fieldOne.getSize(), fieldTwo.getSize());

// trigger computation by using the button driver.findElementByAccessibilityId("ComputeSumButton").click();

// is sum equal? String sum = driver.findElementsByClassName("UIAStaticText").get(0).getText(); TestCase.assertEquals(Integer.parseInt(sum), 20);

Page 121 Copyright Sauce Labs 2015

}

Page 122 Copyright Sauce Labs 2015

Example Node.js Script for Testing iOS Mobile Applications on Sauce

You can also view this script on GitHub.

Example Node.js Script for iOS Testing Expand source "use strict";

require("./helpers/setup");

var wd = require("wd"), _ = require('underscore'), Q = require('q'), serverConfigs = require('./helpers/appium-servers');

describe("ios simple", function () { this.timeout(300000); var driver; var allPassed = true;

before(function () { var serverConfig = process.env.SAUCE ? serverConfigs.sauce : serverConfigs.local; driver = wd.promiseChainRemote(serverConfig); require("./helpers/logging").configure(driver);

var desired = _.clone(require("./helpers/caps").ios81); desired.app = require("./helpers/apps").iosTestApp; if (process.env.SAUCE) { desired.name = 'ios - simple'; desired.tags = ['sample']; } return driver.init(desired); });

after(function () { return driver .quit() .finally(function () { if (process.env.SAUCE) { return driver.sauceJobStatus(allPassed); } }); });

afterEach(function () { allPassed = allPassed && this.currentTest.state === 'passed'; });

Page 123 Copyright Sauce Labs 2015

}; }); return seq.reduce(Q.when, new Q(0)); } it("should compute the sum", function () { return driver .resolve(populate()).then(function (sum) { return driver. elementByAccessibilityId('ComputeSumButton') .click().sleep(1000) .elementByIosUIAutomation('.elements().withName("Answer");') .text().should.become("" + sum); });

Page 124 Copyright Sauce Labs 2015

});

Page 125 Copyright Sauce Labs 2015

Example PHP Script for Testing iOS Mobile Applications on Sauce Labs

This script includes using Sausage, a set of libraries and classes developed for running PHP tests with PHPUnit, either locally or on Sauce Labs.

You can also view this code example on GitHub.

Example PHP Script for iOS Testing Expand source

require_once "vendor/autoload.php"; define("APP_URL", "http://appium.s3.amazonaws.com/TestApp6.0.app.zip");

class SauceTest extends Sauce\Sausage\WebDriverTestCase { protected $numValues = array();

public static $browsers = array( array( 'browserName' => '', 'seleniumServerRequestsTimeout' => 240, 'desiredCapabilities' => array( 'platform' => 'Mac 10.8', 'device' => 'iPhone Simulator', 'app' => APP_URL, 'version' => '6.1', ) ) );

public function elemsByTag($tag) { return $this->elements($this->using('tag name')->value($tag)); }

protected function populate() { $elems = $this->elemsByTag('textField'); foreach ($elems as $elem) { $randNum = rand(0, 10); $elem->value($randNum); $this->numValues[] = $randNum; } }

public function testUiComputation() { $this->populate(); $buttons = $this->elemsByTag('button'); $buttons[0]->click(); $texts = $this->elemsByTag('staticText'); $this->assertEquals(array_sum($this->numValues), (int)($texts[0]->text())); } }

Page 126 Copyright Sauce Labs 2015

Page 127 Copyright Sauce Labs 2015

Example Python Script for Testing iOS Mobile Applications on Sauce Labs

This test assumes you have set your Sauce Labs authentication credentials as environment variables.

You can also view this test on GitHub.

Example Python Script for iOS Testing Expand source """An example of Appium running on Sauce.

This test assumes SAUCE_USERNAME and SAUCE_ACCESS_KEY are environment variables set to your Sauce Labs username and access key."""

from random import randint from appium import webdriver from appium import SauceTestCase, on_platforms

platforms = [{ 'platformName': 'iOS', 'platformVersion': '7.1', 'deviceName': 'iPhone Simulator', 'app': 'http://appium.s3.amazonaws.com/TestApp6.0.app.zip', 'appiumVersion': '1.3.4' }]

@on_platforms(platforms) class SimpleIOSSauceTests(SauceTestCase):

def _populate(self): # populate text fields with two random numbers els = self.driver.find_elements_by_class_name('UIATextField')

self._sum = 0 for i in range(2): rnd = randint(0, 10) els[i].send_keys(rnd) self._sum += rnd

def test_ui_computation(self): # populate text fields with values self._populate()

# trigger computation by using the button self.driver.find_element_by_accessibility_id('ComputeSumButton').click()

# is sum equal ? sum = self.driver.find_elements_by_class_name("UIAStaticText")[0].text self.assertEqual(int(sum), self._sum)

Page 128 Copyright Sauce Labs 2015

Example Ruby Script for Testing iOS Mobile Applications on Sauce Labs

You can also view this script on GitHub.

Example Ruby Script for iOS Testing Expand source # GETTING STARTED # ------# This documentation is intended to show you how to get started with a # simple Appium & appium_lib test. This example is written without a specific # testing framework in mind; You can use appium_lib on any framework you like. # # INSTALLING RVM # ------# If you don't have rvm installed, run the following terminal command # # \curl -L https://get.rvm.io | bash -s stable --ruby # # INSTALLING GEMS # ------# Then, change to the example directory: # cd appium-location/sample-code/examples/ruby # # and install the required gems with bundler by doing: # bundle install # # RUNNING THE TESTS # ------# To run the tests, make sure appium is running in another terminal # window, then from the same window you used for the above commands, type # # bundle exec ruby simple_test.rb # # It will take a while, but once it's done you should get nothing but a line # telling you "Tests Succeeded"; You'll see the iOS Simulator cranking away # doing actions while we're running.

require 'rubygems' require 'appium_lib'

APP_PATH = '../../apps/TestApp/build/release-iphonesimulator/TestApp.app'

desired_caps = { caps: { platformName: 'iOS', versionNumber: '8.1', deviceName: 'iPhone 6', app: APP_PATH, }, appium_lib: { sauce_username: nil, # don't run on Sauce sauce_access_key: nil } }

# Start the driver Appium::Driver.new(desired_caps).start_driver

Page 129 Copyright Sauce Labs 2015 module Calculator Module IOS # Add all the Appium library methods to Test to make # calling them look nicer. Appium.promote_singleton_appium_methods Calculator

# Add two numbers values = [rand(10), rand(10)] expected_sum = values.reduce(&:+)

# Find every textfield. elements = textfields

elements.each_with_index do |element, index| element.type values[index] end

# Click the first button button(1).click

# Get the first static text field, then get its text actual_sum = first_text.text raise unless actual_sum == (expected_sum.to_s)

# Alerts are visible button('show alert').click find_element :class_name, 'UIAAlert' # Elements can be found by :class_name

# wait for alert to show wait { text 'this alert is so cool' }

# Or by find find('Cancel').click

# Waits until alert doesn't exist wait_true { !exists { tag('UIAAlert') } }

# Alerts can be switched into wait { button('show alert').click } # Get a button by its text alert = driver.switch_to.alert # Get the text of the current alert, using

# the Selenium::WebDriver directly alerting_text = alert.text raise Exception unless alerting_text.include? 'Cool title' alert_accept # Accept the current alert

# Window Size is easy to get sizes = window_size raise Exception unless sizes.height == 667 raise Exception unless sizes.width == 375

# Quit when you're done! driver_quit puts 'Tests Succeeded!' end end

Page 130 Copyright Sauce Labs 2015

Page 131 Copyright Sauce Labs 2015

Running Tests in Parallel with Sauce Labs

Running tests in parallel is the secret Sauce for accelerating your development process and creating a continuous integration/continuous delivery pipeline. These topics include examples of using various popular testing frameworks to set up test parallelization with your favorite language.

Running Tests in Parallel with C# Running C# Tests in Parallel with Gallio and MBUnit Running C# Tests in Parallel with PNUnit Running Tests in Parallel with Java Running Java Tests in Parallel with JUnit Running Java Tests in Parallel with TestNG Running Tests in Parallel with PHP Running PHP Tests in Parallel with PHPUnit and Paratest Running Tests in Parallel with Python Running Python Tests in Parallel with nose Running Python Tests in Parallel with pytest Running Tests in Parallel with Ruby Running Ruby Tests in Parallel with RSpec Troubleshooting Parallel Tests on Sauce Labs

Page 132 Copyright Sauce Labs 2015

Running Tests in Parallel with C#

Running C# Tests in Parallel with Gallio and MBUnit Running C# Tests in Parallel with PNUnit

Page 133 Copyright Sauce Labs 2015

Running C# Tests in Parallel with Gallio and MBUnit

The Gallio Automation Framework is an open, extensible, and neutral system for .NET that provides a common object model, runtime services, and tools (such as test runners) that you can use with a variety of test frameworks. MBUnit is an extensible unit testing framework for .NET that is bundled with Gallio. You can find full documentation for both on the MB-Unit project site.

Prerequisites Installing Gallio and MBUnit Code Sample GuineaPigTests.cs Constants.cs Run the Test

Prerequisites

You'll need to have these components installed to set up MBUnit testing on Sauce with C# .NET.

Visual Studio Selenium DLLs for .NET installed and referenced by your project Gallio and MBUnit Bundle

Installing Gallio and MBUnit

1. In the Visual Studio Tools menu, go to Library Package Manager > Manage Nuget Package for Solution. This will open the Manage NuGet Packages screen. 2. Click Online, and then click Next. 3. In the Search Packages field enter Gallio, and then click Search. 4. In the search results, select Gallio & MbUnit, and then click Install. 5. After installing Gallio & MbUnit, select Gallio Bundle from the search results, and then click Install.

Code Sample

GuineaPigTests.cs

You can use the Platform Configurator to specify the desired capabilities for any browser/platform combination you want for your test.

GuineaPigTests.cs Expand source using Gallio.Framework; using Gallio.Model; using MbUnit.Framework; using OpenQA.Selenium; using OpenQA.Selenium.Remote; using OpenQA.Selenium.Support.UI; using System;

namespace SauceLabs.SeleniumExamples

{ ///

tests for the sauce labs guinea pig page

[TestFixture] [Header("browser", "version", "platform")] // name of the parameters in the rows [Row("internet explorer", "11", "Windows 7")] // run all tests in the fixture against IE 11 for windows 7 [Row("chrome", "35", "linux")] // run all tests in the fixture against chrome 35 for linux

Page 134 Copyright Sauce Labs 2015

[Row("safari","6","OS X 10.8")] // run all tests in the fixture against safari 6 and mac OS X 10.8 public class GuineaPigTests

{ #region Setup and Teardown

///

starts a sauce labs sessions

/// name of the browser to request /// version of the browser to request /// operating system to request private IWebDriver _Setup(string browser, string version, string platform) { // construct the url to sauce labs Uri commandExecutorUri = new Uri(""); // set up the desired capabilities DesiredCapabilities desiredCapabilites = new DesiredCapabilities(browser, version, Platform.CurrentPlatform); // set the desired browser desiredCapabilites.SetCapability("platform", platform); // operating system to use desiredCapabilites.SetCapability("username", Constants.SAUCE_LABS_ACCOUNT_NAME); // supply sauce labs username desiredCapabilites.SetCapability("accessKey", Constants.SAUCE_LABS_ACCOUNT_KEY); // supply sauce labs account key desiredCapabilites.SetCapability("name", TestContext.CurrentContext.Test.Name); // give the test a name

// start a new remote web driver session on sauce labs var _Driver = new RemoteWebDriver(commandExecutorUri, desiredCapabilites); _Driver.Manage().Timeouts().ImplicitlyWait(TimeSpan.FromSeconds(30));

// navigate to the page under test _Driver.Navigate().GoToUrl("");

return _Driver; }

///

called at the end of each test to tear it down

public void CleanUp(IWebDriver _Driver) { // get the status of the current test bool passed = TestContext.CurrentContext.Outcome.Status == TestStatus.Passed; try { // log the result to sauce labs ((IJavaScriptExecutor)_Driver).ExecuteScript("sauce:job-result=" + (passed ? "passed" : "failed")); } finally { // terminate the remote webdriver session _Driver.Quit(); } }

\#endregion

Page 135 Copyright Sauce Labs 2015

\#region Tests

///

tests the title of the page

[Test, Parallelizable] // denotes that this method is a test and can be run in parallel public void PageTitle(string browser, string version, string platform) { // start the remote webdriver session with sauce labs var _Driver = _Setup(browser, version, platform);

// verify the page title is correct Assert.Contains(_Driver.Title, "I am a page title - Sauce Labs"); CleanUp(_Driver); }

///

tests that the link works on the page

[Test, Parallelizable] // denotes that this method is a test and can be run in parallel public void LinkWorks(string browser, string version, string platform) { // start the remote webdriver session with sauce labs var _Driver = _Setup(browser, version, platform);

// find and click the link on the page var link = _Driver.FindElement(By.Id("i am a link")); link.Click();

// wait for the page to change WebDriverWait wait = new WebDriverWait(_Driver, TimeSpan.FromSeconds(30)); wait.Until((d) => { return d.Url.Contains("guinea-pig2"); }); // verify the browser was navigated to the correct page Assert.Contains(_Driver.Url, "[saucelabs.com/test-guinea-pig2.html](http://saucelabs.com/test-guinea-pig2.html)");

CleanUp(_Driver); }

///

tests that a useragent element is present on the page

[Test, Parallelizable] // denotes that this method is a test and can be run in parallel public void UserAgentPresent(string browser, string version, string platform) { // start the remote webdriver session with sauce labs var _Driver = _Setup(browser, version, platform);

// read the useragent string off the page var useragent = _Driver.FindElement(By.Id("useragent")).Text;

Assert.IsNotNull(useragent); CleanUp(_Driver); }

Page 136 Copyright Sauce Labs 2015

\#endregion } }

Constants.cs

Use this class to set your Sauce Labs account name and access key. You can find these in the User Profile menu of your Sauce Labs dashboard.

Hardcoding v. Using Environment Variables for Authentication You could also hardcode your authentication credentials in the GuineaPigsTest.cs file in these lines:

Constants.cs Expand source namespace SauceLabs.SeleniumExamples

{ ///

contains constants used by the tests such as the user name and password for the sauce labs

internal static class Constants { ///

name of the sauce labs account that will be used

internal const string SAUCE_LABS_ACCOUNT_NAME = "Your Account Name"; ///

account key for the sauce labs account that will be used

internal const string SAUCE_LABS_ACCOUNT_KEY = "Your Access Key";

// NOTE: // To change the maximum number of parallel tests edit DegreeOfParallelism in AssemblyInfo.cs

} }

Run the Test

1. In the Solution Explorer, select your project and right-click Properties. 2. Select Debug. 3. Under Start Action, select Start an External Program. 4. Set the path to the Gallio Icarus GUI Runner. The default path will probably look something like this: C:\Program Files (x86)\Gallio\bin\Gallio.Icarus.exe. 5. Set the command line arguments to the name of your DLL. For this example, you should set it to SauceLabs.SeleniumExamples.dll. 6. Press F5 to build the project and debug the solution.

Page 137 Copyright Sauce Labs 2015

This launches the Gallio test GUI where you can run your tests. The DLL for this project should be preloaded. If it isn’t, you can open it from the File menu.

Page 138 Copyright Sauce Labs 2015

Running C# Tests in Parallel with PNUnit

Prerequisites Create the Visual Studio Project Install the Selenium DLLs Install NUnit + PNUnit and Import the Libraries into Your Project Code Example SaucePNUnit_Test.cs Constants.cs sauce_test.conf Run the Test

NUnit is a unit-test framework for all .Net languages, written entirely in C# and designed to take advantage of many .NET language features, for example custom attributes and other reflection-related capabilities. PNUnit, which stands for "Parallel NUnit," is an extension of NUnit that allows NUnit tests to run in parallel using a special .conf configuration file that specifies the tests to be executed and where they should run, whether on the same machine or on another machine on the network. For more information and documentation about the framework, as well as how to use it in your testing, you can visit the official NUnit website.

Prerequisites

You'll need to have these components installed to set up PNUnit testing on Sauce with C# .NET.

Visual Studio Selenium DLLs for .NET installed and referenced by your project NUnit + PNUnit Bundle

Create the Visual Studio Project

1. Open a new project in Visual Studio.

Set the Right .net Framework Version To get PNUnit to work for your project, you will need to set the .Net Framework version to 3.5. Click the list of framework versions at the top of the New Project dialog and select .NET Framework 3.5.

2. Select a C# class library template. 3. Give the project a name and click OK.

Install the Selenium DLLs

After you've set up your project in Visual Studio, you need to make sure that it references the required Selenium DLLs for .NET.

1. Download the Selenium DLLs for .NET from http://selenium-release.storage.googleapis.com/index.html?path=2.47/ 2. In the Solutions Explorer, select the project and right-click References. 3. Click Add Reference. 4. Click Browse and navigate to the net35 folder of the directory where you saved the Selenium .NET DLLs. 4. Add all four .DLL references to your project.

Install NUnit + PNUnit and Import the Libraries into Your Project

1. Download the current stable release of NUnit from http://www.nunit.org/index.php?p=download. 2. In the Solutions Explorer, select the project and right-click References. 3. Click Add Reference. 4. Click Browse and navigate to the bin of the directory where you saved NUnit. 5. Add the nunit.framework.dll and pnunit.framework.dll reference to your project.

Code Example

SaucePNUnit_Test.cs

SaucePNUnit_Test.cs Expand source using NUnit.Framework; using PNUnit.Framework; using System;

Page 139 Copyright Sauce Labs 2015 using System.Threading; using System.Web; using System.Text; using System.Net; using OpenQA.Selenium; using OpenQA.Selenium.Remote; using OpenQA.Selenium.Support.UI; namespace SauceLabs.NUnitExample { [TestFixture()] public class SaucePNUnit_Test { private IWebDriver driver; private string[] testParams; [SetUp] public void Init() { testParams = PNUnitServices.Get().GetTestParams(); String params1 = String.Join(",", testParams); Console.WriteLine(params1); String browser = testParams[0]; String version = testParams[1]; String platform = testParams[2]; DesiredCapabilities caps = new DesiredCapabilities(); caps.SetCapability("browserName", browser); caps.SetCapability(CapabilityType.Version, version); caps.SetCapability(CapabilityType.Platform, platform); caps.SetCapability("username", Constants.SAUCE_LABS_ACCOUNT_NAME); caps.SetCapability("accessKey", Constants.SAUCE_LABS_ACCOUNT_KEY); caps.SetCapability("name", TestContext.CurrentContext.Test.Name); Console.WriteLine("Capabilities" + caps.ToString()); driver = new RemoteWebDriver(new Uri("http://ondemand.saucelabs.com:80/wd/hub"), caps, TimeSpan.FromSeconds(420));

} [Test] public void googleTest() { driver.Navigate().GoToUrl("http://www.google.com"); StringAssert.Contains("Google", driver.Title); IWebElement query = driver.FindElement(By.Name("q")); query.SendKeys("Sauce Labs"); query.Submit(); } [TearDown] public void Cleanup() { // Gets the status of the current test bool passed = TestContext.CurrentContext.Result.Status == TestStatus.Passed; try { // Logs the result to Sauce Labs ((IJavaScriptExecutor)driver).ExecuteScript("sauce:job-result=" + (passed ? "passed" : "failed")); } finally { // Terminates the remote webdriver session

Page 140 Copyright Sauce Labs 2015

driver.Quit(); }

} }

Page 141 Copyright Sauce Labs 2015

}

This example test opensGoogle, verifies that "Google" is the title of the page, and then searches for Sauce Labs.

Constants.cs

Constants.cs Expand source using System; namespace SauceLabs.NUnitExample { ///

contains constants used by the tests such as the user name and password for the sauce labs

internal static class Constants { ///

name of the sauce labs account that will be used

internal readonly static string SAUCE_LABS_ACCOUNT_NAME = System.Environment.GetEnvironmentVariable("SAUCE_USERNAME"); ///

account key for the sauce labs account that will be used

internal readonly static string SAUCE_LABS_ACCOUNT_KEY = System.Environment.GetEnvironmentVariable("SAUCE_ACCESS_KEY");

} }

Use this class to set your Sauce Labs account name and access key as environment variables, as shown in the example test. You can find these in the User Profile menu of your Sauce Labs dashboard, under User Settings.

Hard-coding Your User Credentials If you prefer to hard-code your access credentials into your test, you would do so in the `SaucePNUnit_Test.cs` file, in the lines:

caps.SetCapability("username", Constants.SAUCE_LABS_ACCOUNT_NAME); caps.SetCapability("accessKey", Constants.SAUCE_LABS_ACCOUNT_KEY);

However, Sauce Labs recommends using environment variables for authentication as a best practice. This adds a layer of security to your tests, and allow other members of your team to share authentication credentials. sauce_test.conf

Page 142 Copyright Sauce Labs 2015

sauce_test.conf Expand source Testing TestFF-40-Win8 SauceLabs.NUnitExample.dll SauceLabs.NUnitExample.SaucePNUnit_Test.googleTest localhost:8080 firefox 40 Windows 8 TestChrome-44-Win7 PNUnit_Test2.dll SauceLabs.NUnitExample.SaucePNUnit_Test.googleTest localhost:8080 chrome 44 Windows 7

Use this file to set up the configuration for your PNUnit testing.

Configuration for Mobile Testing If you want to do mobile testing, you will need to add additional strings like deviceName and device-orientation inside the TestP arams section, and you will also need to add those as DesiredCapabilities in your C# test file.

Run the Test

Port for PNUnit Agent In the agent.conf file you can specify the port on which the PNUnit agent runs. By default, the port is 8080.

1. Build the project by going to Build > Build Solution, or use the CTRL-SHIFT-B shortcut. 2. Change the sauce_test.conf file to specify your project .dll and tests, and the browsers you want to run the tests against. 3. Add the .dll file of your project, sauce_test.conf, and any other referenced .dll files like the Selenium .DLL files, into the bin of the directory where you saved NUnit. 4. Open an Administrator command prompt. 5. Go to the NUnit > Bin directory. 6. Enter start pnunit-agent.exe agent.conf. This will launch the PNUnit agent and open up a new PNUnit agent command prompt. 7. In the command prompt, enter pnunit-launcher.exe sauce_test.conf. This will launch your tests to the Sauce Labs dashboard, where you will be able to see them run.

Page 143 Copyright Sauce Labs 2015

Page 144 Copyright Sauce Labs 2015

Running Tests in Parallel with Java

1. https://github.com/ndmanvar/SeleniumJavaJunit 2. https://github.com/ndmanvar/SeleniumJavaTestNG

Running Java Tests in Parallel with JUnit Running Java Tests in Parallel with TestNG

Page 145 Copyright Sauce Labs 2015

Running Java Tests in Parallel with JUnit

Tests can be run in parallel using JUnit, but it takes a bit of work. The Java helper library includes a Parallelized class that creates a dynamic thread pool that holds each thread that is running a test.

Parallelizing the WebDriverTest Class

In this example, we're parallelizing tests across different browsers on different platforms. Since testing an app in Firefox on Linux is independent of testing it in Chrome on Windows, we can safely run both tests in parallel. The static method browsersStrings() is annotated with com.sau celabs.junit.ConcurrentParameterized.Parameters, indicating it should be used to determine the parameters for each instance of the test. The method returns a LinkedList of parameters to use for each test instance's constructor. The SampleSauceTest constructor captures these parameters and setUp() uses them to configure the DesiredCapabilities.

Example of Java Test Parallelization Expand source @RunWith(ConcurrentParameterized.class) public class SampleSauceTest implements SauceOnDemandSessionIdProvider {

/** * Constructs a {@link SauceOnDemandAuthentication} instance using the supplied user name/access key. To use the authentication * supplied by environment variables or from an external file, use the no-arg {@link SauceOnDemandAuthentication} constructor. */ public SauceOnDemandAuthentication authentication = new SauceOnDemandAuthentication("${userName}", "${accessKey}");

/** * JUnit Rule which will mark the Sauce Job as passed/failed when the test succeeds or fails. */ @Rule public SauceOnDemandTestWatcher resultReportingTestWatcher = new SauceOnDemandTestWatcher(this, authentication);

/** * Represents the browser to be used as part of the test run. */ private String browser; /** * Represents the operating system to be used as part of the test run. */ private String os; /** * Represents the version of the browser to be used as part of the test run. */ private String version; /** * Instance variable which contains the Sauce Job Id. */ private String sessionId;

/** * The {@link WebDriver} instance which is used to perform browser interactions with. */ private WebDriver driver;

Page 146 Copyright Sauce Labs 2015

/** * Constructs a new instance of the test. The constructor requires three string parameters, which represent the operating * system, version and browser to be used when launching a Sauce VM. The order of the parameters should be the same * as that of the elements within the {@link #browsersStrings()} method. * @param os * @param version * @param browser */ public SampleSauceTest(String os, String version, String browser) { super(); this.os = os; this.version = version; this.browser = browser; }

/** * @return a LinkedList containing String arrays representing the browser combinations the test should be run against. The values * in the String array are used as part of the invocation of the test constructor */ @ConcurrentParameterized.Parameters public static LinkedList browsersStrings() { LinkedList browsers = new LinkedList(); browsers.add(new String[]{"Windows 8.1", "11", "internet explorer"}); browsers.add(new String[]{"OSX 10.8", "6", "safari"}); return browsers; }

/** * Constructs a new {@link RemoteWebDriver} instance which is configured to use the capabilities defined by the {@link #browser}, * {@link #version} and {@link #os} instance variables, and which is configured to run against ondemand.saucelabs.com, using * the username and access key populated by the {@link #authentication} instance. * * @throws Exception if an error occurs during the creation of the {@link RemoteWebDriver} instance. */ @Before public void setUp() throws Exception {

DesiredCapabilities capabilities = new DesiredCapabilities(); capabilities.setCapability(CapabilityType.BROWSER_NAME, browser); if (version != null) { capabilities.setCapability(CapabilityType.VERSION, version); } capabilities.setCapability(CapabilityType.PLATFORM, os); capabilities.setCapability("name", "Sauce Sample Test"); this.driver = new RemoteWebDriver( new URL("http://" + authentication.getUsername() + ":" + authentication.getAccessKey() + "@ondemand.saucelabs.com:80/wd/hub"), capabilities); this.sessionId = (((RemoteWebDriver) driver).getSessionId()).toString();

}

Page 147 Copyright Sauce Labs 2015

/** * Runs a simple test verifying the title of the amazon.com homepage. * @throws Exception */ @Test public void amazon() throws Exception { driver.get("http://www.amazon.com/"); assertEquals("Amazon.com: Online Shopping for Electronics, Apparel, Computers, Books, DVDs & more", driver.getTitle()); }

/** * Closes the {@link WebDriver} session. * * @throws Exception */ @After public void tearDown() throws Exception { driver.quit(); }

/** * * @return the value of the Sauce Job id. */ @Override public String getSessionId() {

Page 148 Copyright Sauce Labs 2015

return sessionId; } }

Setting a Parallelism Limit

To stop tests from timing out when you're already using all your Sauce Labs parallel slots, you need to limit the number of threads.

The Sauce Labs Parallelized JUnit runner uses the junit.parallel.threads System property to control how many threads it runs. You can set this by updating the in your Maven pom.xml file, as shown in the example. Similarly, if you were using Ant as your project comprehension tool, you would update the attribute in your Parallel task.

Example of Updating the Threadcount in a Maven pom.xml file Expand source maven-compiler-plugin 3.0 1.6 1.6 org.apache.maven.plugins maven-surefire-plugin 2.12.4 classes 20 true

Page 149 Copyright Sauce Labs 2015

Running Java Tests in Parallel with TestNG

TestNG has built-in support for running tests in parallel. All you need to do is set a parallelism limit in your Maven pom.xml file or your Ant Parallel task file by updating the .

Match Thread Count to Concurrency Limit You should match your thread count to your concurrency limit, which is shown in the My Account section of your user profile information on the Sauce Labs dashboard.

Example of Updating the in a Maven pom.xml file Expand source maven-compiler-plugin 3.0 1.6 1.6 org.apache.maven.plugins maven-surefire-plugin 2.12.4 classes 20 true

Page 150 Copyright Sauce Labs 2015

Running Tests in Parallel with PHP

Running PHP Tests in Parallel with PHPUnit and Paratest

Page 151 Copyright Sauce Labs 2015

Running PHP Tests in Parallel with PHPUnit and Paratest

Paratest is a command line tool for running PHPUnit that comes bundled with Sausage that you can use to run PHP tests in parallel on Sauce using the PHPUnit test framework.

These examples, for Mac OS X/Linux and Windows, show the commands used to specify the path to the test file you want to run, and to simultaneously run two instances of PHPUnit. In these examples, the path to the WebDriverDemo.php demo test is shown, while the -p 2 arg uments set the number of instances to run.

Mac OSX/Linux Command for Launching Parallel PHP Tests

vendor/bin/paratest -p 2 -f --phpunit=vendor/bin/phpunit WebDriverDemo.php

Windows Command for Launching Parallel PHP Tests

vendor\bin\paratest.bat -p 2 -f --phpunit=vendor\bin\phpunit.bat WebDriverDemo.php

Page 152 Copyright Sauce Labs 2015

Running Tests in Parallel with Python

Running Python Tests in Parallel with nose Running Python Tests in Parallel with pytest

Page 153 Copyright Sauce Labs 2015

Running Python Tests in Parallel with nose

This topic and the example code illustrate how you could set up Python tests to run in parallel using the nose testing framework. Check out the nose documentation for more information on using nose.

This is the command you would issue to Nose to run the tests:

nosetests --processes=8 --process-timeout=120

Page 154 Copyright Sauce Labs 2015

Example of Running Python Tests in Parallel with Nose Expand source import os import sys import inspect from nose.tools import with_setup from selenium import webdriver from sauceclient import SauceClient browsers = [{ "platform": "Windows 10", "browserName": "internet explorer", "version": "11" }, { "platform": "OS X 10.11", "browserName": "safari", "version": "8.1" }] username = 'YOUR_SAUCE_USERNAME' access_key = 'YOUR_SAUCE_ACCESSKEY' def launchBrowser(caps): caps['name'] = inspect.stack()[1][3] return webdriver.Remote( command_executor = "http://%s:%[email protected]:80/wd/hub" % (username, access_key), desired_capabilities = caps); def teardown_func(): global driver driver.quit() sauce_client = SauceClient(username, access_key) status = sys.exc_info() == (None, None, None) sauce_client.jobs.update_job(driver.session_id, passed=status)

# Will generate a test for each browser and os configuration def test_generator_verify_google(): for browser in browsers: yield verify_google, browser

@with_setup(None, teardown_func) def verify_google(browser): global driver driver = launchBrowser(browser) driver.get("http://www.google.com") assert ("Google" in driver.title), "Unable to load google page" elem = driver.find_element_by_name("q") elem.send_keys("Sauce Labs") elem.submit()

Page 155 Copyright Sauce Labs 2015

Running Python Tests in Parallel with pytest

This topic and the example code demonstrate how you could set up Python tests to run in parallel with the pytest testing framework. A decorator is used to iterate over the browsers and run the tests. Check out the pytest documentation for more detailed information on how to use pytest.

This is the command you would issue to pytest to execute the test:

py.test -n2 first_test.py

Example of Running Python Tests in Parallel with py.test Expand source import os import unittest import sys import new from selenium import webdriver from sauceclient import SauceClient

browsers = [{ "platform": "Windows 10", "browserName": "internet explorer", "version": "11" }, { "platform": "OS X 10.11", "browserName": "safari", "version": "8.1" }]

username = 'YOUR_SAUCE_USERNAME' access_key = 'YOUR_SAUCE_ACCESSKEY'

# This decorator is required to iterate over browsers def on_platforms(platforms): def decorator(base_class): module = sys.modules[base_class.__module__].__dict__ for i, platform in enumerate(platforms): d = dict(base_class.__dict__) d['desired_capabilities'] = platform name = "%s_%s" % (base_class.__name__, i + 1) module[name] = new.classobj(name, (base_class,), d) return decorator

@on_platforms(browsers) class FirstSampleTest(unittest.TestCase):

# setUp runs before each test case def setUp(self): self.desired_capabilities['name'] = self.id() self.driver = webdriver.Remote( command_executor="http://%s:%[email protected]:80/wd/hub" % (username, access_key), desired_capabilities=self.desired_capabilities)

# verify google title def test_google(self): self.driver.get("http://www.google.com") assert ("Google" in self.driver.title), "Unable to load google page"

Page 156 Copyright Sauce Labs 2015

# type 'Sauce Labs' into google search box and submit def test_google_search(self): self.driver.get("http://www.google.com") elem = self.driver.find_element_by_name("q") elem.send_keys("Sauce Labs") elem.submit()

# tearDown runs after each test case def tearDown(self): self.driver.quit()

Page 157 Copyright Sauce Labs 2015 sauce_client = SauceClient(username, access_key) status = (sys.exc_info() == (None, None, None)) sauce_client.jobs.update_job(self.driver.session_id, passed=status)

Page 158 Copyright Sauce Labs 2015

Running Tests in Parallel with Ruby

You can use the parallel_tests gem to run Ruby tests in parallel. See the documentation in the parallel_tests GitHub repo to get started.

Running Ruby Tests in Parallel with RSpec

Page 159 Copyright Sauce Labs 2015

Running Ruby Tests in Parallel with RSpec

1. Add the RSpec and ParallelTests gems into your Gemfile, then run bundle install.

gem "rspec", "~> 3.0.0" gem "parallel_tests", "~> 1.6.1

2. Create a spec directory in your project's root directory. 3. In the spec directory, create a file called sauce_driver.rb, which will encapsulate the behaviour we need to create Selenium drivers with Sauce Labs.

Example sauce_driver.rb File for Running Ruby Tests in Parallel Expand source on Sauce require "selenium/webdriver"

module SauceDriver class << self def sauce_endpoint

"http://YOUR_SAUCE_USERNAME:[email protected]:80/wd/hub " end

def caps caps = { :platform => "Mac OS X 10.9", :browserName => "Chrome", :version => "31" } end

def new_driver Selenium::WebDriver.for :remote, :url => sauce_endpoint, :desired_capabilities => caps end end end

Now you need to make RSpec create a new driver for each spec. The cleanest way to do this is by defining an around hook, which will be run, naturally, around every spec. 4. Create a file in the spec directory called spec_helper.rb. This file will be require `d by every spec you create, as this is where, by convention, any setup code is placed.

Page 160 4.

Example spec_helper.rb File for Running Ruby Tests in Parallel Expand source with Spec require "rspec" require_relative "sauce_driver"

RSpec.configure do |config| config.around(:example, :run_on_sauce => true) do |example| @driver = SauceDriver.new_driver begin example.run ensure @driver.quit end end end

On line 5 you set up a new block of code to be run around every example tagged with :run_on_sauce => true. This lets you have non-Selenium tests that don't spin up Selenium sessions. You have to make sure you include :run_on_sauce on all your Selenium tests, though!

On line 6 you use the SauceDriver class you set up earlier to create a new Selenium driver, 'pointed' at Sauce Labs. Then, on line 8, you run the example. On lines 9 - 11 we call quit on the Selenium session; This is done in an ensure block so that each test always closes off its driver, even if something goes wrong.

Now you have RSpec set up to create drivers and close them down. Your next question might be, "How do I use these drivers?" Super simple. Because driver is an instance variable, and the around block runs in the context of the spec, it can use the driver directly. Consider this example spec.

google_spec.rb Expand source require_relative "spec_helper"

describe "Google's Search Functionality" do it "can find search results", :run_on_sauce => true do @driver.manage.timeouts.implicit_wait = 10 @driver.navigate.to "http://www.google.com"

raise "Unable to load Google." unless @driver.title.include? "Google"

query = @driver.find_element :name, "q" query.send_keys "Sauce Labs" query.submit

puts @driver.title end end

On line 1 you require spec_helper. Then, on line 4 you add :run_on_sauce => true to make sure the around block runs. You can use the created driver, @driver, without any further setup, and you don't have to do anything to close it off either. This means you're much less likely to forget to do so when you write more tests.

Now, parallel testing! If you go ahead and create some more specs like this one (you can copy google_spec.rb into other files in the spec dire ctory, just make sure the filenames end in spec.rb), then run the specs from your project root directory with:

Page 161 Copyright Sauce Labs 2015

bundle exec parallel_rspec -n 2 spec/

You should be able to log into Sauce Labs and see your tests running in parallel. If your account has more than two concurrency slots (meaning, you have a paid account), you can increase the number after -n to match your concurrency, and parallel_tests will spin up more tests at once.

Page 162 Copyright Sauce Labs 2015

Troubleshooting Parallel Tests on Sauce Labs

These are good, general practices for investigating problems when parallelising your test suite. Further investigations and assistance will need to come from your own company and language community, because parallelization is a complex topic and we can't hope to cover everything. This topic is specifically aimed at covering issues with Selenium when running tests in parallel. The Selenium protocol is pretty simple, as is Sauce Labs' integration with it: if you send us a correct Selenium command, we'll obey it. Our service doesn't take any action that isn't driven by your test suite, so if we are sure Selenium is working correctly, then it's highly likely something other than Sauce Labs is causing issues.

Add Logging Why log instead of using trace tools? How should I do my logging? What should I actually log? Check How Much Parallelisation Your Tests Can Stand Run Fewer Tests Read Your Logs Incorrect driver setup Non-thread-safe code Selenium Lifecycle Problems Problem Still Unsolved

Add Logging

Why log instead of using trace tools?

Many languages have tools for tracing through code as it runs, monitoring what lines and classes are currently executing. These can be extremely useful for debugging code problems, provided you're already familiar with them. Because many people aren't familiar with these tools, we recommend you start by adding logging to your tests. Lots of logging. All the Logging. Logging also makes it easier to reason about multiple threads over a period of time.

How should I do my logging?

You can use your language or test framework's logging tools, your container's logging tools, or even printing to standard out. Printing to standard out is a brute force approach, but can often be the best one to use when trying to debug problems on a continuous integration system, where logging framework output may be hard to access.

Regardless of how you do your logging, you should always include:

1. Timestamps. 2. ID of the current thread or process (where applicable). 3. ID of the Selenium driver currently in use (where applicable).

A good template to log with is:

[TIMESTAMP] - [PROCESS ID]: [DRIVER ID] - [Log Message]

Where Driver ID is only present if relevant.

If you're logging using a tool that allows for log levels, log to the DEBUG or VERBOSE levels of logging.

What should I actually log?

The purpose of logging is to inform you of concurrency problems that are occurring. So, you should log everything relevant to the problem you're seeing. At a minimum, you should add logging to show:

Just before creating a Selenium driver, with the desired capabilities When the driver is created successfully, with its session ID When a driver is going to be quit, with its session ID When a driver has been successfully quit, with its session ID

Other logging required depends on your specific problem

When a test starts When a test finishes When a test is first about to use a Selenium driver Every time a test is using a Selenium driver

Page 163 Copyright Sauce Labs 2015

When a test is about to do something that is network intensive for the browser When test exceptions occur When driver setup exceptions occur When any pre-test actions that set up data or browser state, are about to run When any pre-test actions that set up data or browser state have run When any post-test actions that clean up data or browser state have run When any post-test actions that clean up data or browser state have run

You should always include test names when talking about specific tests, Session IDs when using Selenium drivers, and Thread or Process IDs.

You should also try, whenever possible, to improve the error messages your test suite throws when problems occur.

Check How Much Parallelisation Your Tests Can Stand

You should also check to see if there's a level of parallelization at which your tests work, and one where. Some Sauce Labs users have problems only once they've exceeded a certain number of parallel tests at once. For instance, problems with queuing and network congestion are often exposed when running higher numbers of parallel tests.

First up, check how many parallel tests your account is able to run simultaneously. You can check this by logging in to Sauce Labs. In the left-hand column you'll see the number of Concurrent VMs allowed for your account, which corresponds to the number of tests you can run in parallel. Make sure you're not exceeding this number, and if you are, simply throttle back on the number of parallel tests you're running until the problem resolves.

If your problem continues try slowly lowering the number of tests and see if there's a level, higher than one, at which you're no longer experiencing problems. If you find the level at which this occurs, then you can start investigating your logging to see what your tests do at higher levels, that differs from lower ones. For instance, you might discover that some of your tests rely on running in browsers with other tests, and when your parallelisation goes higher, this no longer happens.

If there's no level of parallelisation where your tests work correctly in parallel, or there is but it's not related to your Sauce Labs concurrency limit a nd you're certain it's not a networking problem, leave your parallelisation at 2 and carry on diagnosis.

Run Fewer Tests

Take tests out of your test suite! Similar to the steps above, removing tests from your test suite can surface problematic tests. It also shows if you have a problem with the length of individual thread lifecycles.

If there's a point where you remove some tests and things start functioning, try adding some of the tests you removed earlier back in, and see if they're not working correctly also. If they are, you've identified a problematic test you can check out further.

If you don't experience more stable tests by removing a couple of unstable ones, or if every test above a certain number causes issues, configure your tests to run the lowest number of tests that exhibits a problem, and keep debugging.

Read Your Logs

Time to start cross-checking the logs.

Incorrect driver setup

"Empty" or incomplete desired capabilities "Empty" or missing user authentication

Non-thread-safe code

Selenium session IDs changing during a single test Selenium session IDs being used across threads Tests using Selenium sessions that have already been quit Tests using Selenium sessions which don't exist

Selenium Lifecycle Problems

Selenium sessions being created all at once, but only used by tests much later Selenium sessions all quit at once Selenium session IDs being used across multiple threads

Page 164 Copyright Sauce Labs 2015

Selenium sessions being created and never quit Tests using Selenium sessions that have already been quit Tests using Selenium sessions which don't exist

Problem Still Unsolved

Unfortunately, if all of your tests get unique Selenium sessions created, those sessions are started and stopped correctly, are only ever used by a single thread, and don't get queued, then its likely your problems arise from something else in your test suite. Unfortunately, these are out of scope of Sauce Labs' support. It can be helpful to investigate your database and application server's ability to deal with concurrent tests, shared sessions, and test data. Good Luck!

Page 165 Copyright Sauce Labs 2015

Viewing and Managing Sauce Labs Test Results

Once you've run your test, you'll see the results on your Sauce Labs dashboard. From there you can watch a video of the test as it ran, review the commands that were issued, read the test log, and view the metadata associated with the test. You can also manage the results of past tests and builds on the Archives page, and use status badges on your code repository or a web page to keep track of your latest test runs.

Viewing Screenshots, Commands, Logs, and Metadata for Sauce Test Results Using Status Badges and the Browser Matrix Widget to Monitor Test Results Sharing the Results of Sauce Labs Tests Embedding Test Results in HTML Pages Building Links to Test Results Managing Archived Test and Build Results Searching for Test Results and Builds on Your Archive Page Search Fields and Operators Using Favorites to Save Your Searches Deleting Test and Build Results Changing the Layout of Your Archives Page

Page 166 Copyright Sauce Labs 2015

Viewing Screenshots, Commands, Logs, and Metadata for Sauce Test Results

1. Log in to Sauce Labs. 2. Click Dashboard. 3. Select Automated Tests or Manual Tests, depending on the type of test result you want to view. 4. Click the name of the test with the results you want to view. 5. Click Watch to view screenshots and video of the test. 6. Click Commands to see the API commands that were issued during the test. The Commands tab also includes controls for filtering the commands displayed and playing back the commands. 7. Click Logs to see the logs generated by your test. The kinds of logs you can see are determined by the type of test you ran. For example, Web application tests will include a Selenium log, while mobile application tests will contain an Appium log. 8. Click Metadata to see the metadata attached to your test. Some of the test metadata, such as Passed, can be updated via the the Sauce REST API. You can also download video and screenshots of your test in this tab.

Mobile Testing Logs The Appium Log tab in your test indicates that this test ran using the Appium driver. If you take a look at the Appium Log, you will notice that the first line of the log provides information about the Appium version used during your test , for example info: Welcome to Appium v1.4.0.

For iOS tests, you'll notice that the iOS simulator log is embedded within the Appium log. The information from the iOS simulator is grayed out throughout the Appium log and has the tag name: info: [IOS_SYSLOG_ROW ].

For Android tests you can find the Android emulator logs by clicking the Metadata tab in your test page and searching for the Logcat .log file. This file contains all the information from the Android emulator log.

Page 167 Copyright Sauce Labs 2015

Using Status Badges and the Browser Matrix Widget to Monitor Test Results

Status badges and the Sauce Labs browser matrix widget help you keep track of the status of your latest test runs. All you have to do is add either markdown or HTML code to your GitHub README or project site that references your Sauce Labs username and access key, and annotat e your jobs with the REST or WebDriver API.

Status Badges and the Browser Matrix Setting Up Status Badges for Test Results Setting Up the Browser Matrix Widget Status Images for Private Accounts

Status Badges and the Browser Matrix

There are three status badges that correspond to the three states of a finished test: Passing, Failed, and Unknown.

Badge Status

Passing

Failed

Unknown - used for security reasons when a test doesn't have a status of Passing or Failed

With the browser matrix, you can keep track of the test status of your project for various browser/platform/operating system combinations.

Setting Up Status Badges for Test Results

1. Copy this markdown code into your GitHub README.

[![Sauce Test Status](https://saucelabs.com/buildstatus/YOUR_SAUCE_USERNAME)](https://saucelabs .com/u/YOUR_SAUCE_USERNAME)

Alternatively, you can use this HTML code on your project site.

Page 168 1.

Multiple Projects, Multiple Accounts If you just have one project, you can use your main Sauce account name. If you have multiple projects, you will want to create a sub-account for each project.

2. Run your tests. 3. Make sure to set a build number, a pass/fail status, and a public, share, or public restricted visibility for every test that runs with the test configuration API. You'll know that these are set correctly if your tests have a status of Pass or Failed instead of Finished on your dashboard, and that a build number is also displayed.

Setting Up the Browser Matrix Widget

Copy this markdown code into your project's GitHub README.

[![Sauce Test Status](https://saucelabs.com/browser-matrix/YOUR_SAUCE_USERNAME.svg)](https://saucela bs.com/u/YOUR_SAUCE_USERNAME)

Alternatively, you can add this HTML to your project site.

Status Images for Private Accounts

If you want to display the build status of a private Sauce account, you need to provide a Hash-based Message Authentication Code (HMAC) token generated from your username and access key.

This example shows how to generate an HMAC token using Python:

python from hashlib import md5 import hmac "?auth=" + hmac.new("YOUR_SAUCE_USERNAME:YOUR_SAUCE_ACCESSKEY", None, md5).hexdigest()

Once you have the token, append it to the badge URL:

https://saucelabs.com/u/YOUR_SAUCE_USERNAME?auth=HMAC_TOKEN

Page 169 Copyright Sauce Labs 2015

Sharing the Results of Sauce Labs Tests

Once your test has run and generated a Test Details page, you have several options for sharing that page with others. At the top of the test details page is a menu with sharing options. The default selection in this menu is Team.

1. Select a sharing option for the Test Details page. When you select an option, you'll see the message Visibility Changed.

Sharing Description Option

Public Setting the sharing option for your Test Details page to Public means that it is accessible to everyone who has the link to the page, and may be listed on public web pages and indexed by search engines.

Public If you want to share your Test Details page and allow access to the test video and screenshots, but restrict log access to Restricted yourself, select Public Restricted. By restricting access to the raw Selenium log and the job log, you can prevent sensitive information such as passwords from being visible to others.

Private The Test Details page is visible only to your account.

Team The Test Details page is visible to your account and its sub accounts.

2. Copy and send the URL of the Test Details page to anyone you want to share it with.

You can also set the sharing options for a test through the REST API.

Page 170 Copyright Sauce Labs 2015

Embedding Test Results in HTML Pages

Embedding Full Test Pages Embedding the Video Player

Embedding Full Test Pages

You can embed test pages in CI test results or other test reports. Using the format below, add the HTML to any page where you need to embed test results, replacing YOUR_JOB_ID with the ID of the job you want:

Embedding the Video Player

In addition to full test results, you can embed videos as well. Using the format below, add the HTML to any page where you want to embed job videos, replacing YOUR_JOB_ID with the ID of the job you want:

Authentication Required Both of these configurations will only work for browsers logged in using your account, but you can use authentication tokens to make this work for anonymous viewers. Check out the section on creating authentication tokens in the topic Building Links to Test Results for more information.

Building Links to Test Results

There are three types of links you can create to provide access to your tests.

Links to Jobs that Require a Login to View Links to Jobs that Don't Require a Login to View Temporary Links to Jobs

Links to Jobs that Require a Login to View

In Selenium, when a client requests a new browser session, the server returns a session ID, which is used to identify that session throughout the test. The session ID is stored as a member variable of the instantiated Selenium object and named sessionId or session_id, depending on the client library. Sauce uses that session ID as the job id for accessing test results.

To directly access a specific job, you will first need to note the session ID locally, usually by writing it to a log file. You can then use it to create a URL with the following format and replace YOUR_JOB_ID with the session ID.

http://saucelabs.com/jobs/YOUR_JOB_ID

Notice that links to jobs in this format will only work if you are logged in with the account that ran the job or if that account is a sub-account of yours. For generating public links, see the next section.

Links to Jobs that Don't Require a Login to View

The links you create to your jobs in can be constructed in a way that doesn't require anonymous viewers to login and use your credentials. This is based on authentication tokens.

Auth tokens are generated on a per-job basis and give the receiver access using an hmac-based algorithm. You can also find hmac implementations for different programming languages.

The digest algorithm to use is MD5. The message and key used to generate the token should be the following:

Key: sauceUsername:sauceAccessKey Message: job-id

Here's an example in Python for generating the token for a job with id: 5f9fef27854ca50a3c132ce331cb6034

import hmac from hashlib import md5 hmac.new("YOUR_SAUCE_USERNAMEE:YOUR_ACCESS_KEY", "5f9fef27854ca50a3c132ce331cb6034", md5).hexdigest()

Once the auth token has been obtained, you can use it to build a link in this format: https://saucelabs.com/jobs/YOUR_JOB_ID?auth=AUTH_TOK EN

Temporary Links to Jobs

You can extend the links generated with authentication tokens to make them work for either one day or one hour duration, based on parameters that you set. During the hmac generation, set the key like this:

Key: YOUR_USERNAME:YOUR_ACCESS_KEY:YOUR_DATE_RANGE

The date range can take two formats: YYYY-MM-DD-HH and YYYY-MM-DD. These should be set in UTC time and will only work during the date or hour you set.

Managing Archived Test and Build Results

The Sauce Labs Archives page provides a handy way to manage historical information about your tests and builds. The Archives page contains all the tests and builds you have run since you opened your Sauce account, and new tests and builds are automatically added to it, while the Das hboard only displays the the last 50 builds or the last 25 tests you have run.

You can filter your archived test results based on criteria such as build, owner, browser, platform, status, date, and user-defined tags, create structured searches using multiple filters, and search using freeform text. You can also save your search queries as favorites, so you'll always be be able to find the results that are most important to you!

Searching for Test Results and Builds on Your Archive Page Search Fields and Operators Using Favorites to Save Your Searches Deleting Test and Build Results Changing the Layout of Your Archives Page

Searching for Test Results and Builds on Your Archive Page

Basic Filtering Creating Structured Searches Building Structured Searches from Filters Writing Structured Searches Based on Filters Examples of Structured Searches

The Archives page includes a number of filters you can use to search through your tests and builds. You can search using a single filter, or you can use multiple filters to build a structured search.

Basic Filtering

1. Click Archives. 2. Click the filter you want to use to open it. 3. In the filter text field, enter the terms you want to search for. As you type, autocomplete will suggest existing results that match your search terms. If you want to select a date or date range, use the Calendar icon in the Date filter. 4. When you find the items you want to use in your filter, select them. 5. When you've finished selecting your filter criteria, click Apply.

Special Filter Criteria and Operators Some filters include additional options and the ability to use special operators. See Search Fields and Operators for more information.

Creating Structured Searches

You have two options for creating structured searches.

Building Structured Searches from Filters

With this option, you would select filters and filter criteria as you would for basic filtering, but each time you make a selection, you will see it added to the Search field to build the query. Using this option, there is an implied AND between the different filters you select, and an implied OR between the values within a specific filter.

Writing Structured Searches Based on Filters

If you want to create a more complex search using AND, OR, or special operators associated with a specific filter, you can write your own structured search in the Search field. As you enter search criteria, the autocomplete will suggest values, operators, and filters to match your text. If your search query syntax is incorrect, your query text will turn orange, and you will see suggestions for how to correct it below the search field. When your syntax is correct, the text will turn blue.

Tips for Writing Structured Searches Strings must always be enclosed with quotation marks, but criteria values that are supplied by the application (for example, st atus:failed) do not. If you want to search using text only, enter text: in the search field, and then enter the text you want to search for enclosed in quotation marks. You can use * as a wildcard in any of your filter criteria. Use quotation marks around a string to search for that exact string. For example, "ruby test" will return "ruby test 1" and "ruby test 2," but not "ruby example test." Use parentheses around a string to find those terms anywhere in the search field. For example, (ruby test) will return both "ruby test 1" and "ruby example test."

Examples of Structured Searches

Search Description Structured Search

Find all of the tests that were tagged as Selenium_194 or Selenium_ tag:(Selenium_194 Selenium_193) and status:failed 193 that have failed since 03/10/2015 and date:[2015-10-03 TO *]

Find only the IE tests that have failed with the word Main in the title. name:"Main*" AND browser:"Internet Explorer *"

Search Fields and Operators

You can use any of these filters singly or or combination to search through the tests and builds on your Archive page. The Example column shows how you could construct a search using a specific filter in the Search text field. See Searching for Test Results and Builds on Your Archive Page for examples of how to build structured searches using multiple filters in the Search field.

FILTER DESCRIPTION EXAMPLE

Text Search for any mention of the string across test details. text: Appium

Name Search for full or partial test name. name: SauceTest

Platform Search for tests that ran on one or multiple operating systems. This field only accepts platform:("OS X 10.10" "Windows operating systems currently supported by Sauce Labs. 8.1")

Browser Search for tests that ran on one or multiple browsers. Only accepts browsers currently browser:("Google Chrome 43" supported by Sauce Labs. "Internet Explorer 11")

Date Search for tests that ran on a specific date or over a specified range. Dates should be date:[2014-05-05 TO 2015-05-05] in YYYY-MM-DD format. date: [2014-05-05 TO *]

Status Search for tests based on their status. Currently there are four possible states: failed, status: failed passed, complete, error.

Build Search for tests that belong to a specific build. build:main and browser:"Internet Explorer 11"

Tag Search for tests that have one or multiple tags. tag: Jenkins

Owner Search for tests that were created by a specific user. owner: admin

Using Favorites to Save Your Searches

If you've created a search that you want to use in the future, you can save it by adding it to your favorites.

1. After you've built your search from the filters or written it in the Search text field on the Archives page, click the Star icon next to the text field to save it. 2. Click the expand icon next to the Star to view your favorited searches. You can select a favorite search to run it, or remove it by clicking the Delete icon.

Deleting Test and Build Results

You can delete any of the tests or builds listed on your Archives page.

1. Select the test or build result you want to delete. You can also select multiple test or build results for deletion. 2. Click Delete. 3. In the confirmation dialog, click Yes.

Results are Permanently Deleted Once you delete a test or build result, it's gone forever and cannot be recovered. Be careful when you click Yes.

Changing the Layout of Your Archives Page

You can change the layout of your Archives page by changing which columns are displayed, and how the content in those columns is sorted.

1. On your Archives page, click Show Columns. 2. Select the columns you want to display, or clear the selection of columns you want to remove. 3. Click Apply. 4. Click Sort By to change the display of your results based on search score or ascending or descending dates.

Your layout changes will be saved until you change them again.

Using Sauce Storage for Test Assets

What is Sauce Storage?

Sauce Storage lets you securely store apps for use in your tests. Files are uploaded and stored on our internal network, and access is restricted to users in your account hierarchy. They stick around for seven days and are then deleted.

Why Use Sauce Storage?

Your tests will start faster, as files are served across our internal network You don't have to expose your app to the wider internet You don't have to create a hosting solution for your test applications

How do I Host Apps in Sauce Storage?

Upload your app using the REST API

You upload apps using the upload_file API endpoint. You can use any REST client, one of the Sauce REST API libraries, or cURL. Check out Temporary Storage Methods for more information.

OS X/Linux Example of Using curl to Upload an App to Sauce Storage

curl -u : -X POST -H "Content-Type: application/octet-stream" https://saucelabs.com/rest/v1/storage//?overwrite=tru e --data-binary @/

Windows Example of Using curl to Upload an App to Sauce Storage

ccurl -u : -X POST -H "Content-Type: application/octet-stream" https://saucelabs.com/rest/v1/storage//?overwrite=tru e --data-binary @/

Parameter Description

sauce_username Your Sauce Labs username

sauce_access_key Your Sauce Labs access key

upload_filename The name you want to store the file under on Sauce Storage, for example SuperApp.zip

path/to/your_file_name The path to where the file is saved on your local machine, for example users/Tester/builds/super_app _tst_1228_build.zip

If you're using curl for the upload, you must include the @ before path/to/your_file_name

Set Your Desired Capabilities to Use Sauce Storage

Sauce Storage URLs are in the form sauce-storage:. After you're uploaded the file, you can replace the value of the ap p desired capability with the Sauce Storage URL.

Ruby Example for Setting the App Desired Capability

caps = Selenium::WebDriver::Remote::Capabilities.iphone caps['appiumVersion'] = '1.4.0' caps['deviceName'] = 'iPhone Simulator' caps['deviceOrientation'] = 'portrait' caps['platformVersion'] = '8.2' caps['platformName'] = 'iOS' caps['browserName'] = '' caps['app'] = 'sauce-storage:SuperApp.zip'

Sauce Storage URL Filename The filename you use in your sauce-storage URL is not the filename your file is saved as locally, but whatever value you set for upload _filename as part of your REST API command. For instance, in this example:

https://saucelabs.com/rest/v1/storage/someuser/SuperApp.zip?overwrite=true --data-binary @/users/Tester/builds/super_app_tst_1228_build.zip

the correct Sauce Storage URL is sauce-storage:SuperApp.zip, not sauce-storage:super_app_tst_1228_build.zip.

How Often Should I Upload my App?

Files stay in Sauce Storage for seven days, after which time they're erased. So, you should upload your app every 7 days, or whenever the app is changed.

If you're running tests against your app as part of your development process and continuous integration, you should upload your app at the start of every test run to ensure any changes to the app are accounted for. This is usually best accomplished automatically as part of your test suite, using a REST client.

Uploading Assets to Sauce Storage Using C#

This code sample shows how to upload test assets to Sauce Temporary Storage over the REST API using C# and the RestSharp library.

using RestSharp; using RestSharp.Authenticators; using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Threading.Tasks;

namespace ConsoleApplication3 { class Program { // Uploads a file to Sauce Temporary Storage via REST API using the // RestSharp library. Be sure to replace the string values for // FileName, FilePath, UserName, and AccessKey.

static void Main(string[] args) { string FileName = ""; // ex: "NotesList.apk" string FilePath = ""; // ex: "C:\\Users\\Sauce11\\Downloads\\NotesList.apk"; string UserName = ""; // ex: "saucetester" string AccessKey = ""; // ex: "123123-abcd-1234-abcd-abc123abc123";

var client = new RestClient("https://saucelabs.com/rest/v1/"); client.Authenticator = new HttpBasicAuthenticator(UserName, AccessKey); var request = new RestRequest("storage/"+UserName+"/"+FileName+"?overwrite=true", Method.POST); request.AddHeader("Content-Type", "application/octet-stream"); request.AddFile(FileName, FilePath); var result = client.Execute(request); Console.WriteLine(result.Content); } } }

Using Sauce Connect for Testing Behind the Firewall or on Localhost

Sauce Connect is a tunneling app that you set up in your local network that opens a secure connection between a Sauce Labs virtual machine running your browser tests, and an application or website you want to test that's on your local machine or behind a corporate firewall. Sauce Connect is not required to run tests with Sauce Labs, but only in situations where the website or application you want to test is not publicly accessible.

In addition to providing a means for Sauce Labs to access your application or website, Sauce Connect has some other uses in your testing network architecture:

As an alternativee to whitelisting As a means of monitoring upstream traffic through a proxy like BrowserMob As a way to stabilize network connections (for instance, detecting/re-sending dropped packets)

Topics in this section cover the setup and use of Sauce Connect, as well as topics on how Sauce Connect works within a network architecture.

Sauce Connect in the Network Sauce Connect Setup and Teardown Process Sauce Connect and Security Sauce Connect Certificate Handling Sauce Connect Network Configurations Improving Sauce Connect Network Performance Basic Sauce Connect Network Configuration Dysfunctional Sauce Connect Network Configurations Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration Sauce Connect Setup and Configuration System Requirements for Sauce Connect Setting Up Sauce Connect Monitoring Sauce Connect with a Service Management Tool Using Multiple Sauce Connect Tunnels Sauce Connect Proxy Configuration Sauce Connect Command Line Reference Sauce Connect FAQS Troubleshooting Sauce Connect Sauce Connect Best Practices

Sauce Connect in the Network

Sauce Connect IP and Virtual Machine Range Sauce Connect uses the IP range 162.222.72.0/21, and the tunnel VM ranges 162.222.74.0/24, 162.222.75.0/24, 162.22 2.76.0/24, 162.222.77.0/24, 162.222.78.0/24 and 162.222.79.0/24.

For saucelabs.com certificate authentication, the server hosting Sauce Connect may need to connect to Online Certificate Status Protocol (OCSP) or Certificate Revocation List (CRL) services as well. Check out Sauce Connect Certificate Handling for more info.

Topics in this section describe how Sauce Connect functions in a network, and provide some examples of different network configurations using Sauce Connect as an alternative to whitelisting, connecting to proxied sites, or monitoring traffic through a proxy like BrowserMob.

Sauce Connect Setup and Teardown Process Sauce Connect and Security Sauce Connect Certificate Handling Sauce Connect Network Configurations Improving Sauce Connect Network Performance Basic Sauce Connect Network Configuration Dysfunctional Sauce Connect Network Configurations Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration

Sauce Connect Setup and Teardown Process

Sauce Connect Setup Process Sauce Connect Teardown Process

Sauce Connect Setup Process

During startup, Sauce Connect issues a series of HTTPS requests to the Sauce Labs REST API. These are outbound connections to saucelabs .com on port 443. Using the REST API, Sauce Connect checks for updates and other running Sauce Connect sessions, and ultimately launches a remote tunnel endpoint virtual machine (VM). Once the VM is started, a tunnel connection is established to a makiXXXXX.miso.saucelabs. com address on port 443, and all traffic between Sauce Labs and Sauce Connect is then multiplexed over this single encrypted TLS connection.

1. Sauce Connect makes HTTPS REST API calls to saucelabs.com:443 using the username and access key provided when starting Sauce Connect. 2. Sauce Labs creates a dedicated virtual machine that will serve as the endpoint of the tunnel connection created by Sauce Connect. 3. Sauce Labs responds with the unique ID of the virtual machine created in step 2. 4. Sauce Connect establishes a TLS connection directly to the dedicated virtual machine created in step 2. (makiXXXXX.miso.saucelab s.com). 5. All test traffic is multiplexed over the tunnel connection established in step 4.

Sauce Connect Teardown Process

Once Sauce Connect is terminated (typically via ctrl-c), a call will be made from Sauce Connect to the REST API with instructions to terminate the tunnel VM. Sauce Connect will continue to poll the REST API until the tunnel VM has been halted and deleted.

Sauce Connect and Security

Though it may take a few seconds to start up a Sauce Connect tunnel, once it's up you have the highest possible security for communications between the machine where it's running and the Sauce Labs API and browser cloud. In addition to the security of the tunnel itself, each tunnel connection spins up a fresh virtual machine that is used only for your tests, and which is destroyed when the tunnel is closed. This is why one of the recommended best practices for Sauce Connect is to always create a new tunnel for each test suite or build, and then tear it down at the end of your test.

Data transmitted by Sauce Connect is encrypted through the TLS protocol, using the AES-256 cipher. Sauce Connect also uses a caching web proxy to minimize data transfer. You can disable this with the command line option -B, --no-ssl-bump-domains, which is described further in the Sauce Connect Command Line Reference.

Sauce Connect in the DMZ Within your infrastructure, Sauce Connect must be on the same network as the application or network you want to test, but can be firewalled from the rest of your internal network. We recommend running Sauce Connect in a firewall DMZ, on a dedicated machine, and setting up firewall rules to restrict access from that DMZ to your internal network. However, you must be careful about how to locate Sauce Connect in a DMZ. The topics in Sauce Connect Network Configurations include examples of various ways to set up Sauce Connect, including a dysfunctional DMZ architecture.

Sauce Connect Certificate Handling

The security of Sauce Connect connections to both the Sauce Labs API and the virtual machine hosting your tests in the Sauce Labs cloud is managed through public key certificates. For connection to the API, Sauce Connect uses certificates issued by certificate authorities, and which are are integrated into the operating system of the machine where Sauce Connect is running. For connection to the Sauce Labs virtual machines, Sauce Connect uses a self-signed certificate that is part of the application itself.

Connecting to the Sauce Labs REST API Linux Windows OS X Tunnel Connection to the Sauce Labs Virtual Machine over SSL/TLS

Connecting to the Sauce Labs REST API

Connections to the Sauce Labs API go through https://saucelabs.com. The way in which Sauce Connect is able to access the certificates to secure the connection depends on the operating system of the machine where Sauce Connect is installed.

Linux

On Linux machines, Sauce Connect will look for the directory where the certificate bundle is installed, typically something like /etc/ssl/certs. If it can't find the directory, it will generate an error when trying to connect to the Sauce Labs API.

Windows

On Windows machines, certificates are managed through the Security Support Provider Interface API over SChannel, which requires access to the OCSP and CRL URLs to verify certificates. If you have set up highly restrictive firewalls or proxies on the machine where Sauce Connect is running and it can't connect to these URLs, you'll get an error when attempting to connect to the Sauce Labs API.

OS X

On OS X machines, certificates are pre-installed as part of the Trust Store and are accessible through the Keychain. If Sauce Connect is installed on an OS X machine, no troubleshooting should be necessary as long as it can access the Keychain.

Tunnel Connection to the Sauce Labs Virtual Machine over SSL/TLS

Connections from Sauce Connect to the virtual machine that run your tests on browsers in the Sauce Labs cloud are managed through the SSL/T LS protocol, and a Sauce Labs self-signed certificate that is included in the application.

Sauce Connect Network Configurations

Topics in this section illustrate various network configurations for Sauce Connect.

Improving Sauce Connect Network Performance Basic Sauce Connect Network Configuration Dysfunctional Sauce Connect Network Configurations Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration

Improving Sauce Connect Network Performance

If you're trying to improve the performance of Sauce Connect on your network, there are a couple approaches you can try with Sauce Connect command line options. These include -D, --direct-domains, -t, --tunnel-domains, and -F, --fast-fail-regexps. These allow for careful curating of which traffic will go through the tunnel and which will go directly to the internet.

Only Route Needed Requests through Sauce Connect Whitelist Domains Blacklist Domains Drop Analytics and Ad-based Requests

Only Route Needed Requests through Sauce Connect

Whitelist Domains

A common use case for this is if you need your requests to your internal environment to route through Sauce Connect, with external resources being pulled as usual.

Set the -t flag to whitelist a domain:

-t internal-site.com

Blacklist Domains

In this case, the use case is that you need most things to go through the tunnel, but certain external assets must be retrieved directly, for example from a Content Delivery Network (CDN).

Set the -D flag to blacklist a domain:

-D cdn.external-site.com

Drop Analytics and Ad-based Requests

You may not access to some external assets at all, for the sake of speed or just not interfering with user metrics, such as analytics.

Set the the -F flag to fail a domain:

-F analytics-provider.com

Basic Sauce Connect Network Configuration

This the basic configuration for Sauce Connect. The SC Host has a direct connection to the internet, and the SUT is on the same local network as the SC Host machine. In this configuration, if you're unable to connect, the most likely culprit are the firewall settings on the SC Host machine.

Shorthand Longhand

SC Host Machine in your network that the Sauce Connect application is running on.

SUT Site Under Test, the site that you're testing.

Sauce SC The virtual machine that hosts Sauce Connect on the Sauce Labs side. The configuration is more complicated than just one Host machine, but for representational purposes it's treated as a single machine.

Dysfunctional Sauce Connect Network Configurations

Dysfunctional Geographic Domain Configuration Dysfunctional DMZ Site Under Test (SUT) Configuration

Dysfunctional Geographic Domain Configuration

This is a good example of a dysfunctional setup with Sauce Connect. The problem is that the SC host is in the same VPN, and thus the same internal network, as the Site Under Test (SUT), but in geographically separate locations. This means that requests from Sauce Connect to the SUT need to reach back through the internet to be completed, rather than over the same internal network. An example this network configuration would be an SC Host in Berlin, with the SUT located in a data center in Chicago. This would require a number of network hops, which delays communication with the test virtual machine at Sauce Labs. The way to prevent this is to always place the SC Host in the same geographic domain as the SUT.

Dysfunctional DMZ Site Under Test (SUT) Configuration

Here, the SUT is in a network DMZ or Demilitarised Zone. It's exposed to the internet but isolated from the internal network.

Shorthand Longhand

SC Host Machine in your network that the Sauce Connect application is running on.

SUT Site Under Test, the site that you're testing.

Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration

Internet Control/Transaction Monitoring with a Proxy Configuration Proxied Site Under Test (SUT)

You can find information on how to configure Sauce Connect to work with proxies, either through autodetection of the proxies already set up on the SC Host, or through command line configuration, in the topic Sauce Connect Proxy Configuration. This topic will show you some of the basic ways to set up Sauce Connect within a proxied or firewalled network architecture.

Internet Control/Transaction Monitoring with a Proxy Configuration

You may have a network configuration in which you use a proxy to control the flow of traffic from your internal network to the external Internet, either as a means of adding security to your network, or whitelisting or blacklisting certain sites. In this situation you could set up Sauce Connect behind a proxy like BrowserMob, which records all traffic that passes through it, and which can later provide a history of that traffic.

Proxied Site Under Test (SUT)

In this configuration the SUT is behind a proxy, to allow even more control over traffic before it reaches the SUT. Typical this setup is used to control access to the SUT by means of IP whitelisting, or requiring a username and password to access the proxy.

Shorthand Longhand

SC Host Machine in your network that the Sauce Connect application is running on.

SUT Site Under Test, the site that you're testing.

Sauce Connect Setup and Configuration

Topics in this section explain the set up and configuration process for Sauce Connection, including requirements, setting up Sauce Connect to work with proxies, and using multiple Sauce Connect tunnels.

System Requirements for Sauce Connect Setting Up Sauce Connect Monitoring Sauce Connect with a Service Management Tool Using Multiple Sauce Connect Tunnels Sauce Connect Proxy Configuration

System Requirements for Sauce Connect

System Requirements

System requirements vary depending on the number of parallel tests you plan to run. Here are some samples based on simultaneous test volume:

Machine Type RAM Processor Parallel Tests

Local 4GB 4GHz 10

Dedicated 8GB 4GHz 400+

EC2 T2 small 2GB - 0-49

EC2 T2 medium 4GB - 50-100

EC2 T2 large 8GB - 400+

For increased reliability and security, use a dedicated server. You will also want an uplink capable of at least 5MB/s when running 50+ tests in parallel.

On Unix-based systems, you may need to increase your open file limit if your parallel test count is high (for example, ulimit -n 8192 ).

Setting Up Sauce Connect

Prerequisites

Make sure you've reviewed the System Requirements for Sauce Connect You should look over the examples of Sauce Connect Network Configurations to understand the best way to set up Sauce Connect in your network architecture You should read Sauce Connect Setup and Teardown Process to understand how Sauce Connect uses tunnels to connect your site or application under test to a browser running on a virtual machine in the Sauce Labs data center, and Sauce Connect and Security to understand how that tunnel is secured Take a few minutes and read over the Sauce Connect Best Practices

Different Machine, But Same Network The most important thing to understand about setting up Sauce Connect is that while it doesn't need to be set up on the same machine as the site or application you're testing, it must be on the same network. Dysfunctional Sauce Connect Network Configurations illustrate s some examples of network architectures in which Sauce Connect will not be able to create a tunnel, or will be too slow to carry out effective testing.

Procedure

1. Download the appropriate version of Sauce Connect for your operating system.

Download Link SHA1 Checksum

Download Sauce Connect v4.3.12 for OS X 10.8+ 99c2577373d31061af4c6c24c53f65fdbd7f17c0

Download Sauce Connect v4.3.12 for Windows 01c2b09a2b5e8b3c5f5f18b4481c69988927318a

Download Sauce Connect v4.3.12 for Linux 8c24f989091e869d9a0cc387c3197590a1f20521

Download Sauce Connect v4.3.12 for Linux 32-bit 133501f013eccb5349ca2fa1d2c7cd5f540acc24

2. On the machine where you want to install Sauce Connect, open outbound port 443. Alternatively, you can configure Sauce Connect with a proxy that can reach saucelabs.com by using the Sauce Connect command line options --proxy or --pac. See the topic Sauce Connect Proxy Configuration for more information. 3. After extracting the Sauce Connect files onto your machine, go to the install directory and run this command.

bin/sc -u YOUR_USERNAME -k YOUR_ACCESS_KEY

When you see connected, you're ready to go!

Finding Your Username and Access Key You can find your Sauce Labs username and access key in the User Profile > User Settings section of your Sauce Labs dashboard. You should also check out our topic on setting your username and access key as environment variables.

Monitoring Sauce Connect with a Service Management Tool

You can monitor Sauce Connect more easily with a Service Managment tool like systemd or upstart. These tools make using Sauce Connect a more fluid experience, and allow time for Sauce Connect to clean up upon exiting. This is useful in situations where you want to kill the Sauce Connect process, and start one instantly after that, because it takes time to shut down Sauce Connect remotely. By using these tools, you have a graceful way to shut down and restart your Sauce Connect instances that already account for the time involved for the shut down and clean up processes.

Setting Up Systemd Setting Up Upstart

Setting Up Systemd

1. Navigate to your /local/bin directory where you want to install Sauce Connect.

cd /usr/local/bin

2. Download Sauce Connect.

wget https://saucelabs.com/downloads/sc-4.3.11-linux.tar.gz

3. Untar the Sauce Connect file.

tar -zxvf sc-4.3.11-linux.tar.gz

4. Copy the Sauce Connect file into a dedicated directory.

cp sc-4.3.11-linux/bin/sc

5. Make sure Sauce Connect is located in the correct directory.

ls /usr/local/bin/sc

6. Change to the system directory.

cd /etc/systemd/system

7. In the system directory, reate a service file sc.server with these contents. Change the username and access key in the file to match your own.

Page 196 7.

sc.server example file Expand source [Unit] Description=Sauce Connect After=network.target

[Service] Type=simple User=nobody Group=nogroup ExecStart=/usr/local/bin/sc -u -k -l /tmp/sc_long.log --pidfile /tmp/sc_long.pid --se-port 0

[Install] WantedBy=multi-user.target

8. Reload the service file.

sudo systemctl daemon-reload

9. Start the service.

sudo systemctl start sc.service

10. Check the status of the service.

sudo systemctl status sc.service

11. You can stop the service with this command.

sudo systemctl stop sc.service

Setting Up Upstart

1. Navigate to your /local/bin directory where you want to install Sauce Connect.

cd /usr/local/bin

2. Download Sauce Connect.

wget https://saucelabs.com/downloads/sc-4.3.11-linux.tar.gz

3. Untar the Sauce Connect file.

tar -zxvf sc-4.3.11-linux.tar.gz

4. Copy the Sauce Connect file into a dedicated directory.

cp sc-4.3.11-linux/bin/sc

5. Make sure Sauce Connect is located in the correct directory.

ls /usr/local/bin/sc

6. Change to the /etc/init directory.

cd /etc/init

7. In the /etc/init directory, create a file sc.conf with these contents. Change the username and access key in the file to match your own.

sc.conf example file Expand source # #This Upstart config expects that Sauce Connect is installed at #/usr/local/bin/sc. Edit that path if it is installed somewhere else. # #Copy this file to /etc/init/sc.conf, and do: # # $ sudo initctl reload-configuration # #Then you can manage SC via the usual upstart tools, e.g.: # #$ sudo start sc #$ sudo restart sc #$ sudo stop sc #$ sudo status sc # start on filesystem and started networking stop on runlevel 06

respawn respawn limit 15 5

#Wait for tunnel shutdown when stopping Sauce Connect. kill timeout 120

#Bump maximum number of open files/sockets. limit nofile 8192 8192

#Make Sauce Connect output go to /var/log/upstart/sc.log. console log

env LOGFILE="/tmp/sc_long.log" env PIDFILE="/tmp/sc_long.pid"

Page 198 7.

env EXTRA_ARGS="--se-port 0" env SAUCE_USERNAME="CHANGEME" # XXX env SAUCE_ACCESS_KEY="CHANGEME" # XXX

post-start script # Save the pidfile, since Sauce Connect might remove it before the # post-stop script gets a chance to run. n=0 max_tries=30 while [ $n -le $max_tries ]; do if [ -f $PIDFILE ]; then cp $PIDFILE ${PIDFILE}.saved break fi n=$((n+1)) [ $n -ge $max_tries ] && exit 1 sleep 1 done end script

post-stop script # Wait for Sauce Connect to shut down its tunnel. n=0 max_tries=30 pid="$(cat ${PIDFILE}.saved)" while [ $n -le $max_tries ]; do kill -0 $pid || break n=$((n+1)) [ $n -ge $max_tries ] && exit 1 sleep 1 done end script

setuid nobody setgid nogroup

Page 199 7.

chdir /tmp

exec /usr/local/bin/sc -l $LOGFILE --pidfile $PIDFILE $EXTRA_ARGS

8. Reload the service.

sudo initctl reload-configuration

9. Start the service.

sudo start sc

10. Check the status of the service.

sudo status sc

11. You can stop the service with this command.

sudo stop sc

Using Multiple Sauce Connect Tunnels

In most situations, you will only need to use one Sauce Connect tunnel for your tests, and all traffic from jobs that are run under your account will use that tunnel automatically and transparently. However, there may be situations in which you want to run multiple Sauce Connect tunnels at the same time. For example, you may want to use a tunnel on your local machine and another on your CI server, or perhaps run multiple jobs on the same CI server. This topic describes how to use tunnel identifiers to run multiple tunnels on the same machine or different machines, and how identified and unidentified tunnels respond to starting up a new tunnel.

Using Tunnel Identifiers with Multiple Tunnels Additional Flags for Multiple Tunnels on the Same Machine

Using Tunnel Identifiers with Multiple Tunnels

If you want to use multiple Sauce Connect tunnels, you need to create tunnel identifiers to manage them. This is because Sauce Connect only supports running one tunnel with the same identifier at at a time. This table shows how tunnels running on different machines with different tunnel identifiers will interact with each other when they start up.

An Already Running Unidentified Tunnel Tunnel Alpha Tunnel Beta

When you start a new

Unidentified Tunnel Will shut down Will remain up Will remain up

Tunnel Alpha Will remain up Will shut down Will remain up

Tunnel Beta Will remain up Will remain up Will shut down

To create a tunnel identifier, start Sauce Connect using the --tunnel-identifier flag (or -i) and provide your own unique identifier string. Once the tunnel is up and running, any tests that you want going through this tunnel will need to provide the correct identifier using the tunnelId entifier desired capability. This is how you make sure that a job uses the correct tunnel, as jobs will also automatically make use of unidentified tunnels. This table shows what to expect when you are running jobs with both identified and unidentified tunnels.

When you No tunnels An unidentified tunnel Tunnel Alpha run An unidentified tunnel AND tunnel Alph have running running ning a running

When tunnel-identifier is

Not provided No tunnel is used The unidentified tunnel is No tunnel is used The unidentified tunnel is used used

Alpha Your tests don't Your tests don't work Tunnel Alpha is Tunnel Alpha is used work used

Beta Your tests don't Your tests don't work Your tests don't Your tests don't work work work

Using Tunnels with Some Tests But Not Others If you want to have some tests use tunnels, and some not use any tunnels at all, use tunnel identifiers for all your tunnels.

Additional Flags for Multiple Tunnels on the Same Machine

If you want to run different tunnels on the same machine, you need to use additional flags to start them, because multiple tunnels on the same machine require different ports, logfiles and PID files. The required flags are --pidfile, --logfile, --sc-proxy-port, --se-port, and -i , as shown in this example:

sc --pidfile /tmp/sc2.pid --logfile /tmp/sc2.log --scproxy-port 29999 --se-port 4446 -i my-tun2

Flag Description

--pidfile A file used to record the process identifier. It can be erased or reused once the tunnel is shut down.

--logfile The location where Sauce Connect writes its logs. This can be reused or erased once the tunnel is shut down, but since Sauce Labs Support will sometimes request this file if you're having test issues, deleting it automatically isn't a great idea.

--sc-proxy-port An internal Sauce Connect port. This simply needs to be a free port.

--se-port The port for Sauce Connect's inbuilt Selenium Relay. This can be any free port if your test code sends Selenium Commands directly to Sauce Labs. Otherwise, you'll need to match this port to the port used by the test code that will be using this tunnel.

--i The tunnel identifier

Sauce Connect Proxy Configuration

Automatic Proxy Configuration Command Line Configuration BrowserMob Proxy Configuration

Automatic Proxy Configuration

As of Sauce Connect 4.3.1, proxies and PAC settings are autoconfigured based on the settings of the operating system on the machine where it is installed.

On Windows, Sauce Connect will use the proxy settings for Internet Explorer, as well as the system-wide proxy settings that set in the C ontrol Panel. On Mac OS X, Sauce Connect will use the proxy settings in Preferences/Network. Both proxy and PAC settings are supported. On Linux, Sauce Connect looks for these variables, in this order:

http_proxy HTTP_PROXY all_proxy ALL_PROXY. They can be in the form http://host.name:port or host.name:port.

You can disable automatic proxy detection with the command line option ./sc -z --no-autodetect.

Command Line Configuration

If automatic proxy configuration fails, you need to override the settings, or you need to enable proxies through a test script, there are several com mand-line options that you can use to configure proxies manually.

-p --proxy -w --proxy-userpwd -T --proxy-tunnel --pac

Why are there -p, --proxy and a -T, --proxy-tunnel options? Fundamentally, Sauce Connect makes two separate outbound connections for two separate purposes. The first, which -p, --proxy uses, is a lightweight connection to our REST API that simply tells our servers basic information about the status of Sauce Connect's status (for example, starting up, ready, stopping).

The second connection is to the actual tunnel virtual machine (VM) created for your Sauce Connect instance. Enabling the -T, --proxy-tunnel flag will cause some proxy specified with -p, --proxy to be used for this connection as well. You should try to avoid using a proxy for this connection, since it is already TLS secured, and a large volume of data tends to go over this connection. Adding another step in the middle, in the form of a proxy, can hinder test performance.

Ideally you should only need to use -p, --proxy (and perhaps -w, --proxy-userpwd for credentials), but -T, --proxy-tunnel is available if your network doesn't allow outgoing connections on port 443. If your tests are slow, you may want to ask your network administrator about making an exception for this connection.

BrowserMob Proxy Configuration

You can use BrowserMob Proxy with Sauce Connect 4.2 and higher by using the Sauce Connect PAC feature.

Since BrowserMob bumps SSL connections, and Sauce Connect 4 verifies the certificate when connecting to the REST API, that means the --pr oxy option can't be used with a BrowserMob proxy. Sauce Connect will complain that the certificate received from the REST endpoint is invalid.

To choose a proxy in a dynamic fashion, however, you can create a PAC file that matches the REST and tunnel VM hostnames, and uses the local BrowserMob proxy for everything else.

1. Start BrowserMob Proxy.

Page 203 1.

$ ./browsermob-proxy --port 9000

2. Start another terminal and create a proxy instance in BrowserMob.

$ curl -X POST http://localhost:9000/proxy {"port":9091}

3. Create a file, browsermob.js, with this content. This is what you will pass into Sauce Connect 4.

function FindProxyForURL(url, host) { if (shExpMatch(host, "*.miso.saucelabs.com") || shExpMatch(host, "saucelabs.com")) { // KGP and REST connections. Another proxy can also be specified. return "DIRECT"; }

// Test HTTP traffic, route it through the local BrowserMob proxy. return "PROXY localhost:9091"; }

4. Copy the file to the same directory as Sauce Connect 4, and start Sauce Connect 4.

$ ./sc -v --pac file://$(pwd)/browsermob.js

Sauce Connect 4 will print the proxy used for REST and the tunnel VM connection, which should be Using no proxy for connecting to Sauce Labs REST API and Using no proxy for connecting to tunnel VM if their hostnames are configured with DIRECT access.

Sauce Connect Command Line Reference

Usage: ./sc

Command Arguments Description

-u, --user You can also use the environment variable SAUCE_USERNAME

-k, --api-key You can also use the environment variable SAUCE_ACCESS_KEY

-B, --no-ssl-bump-domains Comma-separated list of domains. Requests including hosts that matches one of these domains <...> will not be SSL re-encrypted. See the section on SSL Bumping Enabled by Default in the Troubl eshooting Sauce Connect topic for more information about situations in which you would want to use this command.

-D, --direct-domains <.. Comma-separated list of domains. Requests including hosts that matches one of these domains .> will be relayed directly through the Internet, instead of through the Sauce Connect tunnel.

-t, --tunnel-domains <.. Inverse of --direct-domains. Only requests for domains in this list will be sent through the .> Sauce Connect tunnel.

-v, --verbose Enable verbose debugging. -vv will output HTTP headers as well.

-F, --fast-fail-regexps Comma-separated list of regular expressions. Requests with URLs matching one of these will get <...> dropped instantly and will not go through the tunnel. See the question How Do I Use Sauce Connect to Test Graceful Degradation in the Sauce Connect FAQs for an example of how you would use this command to test for application or site degradation based on missing assets or resources.

-i, --tunnel-identifier Assign to this Sauce Connect instance. Future jobs will use this tunnel only when explicitly s pecified by the tunnel-identifier DesiredCapability in a Selenium client. Check out the topic Using Multiple Sauce Connect Tunnels for information on using tunnel-identifier to run multiple Sauce Connect tunnels simulteanously. Test Configuration Options contains more information about the syntax for setting tunnel-identifier as a DesiredCapability.

-l, --logfile

-P --se-port Port on which Sauce Connect's Selenium relay will listen for requests. Selenium commands reach ing Sauce Connect on this port will be relayed to Sauce Labs securely and reliably through Sauce Connect's tunnel. Defaults to 4443.

-p, --proxy Proxy host and port that Sauce Connect should use to connect to the Sauce Labs cloud. See Sau ce Connect Proxy Configuration and Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration for more information about using Sauce Connect with proxies.

-w, --proxy-userpwd Proxy Configuration and Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration for more information about using Sauce Connect with proxies.

--pac Proxy autoconfiguration. Can be a http(s) or local file:// URL. See Sauce Connect Proxy Configuration and Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration for more information about using Sauce Connect with proxies.

-T, --proxy-tunnel Use the proxy configured with -p for the tunnel connection. See Sauce Connect Proxy Configuration and Sauce Connect with a Proxy for Network Monitoring and Filtering Configuration for more information about using Sauce Connect with proxies.

-s, --shared-tunnel Allows sub-accounts of the tunnel owner to use the tunnel. See the question Can I Reuse a Tunnel Between Multiple Accounts? in the Sauce Connect FAQs for more information on using this command.

-x, --rest-url Advanced feature: Connect to SauceREST API at alternative URL. Use only if directed to do so b y Sauce Labs support.

-f, --readyfile File that will be touched to indicate when tunnel is ready.

-a, --auth asks for a username and password. er:pwd> This option can be used multiple times.

-z, --log-stats . Information includes bytes transmitted, reque > sts made, and responses received.

--max-logsize size. Disabled by default. >

--doctor Perform checks to detect possible misconfiguration or problems.

--no-autodetect Disable the autodetection of proxy settings.

-h, --help Display the help text.

Sauce Connect FAQS

Can I Reuse a Tunnel Between Multiple Accounts? I Have Verbose Logging On, but I'm Not Seeing anything in stdout. What Gives? How Can I Periodically Restart Sauce Connect? How Can I Use Sauce Connect to Test Graceful Degredation? Can I Access Applications on localhost? If We Have Five Users, Can We Use Five instances of Sauce Connect, or Do We Have to Set Up One Shared Instance?

Can I Reuse a Tunnel Between Multiple Accounts?

Tunnels started by an account can be reused by its sub-accounts.

1. Start Sauce Connect with the --shared-tunnel command from the main account in your account tree.

sc -u USERNAME -k ACCESS_KEY --shared-tunnel

2. Once the tunnel is running, provide the special parentTunnel DesiredCapability on a per-job basis. The value of this capability should be the username of the parent account that owns the shared Sauce Connect tunnel as a string.

capabilities['parentTunnel'] = "parentAccount"

Jobs that request this capability will route all their traffic through the tunnel created using your parent account

What Firewall Rules Do I Need?

Sauce Connect needs to make outbound connections to saucelabs.com and *.miso.saucelabs.com on port 443 for the REST API and the primary tunnel connection to the Sauce cloud. Verifying the SSL certificate in use may require an outbound connection on ports 443 and 80 to g.symcd.com. Sauce Connect can optionally create tunnels through a web proxy.

I Have Verbose Logging On, but I'm Not Seeing anything in stdout. What Gives?

Output from the -v flag is sent to the Sauce Connect log file rather than stdout. See the section on Generating Logs for Troubleshooting in Tro ubleshooting Sauce Connect for more information.

How Can I Periodically Restart Sauce Connect?

Sauce Connect handles a lot of traffic for heavy testers. This is one way to keep it 'fresh' to avoid leakages and freezes.

1. First write a loop that will restart Sauce Connect every time it gets killed or crashes:

while :; do killall sc; sleep 30; sc -u $SAUCE_USERNAME -k $SAUCE_ACCESS_KEY; done

2. Write a cron task that will kill Sauce Connect on a regular basis:

rontab -e 00 00 * * * killall sc

Page 207 2.

This will kill Sauce Connect every day at 12am, but can be modified to behave differently depending on your requirements.

How Can I Use Sauce Connect to Test Graceful Degredation?

You can use the --F, --fast-fail-regexps command line option to drop requests that fit a description altogether. You could use it to simulate non-loading of scripts, styles, or other resources.

Can I Access Applications on localhost?

When using Sauce Connect, local web apps running on commonly-used ports are available to test at localhost URLs, just as if the Sauce Labs cloud were your local machine.

However, because an additional proxy is required for localhost URLs, tests may perform better when using a locally-defined domain name (which can be set in your hosts file) rather than localhost. Using a locally-defined domain name also allows access to applications on any port.

Android Ports and Sauce Connect On Android devices, ports 5555 and 8080 cannot be used with Sauce Connect

The Safari browser on OS X 10.10 Yosemite and mobile iOS 8+ proxies these common ports:

80, 443, 888, 2000, 2001, 2020, 2109, 2222, 2310, 3000, 3001, 3030, 3210, 3333, 4000, 4001, 4040, 4321, 4502, 4503, 4567, 5000, 5001, 5050, 5555, 5432, 6000, 6001, 6060, 6666, 6543, 7000, 7070, 7774, 7777, 8000, 8001, 8003, 8031, 8080, 8081, 8765, 8777, 8888, 9000, 9001, 9080, 9090, 9876, 9877, 9999, 49221, 55001

If you're testing on Safari and need a different port, please let us know! We do our best to support ports that will be useful for many customers, such as those used by popular frameworks.

If We Have Five Users, Can We Use Five instances of Sauce Connect, or Do We Have to Set Up One Shared Instance?

Feel free to use either, even if you only have one Sauce account! If you do decide to use five separate instances, you will need to create unique identifiers for each. You can also create sub-accounts that each have their own individual access keys.

Troubleshooting Sauce Connect

Generating Logs for Troubleshooting Network Configuration with Firewalls and Proxys Checking Network Connectivity to Sauce Labs SSL Bumping Enabled by Default For More Help

Generating Logs for Troubleshooting

By default, Sauce Connect generates log messages to your local operating system's temporary folder. On Linux / Mac OS X this is usually /tmp; on Windows, it varies by individual release. You can also specify a specific location for the output using the -l command line option.

You can enable verbose logging with the -v flag. Verbose output will be sent to the Sauce Connect log file rather than standard out.

Network Configuration with Firewalls and Proxys

Is there a firewall or proxy server in place between the machine running Sauce Connect and Sauce Labs (*.saucelabs.com:443)? You may need to allow access in your firewall rules, or configure Sauce Connect to use a proxy. Sauce Connect needs to establish outbound connections to saucelabs.com (162.222.73.28) on port 443, and to one of many hosts makiXXXXX.miso.saucelabs.com IPs (162.222.72.0/21), also on port 443.

Check out the topics under Sauce Connect in the Network for examples of how to set up Sauce Connect within various network environments, including examples of dysfunctional network configurations.

Checking Network Connectivity to Sauce Labs

Make sure that saucelabs.com is accessible from the machine running Sauce Connect. This can be tested by issuing a ping, telnet or cURL command to saucelabs.com from the machine's command line interface. If any of these commands fail, you should work with your internal network team to resolve them.

ping Method for Checking Network Connectivity

ping saucelabs.com

telnet Command for Checking Network Connectivity

telnet saucelabs.com 443

This command should return an IP address of 162.222.73.2.

cURL Method for Checking Connectivity

curl -v https://saucelabs.com/

This command should return the status message connected to saucelabs.com.

SSL Bumping Enabled by Default SSL bumping is on by default, but this can cause problems with WebSocket-based traffic and AJAX-reliant sites. You can disable SSL bumping for specific URLs with the -B command.

For More Help

If you need additional help, get in touch at [email protected]. To provide our support team with additional information, please add the -vv and -l sc.log options to your Sauce Connect command line, reproduce the problem, and attach the resulting log file (called sc.log) to your support request.

Sauce Connect Best Practices

Be Aware of How Sauce Connect Will Interact with Your Network Architecture Use the Latest Version of Sauce Connect Use a Single Sauce Connect Tunnel for Each Test Suite or Build

Be Aware of How Sauce Connect Will Interact with Your Network Architecture

When you set up Sauce Connect, you don't have to set it up on the same machine as the application or website you're testing, but it must be on the same network as that machine. For this reason, it's very important to understand how Sauce Connect will interact with components of your network architecture such as proxies, firewalls, and geographically distributed data centers. You should read the topics under Sauce Connect in the Network and Sauce Connect Network Configurations for more information.

Use the Latest Version of Sauce Connect

Since every new version of Sauce Connect includes improvements to functionality and performance, you should make sure to always use the latest version, available from these download links:

Operating System Download Link SHA1 Checksum

Mac OS X Sauce Connect v4.3.11 for OS X SHA1 checksum: 5d0aa851d21f3d4a21f298b6a921761c6aa15217

Windows Sauce Connect v4.3.11 for Windows SHA1 checksum: 7806e9753f66eece56a174fddbba4455ecaf72a4

Linux Sauce Connect v4.3.11 for Linux SHA1 checksum: b4f3464738dbcba57d9c6d316d711cfc98dd70f3

Linux 32-bit Sauce Connect v4.3.11 for Linux 32-bit SHA1 checksum: f27be22c5e7b8ba905c5886cf93a4e63e6c370b3

Use a Single Sauce Connect Tunnel for Each Test Suite or Build

If you're using Sauce Connect to build a tunnel between your application and the Sauce Labs testing infrastructure, you should use a single Sauce Connect instance for each test suite or build. Your test automation framework should launch Sauce Connect before the test suite is triggered, and should shut it down when the suite. If you're using a continuous integration platform like Jenkins, you can use the Sauce OnDemand plugin to launch and tear down your Sauce Connect instance.

If you use persistent tunnels for your tests and builds, you can run into issues with tunnels being unexpectedly down (an example of avoiding an external dependency for your tests), and you can avoid the need to build logic into your tests to see if a a specific persistent tunnel is up and passing traffic.

Using Sauce Labs with Continuous Integration Platforms

There are two types of Sauce Labs integrations with CI platforms.

Integrating a CI Platform with Sauce Labs Through Sauce Plugins

Sauce Labs has developed plugins to make it easy to integrate testing in our browser cloud with many of the most popular CI platforms. These integrations are directly supported by Sauce Labs.

Setting Up Sauce for Visual Studio Online PREVIEW Setting Up Sauce Labs with Bamboo Setting Up Sauce Labs with Jenkins Setting Up Sauce Labs with TeamCity

Integrating Other CI Platforms with Sauce Labs

In addition to the CI/CD platform integrations that are directly supported by Sauce Labs through our plugins, several CI/CD platform developers have created their own integrations with Sauce Labs. This table lists the integrations that are currently available, along with links to the documentation that describes how to set them up. If you need assistance setting up these integrations, contact the Support group for the CI/CD platform developer.

CI/CD Platform Documentation for Integration with Sauce Labs

CircleCI Test with Sauce Labs

Solano CI Setting Up Sauce Labs

Travis CI Using Sauce Labs with Travis CI

Setting Up CI Platform and Sauce Labs Integrations with Sauce Plugins

To make it easier to integrate testing in the Sauce Labs browser cloud with your continuous integration/continuous delivery platforms, Sauce Labs has developed plugins for several of the most popular platforms. These plugins have three key features:

1. They provides a user interface that lets you populate environment variables your CI/CD server that can be used in your tests (for example, platform configurations, or your Sauce username and access key). Much of what the plugins accomplish relates to the setting of environment variables. 2. They automatically launch Sauce Connect when you enable it for a project. 3. They handle reporting between your CI/CD platform and Sauce.

Setting Up Sauce for Visual Studio Online PREVIEW Setting Up Sauce Labs with Bamboo Setting Up Sauce Labs with Jenkins Setting Up Sauce Labs with TeamCity

Setting Up Sauce for Visual Studio Online PREVIEW

Visual Studio Online (VSO) enables continuous delivery by speeding up the testing cycle while increasing the quality of mobile and desktop applications. With Sauce for VSO you can easily authenticate on Sauce Labs as a part of the VSO build process. You can also use it to launch Sa uce Connect for testing applications on localhost or behind a firewall.

Setting Up the Sauce Labs Manage Credentials Task for Your Build

The Manage Credentials task is what allows you to authenticate with your Sauce Labs account via VSO and start Sauce Connect. You need to configure the task to create a new service endpoint that will contain your Sauce Labs username and access key.

1. In your VSO dashboard, click Build. 2. Under Build Definition, choose an appropriate template for your project. 3. Configure your Source Settings for the project. 4. Click Create. 5. Click Add Build Step. 6. Under Test, search for and add Sauce Labs - Manage Credentials. 7. Under Add build step, select Sauce Labs - Manage Credentials. 8. Next to the Sauce Labs Credentials menu, click Manage. This will open the Services tab. 9. Click New Service Endpoint. 10. Select New Sauce Labs Credentials. 11. For Connection Name, enter an appropriate name. 12. For User name and API Token, enter your Sauce username and access key. 13. Click OK. 14. Go back to the build steps for your project, and make sure that under Control Options for the Manage Credentials task, the Enabled o ption is selected

Using Sauce Connect

Sauce Connect is a tunneling application that established a secure connection between applications or sites under test in your local network or machine, and the virtual machines running browsers in the Sauce Labs testing cloud. It is not necessary to set up Sauce Connect to run tests with the Sauce Labs browser cloud, you only need to use it if the application or site under test is not publicly accessible over a network. Check out the topics under Using Sauce Connect for Testing Behind the Firewall or on Localhost for more information.

If you want to use Sauce Connect with VSO, you need to both Enable Sauce Connect in your Manage Credential task and add the Sauce Labs - Stop Sauce Connect task to your build. These will start up a Sauce Connect tunnel when your build begins, and then close the tunnel when it finishes.

1. In your Manage Credentials task, click Enable Sauce Connect. 2. Select whether the task should Always run and/or Continue on error. 3. In your build definition, search for and add Sauce Labs - Stop Sauce Connect. 4. Make sure that the Sauce Labs - Stop Sauce Connect task is enabled. We recommend that you set this task to Always run so there are no extra Sauce Connects tunnels stay running after your job ends.

Task Order is Important If you use the Sauce Connect tasks in your build, you must have your build steps set up so that the Sauce Labs - Manage Credentials task executes first.

Setting Up Sauce Labs with Bamboo

These topics explain how to integrate Atlassian Bamboo with Sauce by using the Sauce Plugin for Bamboo.

Installing and Configuring the Sauce Labs Plugin for Bamboo Configuring Bamboo for a Java Project with Sauce Labs Configuring Bamboo for a Python Project with Sauce Labs Referencing Environment Variables for Bamboo Jobs Outputting the Bamboo Session ID to stdout

Installing and Configuring the Sauce Labs Plugin for Bamboo

You can install the Sauce plugin through the Bamboo Administration page.

Plugin Source Code on GitHub You can find the source code for the Sauce Bamboo Plugin at https://github.com/saucelabs/bamboo_sauce

1. In Bamboo, go to the Administration page. 2. In the Plugins section, click Plugin Manager. 3. In the Search field, enter sauce, and then click Search. 4. In the search results, click the expand icon next to the Bamboo Sauce OnDemand Plugin name to access the Install button. 5. Click Install. 6. After the plugin downloads, restart Bamboo. 7. After Bamboo restarts, go to the Administration page. 8. Under Communication, click Sauce OnDemand. 9. Under Credentials, enter the username and access key for your Sauce account. You're now ready to configure Bamboo to work with projects in your favorite language.

Configuring Bamboo for a Java Project with Sauce Labs

This topic describes how to configure Bamboo to work with Sauce for a Java-based project. It includes a set of demo tests you can use to test your configuration and see how Sauce interacts with Bamboo.

Using the Java Helper Library with TestNG and JUnit If you are using the Java Helper Libraries with TestNG or JUnit

Create a Plan Configure Tasks Configure the Plan Enable the Sauce Plugin Run the Example Tests

Create a Plan

1. In Bamboo, click Create Plan. 2. Click Create New Plan. 3. Under Plan Details, for Project, select New Project. 4. For Project Name, enter Sauce Demo. 5. For Project Key, enter SAUCE. 6. For Plan Name, enter Java. 7. For Plan Key, enter Demo. 8. Under Source Repositories, in the Source Repository menu, select Git. 9. For Repository URL, enter https://github.com/rossrowe/sauce-ci-java-demo. 10. For Branch, enter Master. 11. For Authentication Type, select None. 12. Select Use shallow clones.

Configure Tasks

1. Click Configure Tasks. 2. Click Add Task. 3. Select Maven 3.x. 4. For Task Description enter Run Tests. 5. Click Save. 6. Click Create.

Configure the Plan

1. Under Plan Configuration > Stages and Jobs > Default Stage, select Default Job. 2. Click Miscellaneous. 3. For Job Name, enter Default Job. 4. Select Job Enabled. 5. Click Save.

Enable the Sauce Plugin

1. Select Enable Sauce OnDemand. 2. In General Settings, select the Selenium Version you want to use for your tests. 3. Select the Browser you want to run your tests against. See Referencing Environment Variables for Bamboo Jobs for more information about how your browser and general settings are used to populate environment variables for your tests. 4. Enter the Max Duration, Idle Timeout, and Starting Browser URL settings for your test. 5. Click Save.

Sauce Connect Automatically Enabled In the General Settings you will see that Enable Sauce Connect is selected by default, which will launch an instance of Sauce Connect prior to the running of your Job. This instance will close when the Job completes.

Run the Example Tests

1. Go the Bamboo dashboard. 2. Click the Enable icon. 3. Click the Run icon. 4. After the tests complete, click Sauce Jobs. 5. Click the Job ID of any job to see the steps performed by the test as well as a test video.

Configuring Bamboo for a Python Project with Sauce Labs

This topic describes how to configure Bamboo to work with Sauce for a Python-based project. It includes a set of demo tests you can use to test your configuration and see how Sauce interacts with Bamboo.

Nose Testing Library For illustration purposes, this topic assumes that you are using the Nose testing library, and that this library has already been installed in your testing environment. For more information about Nose, check out the official Nose docs site.

Create a Plan Configure Tasks Configure the Plan Enable the Sauce Plugin Run the Example Tests

Create a Plan

1. In Bamboo, click Create Plan. 2. Click Create New Plan. 3. For Project, select New Project. 4. For Project Name, enter Sauce Demo. 5. For Project Key, enter Sauce. 6. For Plan Name, enter Python. 7. For Plan Key, enter Demo. 8. Under Source Repositories, in the Source Repository menu, select Git. 9. For Repository URL, enter https://github.com/rossrowe/sauce-ci-python-demo. 10. For Branch, enter Master. 11. For Authentication Type, select None. 12. Select Use shallow clones.

Configure Tasks

1. Click Configure Tasks. 2. Click Add Task. 3. Click Command. 4. In the Command Configuration dialog, for Task Description, enter Run task. 5. Next to Executable, click Add Executable. 6. In the New Executable dialog, for Executable Label, enter nosetests. 7. For Path, enter the path to your nose library. 8. Click Save. 9. In the Command Configuration dialog, for Argument, enter --with-nosexunit simple_test.py. 10. Click Save. 11. Click Create.

Don't Enable the Plan Yet! You'll need to enter the Sauce configuration after you configure the plan, so don't select Enable this plan until after you've completed the plan configuration steps.

Configure the Plan

1. Under Plan Configuration > Stages and Jobs > Default Stage, select Default Job. 2. Click Miscellaneous. 3. For Job Name, enter Default Job. 4. Select Job Enabled. 5. Click Save.

Enable the Sauce Plugin

1. Select Enable Sauce OnDemand. 2. In General Settings, select the Selenium Version you want to use for your tests. 3. Select the Browser you want to run your tests against. 4. Enter the Max Duration, Idle Timeout, and Starting Browser URL settings for your test.

5. Click Save.

Run the Example Tests

Referencing Environment Variables for Bamboo Jobs

The Sauce Bamboo plugin will set a series of environment variables that reflect the values you enter for the Bamboo job configuration. Your unit tests can reference these environment variables explicitly, or through the use of the Selenium Client Factory Library described in this topic.

Variable Description

SELENIUM_HOST The hostname of the Selenium server

SELENIUM_PORT The port of the Selenium server

SELENIUM_PLATFORM The operating system of the selected browser

SELENIUM_VERSION The version number of the selected browser

SELENIUM_BROWSER The name of the selected browser

SELENIUM_DRIVER Contains the operating system, version and browser name of the selected browser, in a format designed for use by the Selenium Client Factory

SELENIUM_URL The initial URL to load when the test begins

SAUCE_USERNAME The user name used to invoke Sauce OnDemand

SAUCE_ACCESS_KEY The access key for the user used to invoke Sauce OnDemand

SELENIUM_STARTING_URL The value of the Starting URL field

SAUCE_ONDEMAND_BROWSERS A JSON-formatted string representing browsers you selected for the job configuration, as described Setting Desired Capabilities for Jenkins Projects

SAUCE_BUILDNUMBER The build number

Outputting the Bamboo Session ID to stdout

As part of the post-build activities, the Sauce plugin will parse the test result files in an attempt to associate test results with Sauce jobs. It does this by identifying lines in the stdout or stderr that have this format:

SauceOnDemandSessionID= job-name=

The session id can be obtained from the RemoteWebDriver instance and the job-name can be any string, but is generally the name of the test class being executed.

To make sure that your test results and Sauce jobs are associated properly, you need to output the session id to stdout. For example, this is the code you would use to output the session id to the Java stdout.

private void printSessionId() {

String message = String.format("SauceOnDemandSessionID=%1$s job-name=%2$s", (((RemoveWebDriver) driver).getSessionId()).toString(), "some job name"); System.out.println(message); }

Setting Up Sauce Labs with Jenkins

Jenkins is one of the most popular continuous integrations tools used in software development, and to make it easy for you to integrate your Sauce Labs testing with Jenkins, we developed the Sauce Jenkins Plugin. You don't have to use the Sauce Jenkins Plugin to integrate Sauce and Jenkins, but it does do several handy things for you:

1. It provides a user interface that lets you populate environment variables on the Jenkins server that can be used in your tests (for example, platform configurations, or your Sauce username and access key). Much of what the plugin does relates to the setting of environment variables. 2. It automatically launches Sauce Connect when you enable it for a project. 3. It handles reporting between Jenkins and Sauce.

These topics describe how to install and configure the Jenkins plugin. They assume that you have some familiarity with Jenkins and the basics of automated testing, but even if this is your first time working with Jenkins and Sauce, you should be able to set up a successful integration.

Sauce First, Jenkins Second You should make sure your tests run on Sauce without Jenkins before attempting to install and configure the Jenkins plugin.

Installing and Configuring the Sauce OnDemand Plugin for Jenkins Configuring Sauce Connect with the Jenkins Plugin Setting Desired Capabilities for Jenkins Projects Environment Variables Used by the Jenkins Plugin Running Parallel Tests with Jenkins Configuring Jenkins Matrix Projects with Sauce Setting Up Parameterized Builds for Jenkins and Sauce Setting Up Reporting between Sauce Labs and Jenkins

Installing and Configuring the Sauce OnDemand Plugin for Jenkins

Requirements

You need to be able to access the IP range 162.222.72.0/21, saucelabs.com, and ondemand.saucelabs.com from your Jenkins server. Check out the Jenkins documentation if you need to set up a proxy or other advanced network configuration.

Install the Plugin

You can install the Sauce OnDemand plugin for Jenkins though the Jenkins Administration page.

1. In Jenkins, go to the Administration page. 2. Click Manage Jenkins. 3. Click Manage Plugins. 4. Click the Available tab. 5. In the list of plugins, find and select Sauce OnDemand Plugin. 6. Click Download now and install after restart. The plugin file is fairly large, so download may take several minutes. 7. In the plugin installation dialog, select Restart Jenkins when installation is complete and no jobs are running.

Configure Your Sauce Labs Credentials

The Jenkins plugin provides an interface for storing your Sauce Labs authentication credentials as environment variables on the Jenkins server, which is one of our best practices for testing with Sauce. This allows you to reference your credentials without having to hardcode them into your tests, and because the plugin manages authentication at the global level, you can have multiple jobs running at the same time that use these credentials.

1. After the plugin has installed and Jenkins has restarted, go to the Administration page in Jenkins. 2. Click Manage Jenkins. 3. Click Configure System. 4. Under Sauce Support, enter the Username and API Access Key for your Sauce account.

Setting Authentication Environment Variables in Your Tests Once you've set up your authentication credentials as environment variables, you can refer to them in your tests like this:

WebDriver driver = new RemoteWebDriver( new URL("http://"+System.getenv("SAUCE_USERNAME")+":"+System.getenv("SAUCE_ACC ESS_KEY")+"@ondemand.saucelabs.com:80/wd/hub", desiredCapabilities);

Check out the topic Setting Desired Capabilities for Jenkins Projects for more information about using environment variables.

Configuring Sauce Connect with the Jenkins Plugin

Developing apps on localhost is quick and efficient. The drawback is that localhost is not a publicly-accessible address on the Internet, so the browsers in the Sauce Labs cloud can't load and test your app. The solution is to use Sauce Connect. Sauce Connect is an application that creates a secure tunnel connection between the Sauce Labs virtual machine that runs your test and your local network. You can also use Sauce Connect to test applications and websites that are located within your corporate firewall. Sauce Connect is not required to run tests on Sauce Labs, only in situations where the application or website you want to test is not publicly accessible.

The Jenkins plugin for Sauce automatically installs Sauce Connect on your Jenkins server, but you will need to configure your project to use it. There are also global and per-project configuration options for Sauce Connect.

Configuring Your Project to Use Sauce Connect Changing the Default Location of the Sauce Connect Binary Changing the Global Default Location Changing the Per-Project Default Location Setting Sauce Command Line Options Setting Global Sauce Connect Command Line Options Setting Per-Project Sauce Connect Command Line Options Launching Sauce Connect on the Jenkins Slave Node

Configuring Your Project to Use Sauce Connect

1. Go to your project in Jenkins. 2. In the left-hand navigation, select Configure. 3. Under Build Environment, select Sauce OnDemand Support. This will open up a section for Sauce Labs Options. 4. Select Enable Sauce Connect? After you make this selection, a new Sauce Connect tunnel will launch whenever Jenkins starts a build for the project.

SELENEIUM_HOST and SELENIUM_PORT Environment Variables If you enable Sauce Connect for your project, then the environment variables for SELENIUM_HOST and SELENIUM_PORT will be set to localhost:4445.

Changing the Default Location of the Sauce Connect Binary

When you run a Jenkins build with Sauce Connect enabled, the default behaviour of the plugin is to extract the Sauce Connect binary that is appropriate for your operating system to your home directory. You can change the location where the plugin extracts Sauce Connect for specific projects, or at the global level for all projects. Note that Sauce Connect should always run on the same network as the site or application under test, but does not have to be on the same machine.

Changing the Global Default Location

1. In Jenkins, go to the Administration page. 2. Click Manage Jenkins. 3. Click Configure System. 4. Under Sauce Support, enter the default location for Sauce Connect in the Sauce Connect Working Directory field.

Changing the Per-Project Default Location

1. In Jenkins, go to your project. 2. Select Configure. 3. In the Sauce Connect Advanced Options section, click Advanced. 4. Enter the default location for Sauce Connect for this project in the Sauce Connect Working Directory field.

Setting Sauce Command Line Options

There are a number of command line options you can use with Sauce Connect, which you can configure to execute at both the global level and on a per-project basis when Sauce Connect launches.

Setting Global Sauce Connect Command Line Options

1. In Jenkins, go to the Administration page. 2. Click Manage Jenkins. 3. Click Configure System. 4. Under Sauce Support, enter the command line options you'd like to use in the Sauce Connect Options field.

Setting Per-Project Sauce Connect Command Line Options

1. In Jenkins, go to your project. 2. Select Configure. 3. In the Sauce Connect Advanced Options section, click Advanced. 4. Enter the command line options you'd like to use in the Sauce Connect Options field.

Launching Sauce Connect on the Jenkins Slave Node

Another advanced option is launching Sauce Connect on the Jenkins slave node that is executing the build, rather than on the Master node.

1. In Jenkins, go to your project. 2. Select Configure. 3. In the Sauce Connect Advanced Options section, click Advanced. 4. Select Launch Sauce Connect on Slave.

Setting Desired Capabilities for Jenkins Projects

The Sauce plugin for Jenkins provides an easy way for you to populate the desired capabilities for operating system and browser combinations for your tests as environment variables, rather than needing to hardcode them into your tests. This is a best practice for automated testing, since it means you can use your tests with multiple operating system/browser combinations without having to rewrite your test.

Prerequisites Setting Desired Capabilities for Your Project Setting Up Your Tests to Use the Environment Variables

Prerequisites

If you haven't enabled the Sauce plugin for Jenkins yet, you should follow the instructions in Installing and Configuring the Sauce OnDemand Plugin for Jenkins.

Overriding the Environment Variables for Authentication If you want to override the default global authentication credentials that you set as environment variables when you configured the plugin, select Override default authentication in the Build Environment section of the project configuration page. After you select this option, enter the authentication credentials you want to use for the project.

Setting Desired Capabilities for Your Project

1. In Jenkins, go to your project. 2. In the left-hand navigation, select Configure. 3. In the Sauce Labs Options section, select the browser automation tool you want to use, for example WebDriver. When you select a browser automation tool, it will open up a menu of operating system and browser combinations that you can test against with that tool. 4. Select the the operating systems and browsers you want to test against. Select Use latest version of selected browsers to default to the latest version of the browsers you select.

If you select a single operating system/browser combination, then the environment variables SELENIUM_PLATFORM, SELENIUM_VERSIO N, and SELENIUM_BROWSER will be populated with your selections. If you select a mobile operating system/browser, the variables SELEN IUM_DEVICE and SELENIUM_DEVICE_TYPE will be populated. If you select multiple OS/browser combinations, the SAUCE_ONDEMAND_BROWSERS environment variable will be populated with a JSON-formatted string containing the attributes of the selected browsers, as shown in this example:

SAUCE_ONDEMAND_BROWSERS Environment Variable

[ { "platform":"LINUX", "os":"Linux", "browser":"firefox", "url":"sauce-ondemand:?os=Linux&browser=firefox&browser-version=16", "browserVersion":"16" }, { "platform":"VISTA", "os":"Windows 2008", "browser":"iexploreproxy", "url":"sauce-ondemand:?os=Windows 2008&browser=iexploreproxy&browser-version=9", "browserVersion":"9" } ]

Setting Up Your Tests to Use the Environment Variables

In your test script, you reference the environment variables as part of your desired capabilities. Though the exact syntax will vary depending on your scripting language, this example illustrates the way you would reference the environment variables SELENIUM_BROWSER, SELENIUM_VERS ION, AND SELENIUM_PLATFORM in your test script.

desiredCapabilities.setBrowserName(System.getenv("SELENIUM_BROWSER")); desiredCapabilities.setVersion(System.getenv("SELENIUM_VERSION")); desiredCapabilities.setCapability(CapabilityType.PLATFORM, System.getenv("SELENIUM_PLATFORM"));

This example is for a single operating system/browser combination. If you have multiple selections, you can load the JSON string for the SAUCE_ ONDEMAND_BROWSERS environment variable by using the JSON library for your scripting language, and then loop through the string to send the various combinations to your test framework.

Environment Variables Used by the Jenkins Plugin

Variable Description

SELENIUM_HOST The hostname of the Selenium server

SELENIUM_PORT The port of the Selenium server

SELENIUM_PLATFORM The operating system of the selected browser

SELENIUM_VERSION The version number of the selected browser

SELENIUM_BROWSER The name of the selected browser

SELENIUM_DRIVER Contains the operating system, version and browser name of the selected browser, in a format designed for use by the Selenium Client Factory

SELENIUM_URL The initial URL to load when the test begins

SAUCE_USERNAME The user name used to invoke Sauce OnDemand

SAUCE_ACCESS_KEY The access key for the user used to invoke Sauce OnDemand

SELENIUM_STARTING_URL The value of the Starting URL field

SAUCE_ONDEMAND_BROWSERS A JSON-formatted string representing browsers you selected for the job configuration, as described Setting Desired Capabilities for Jenkins Projects

SAUCE_BUILDNUMBER The build number

Running Parallel Tests with Jenkins

There are three ways you can run tests in parallel as part of a Jenkins build.

1. Configure your test script to run tests in parallel following the examples for your scripting language. 2. Set up your Jenkins project as a Matrix Project. 3. Configure you Jenkins build as a Parameterized Build.

Configuring Jenkins Matrix Projects with Sauce

Jenkins matrix projects, also know as multi-configuration projects, allow you to run the same build with different input parameters. The Sauce plugin for Jenkins provides an additional option for multi-configuration projects to specify the browser combination for each build.

1. When setting up your job in Jenkins, select Multi-Configuration Project, and then click OK. 2. under Configuration Matrix, click Add Axis. 3. Select the type of test you want to run with Sauce, for example Sauce OnDemand WebDriver tests. The option you select will determine the list of operating systems and browsers you can choose in the next step. 4. Select the operating systems and browser combinations that you want to test against. A separate job will run for each OS and browser combination that you select. The environment variables for SELENIUM_PLATFORM, SEL ENIUM_VERSION, SELENIUM_BROWSER, and SELENIUM_DRIVER will be populated for each job with your selections.