Tricentis Continuous Testing Platform Mike Schlabinger, VP Global Sales Enablement 1 What is Continuous Testing? Continuous Testing is the process of executing automated tests as part of the software delivery pipeline in order to obtain feedback on the business risks associated with a software release as rapidly as possible. • Right Feedback, Right Time, for the Right Stakeholder • Can happen at any point in the SDLC 4 Tricentis Continuous Testing Platform Agile Dev Testing Automated Distributed Continuous Testing Load Testing Scalable, in-sprint test Resilient regression testing Cloud-based performance labs management for open source across any architecture or at the fingertips of every test automation, exploratory application stack at the speed developer and tester for on- testing and BDD. of change. demand load testing. Software Testing Reinvented for Agile and DevOps 5 Tricentis Continuous Testing Platform Agile Dev Testing Automated Distributed Continuous Testing Load Testing Open source testing and agile Resilient regression testing at Cloud-based performance test management the speed of change testing labs on-demand Software Testing Reinvented for Agile and DevOps 6 Tricentis Continuous Testing Platform Tricentis Analytics Cross Project and Portfolio Visibility Agile Dev Testing Automated Distributed Continuous Testing Load Testing Open source testing and agile Resilient regression testing at Cloud-based performance test management the speed of change testing labs on-demand 7 Tricentis Continuous Testing Platform Agile Dev Testing Automated Distributed Continuous Testing Load Testing 2x 2-3x 30-40% Risk Coverage Release Speed Cost Reduction 10x Performance Compared to other testing tools 8 Metrics Tricentis Analytics Joint Analytics & Reporting Custom Dashboard qTest Tosca Reporting Analytics Projects Projects Database Dashboard Cross-Project Reporting 9 Integration Roadmap 1 Analytics 12.0, Oct 2018 Presented at Tricentis Accelerate Conference Unified Dashboard 1 2 in Vienna Agile Dev Automated 12.1, Q1 2019 Public availability Testing Continuous Testing 3 Exploratory Testing 2 UI Automation Test Management 12.2, Q2 2019 Public availability Risk-based Testing and Manage Automate API Testing Manual Testing integrate Test Case Design shared qTest and Tosca Risk Based PrioritizationRisk Based Prioritization 3 Test Case Design Test Case Design Optimize Optimize 4 13.0, Q3/Q4 2019 Public availability Active Test Data ManagementActive Test Data Management 4 Active TDM shared qTest and Tosca Test Driven Service Virtualization Provide Provide Internal, Subject to Change 10 DEVOPS TECHNOLOGY OPEN Expansive Technology Support Technology Expansive SUPPORT SUPPORT SOURCE RPG 11 Make open source tools more productive • Central,Software agile test management • OrchestrateTesting execution across tools • ConsolidatedReinvented analytics and reporting for DevOps 12 Language and legacy technology support Software RPG • SingleTesting interface for multiple languages • Model-Based Test Automation for rapid Reinventedcreation, management and for execution • ConsolidatedDevOps analytics and reporting 13 Industry’s broadest support for PackagedSoftware Applications Testing • Business focused interface for ease of use Reinvented for • Model-Based Test Automation for rapid DevOpscreation, management and execution • Consolidated analytics and reporting 14 Complete support for SAP Software SAP WinGUI SAP Fiori • Single,Testing business interface for all SAP UIs • Model-Based Test Automation for rapid Reinventedcreation, management and for execution • ResilientDevOps end-to-end tests across technologies 15 Agile Dev Testing • Agile Test Management by providing manual and automated test traceability to Jira issues,Software bugs, and releases in real-time • Shift QualityTesting Left with open source test automation management, test script scheduling,Reinvented and support for BDD for • PredictiveDevOps Test Analytics to gauge release readiness through executive dashboards & portfolio level reporting 17 Automated Continuous Testing • Continuous Testing with model-based test automation and support for open source Softwaretools throughout the DevTest processTesting • Central Platform with seamless integrationReinvented into DevOps toolchains for for developers, testersDevOps and business users • High Speed Releases by optimizing and orchestrating end-to-end tests across teams, projects and applications 18 Distributed Load Testing • Easy Test Creation with ability to create tests inSoftware open source and proprietary tools for performanceTesting testers of all skill levels • Flexible, Massive Scale by leveraging the cloud Reinventedthrough shared or private AWSfor and Azure integrations that fits into your DevOpsDevOps pipeline • Rich Bottleneck Analysis that highlights areas of risk for quick resolution of issues 19 PLAN DESIGN BUILD CONTINUOUS TESTING DEPLOY MONITOR Plan Code & Test Design Version Build Provision Test in Sandbox Deploy to Monitor & Stage RPA Integrate Integrate Integrate Integrate Report, Prioritize Explore Virtualize Distribute Automate & Integrate Design Manage Automate Execute Configure 23 A Sprint in a Tester’s Life As a User I want to view customer data (UI) DEVs As UI-User I want to retrieve customer data via json service (Middleware) R Epic / User Stories created E P Load Test Exploratory Testing O PO User Stories (de-)refined R T Test Data Management Acceptance criteria (ATDD/BDD) aligned I N Model based UI Test, Web Desktop & Mobile Risk & TCD G Model-based API Test GUI Load Test Model based SV – Test QA Refinement Sprint Provide Test Artifacts as “Self-Service“ in & across teams 24 About Tricentis 26 Top Analyst Recognitions “Tricentis‘ Tosca Testsuite makes “It should be considered by enterprises that automation easy with model-based have struggled to make test automation automation. Via Tosca Testsuite, Tricentis work, and by those seeking to support agile provides top test automation and continuous automation practices.” optimization design capabilities, test asset reuse and combined automation.“ 30 Dec 2018 Nov 2017 Dec 2015 Aug 2014 Jan 2011 31 Dec 2018 Nov 2017 Dec 2015 Aug 2014 “It should be considered by enterprises that have struggled to make test automation work, and by those seekingJan 2011 to support agile continuous automation practices.” 32 Awards & Accolades #44 33 Agile testing for Continuous testing for Continuous Testing core insurance system, online banking system through a Digital TCoE, 4.5M lines of code all core systems 19K tests → 3.5K tests 2K tests → 0.6K tests 12K tests → 4.2K tests (-82%) (-70%) (-65%) 46% → 93% 30% → 85% 43% → 92% (2x) (2.8x) (2.2x) 4 weeks → 3 days 120 hours → 30 min 6 weeks → 5 days (9x) (240x) (8.4x) 34 Bringing standardization Providing cutting edge Moving from Legacy to 600+ agile teams with digital experiences for QC/ALM to modern agile 30+ automation tools 65 million active users practices Clear visibility into manual Standardized all Best of breed approach for to automated progress for hardware and digital agile transform using Jira executives to see QA value app testing across qTest Software + qTest Centralized reporting Launched PlayStation Store Migrated 25 large scale across 30+ automation by centralizing QA results projects including test cases, tools and frameworks with integrating to Jira plans, linked traceability More than 400+ test Doubled QA licenses of Scalable users across automation agents qTest in 2nd year to multiple NetApp divisions connected for a single view support global teams in will millions of test results of application health Japan and California from home grown test 35 automation Transformed TCOE from Scaling extreme test Providing digital Legacy ALM to modern automation for API, SDK, customer experiences CI/CD model UI, & performance across web, and mobile Reduce test maintenance Scaled monthly test Compliance through time through trend activity to 153 platforms centralizing QA results analysis tested, 422 builds & and integrating with Jira 300,000 test runs Increased test automation Reduced a test run Decreased average to 83% reporting from hours to regression testing time minutes from 2 days to 2 hours Substantially reduced Scaled number of test Scaled open source test critical production defects runs executed from 45 to automation from hundreds from 15 avg to 0 to thousands 7,000 per minute 36 Testing Present Testers Coverage 100% Manual Testing 11 ? Manual Unknown There’s a way to do it Test Cases Execution better. Find it! Project Charter 0% Automated 4755 10 Testing Manual Weeks Customer Case Study Risk Coverage Optimization Test Data Management Automation (UI & API) & OSV Continuous Integration & Distributed Execution Test Cases Testers Coverage Coverage Coverage Execution Test Case Count 7% 92% 11 37? % 89% 53% 34 Automated Manual BusinessUnknown Risk Business Risk Business Risk Minutes 64% API Tests Smoke Testing Smoke Testing Test Cases Test Cases Execution Execution Execution Execution 1193 4755 10 5 2.5 488 Manual Manual Weeks Weeks Weeks Hours 75% Redundancy 50% Effort Test Data MultipleSingle Agent Agents Customer Case Study Risk Coverage Optimization Test Data Management Automation (UI & API) & OSV Continuous Integration & Distributed Execution Test Cases Production Defects Coverage Coverage Execution Test Case Count 7% 92% 89% 53% 34 Automated Business Risk Business Risk Minutes 64% API Tests Smoke Testing Smoke Testing 72% Test Cases Critical Defect Reduction Execution Execution Execution 1193 5 2.5 8 Manual Weeks Weeks Hours Test Case Design & Exploratory Testing Multiple Agents Customer Case Study RMS Case Study SAP Within five months a team of three implemented the core regression suite, including 333 business scenarios across various business functions such as Financial Processes (Accounts Payable, Accounts Receivable, General Ledger), Procurement, Controlling, HR, Project related Processes, and automatic Invoice Processing, covering 34,237 test actions covering 6,734 verification points. The total execution of the SAP automated regression suite takes less than 24 hours; this would take 518 hours to execute manually. By running the automation suite the total effort is reduced by 96%. Questions? 42.