Quality Engineering in Devops World – a Strategic Enabler

Quality Engineering in DevOps world – a Strategic Enabler

» DevOps in a nutshell

» DevOps vs DevTestOps

» Continuous Testing & Delivery leveraging QE

» Unit Testing & SAST Agenda » Automation , Performance & Security

» Service Virtualization

» Continuous Integration , Deployments & Delivery

» Operations Stabilization in Devops

» Continuous Automation & Delivery Pipeline

» Collaborative Platform & How does it help?

To date, Gartner has had more than 3,500 client interactions (inquiries, best-practice calls and conference one-on-one meetings) discussing DevOps awareness, initiatives, successes and failures

Forrester - Development and operations (DevOps), the purview of unicorns, is now transitioning to the mainstream. Business technology (BT) leaders are realizing that Agile is no longer enough; they need to involve the full technology management organization to drive competitiveness in an era when customer centricity is pivotal to success

DevOps toolchains are often built from discrete Orchestrating the DevOps toolchain can be and sometimes disconnected tools, making it compelling, but it can introduce another set of difficult to understand where bottlenecks are challenges in the application delivery pipeline

02 03

01 04

Toolchain orchestration promises to simplify DevOps initiatives continue to grow in the DevOps toolchain; however, one tool is not complexity, driving many IT organizations to able to support the entire toolchain resulting in seek tools that provide greater control and additional integration, ownership and handoff visibility over the entire DevOps process challenges when using multiple orchestration tools

Culture and Release velocity is pivotal There’s no “easy” button people come First; process follows

Source: Forrester’s Q1 2017 Global DevOps Benchmark Online Survey Source: Forrester Data Global Business Technographics Developer Survey, 2016 & Forrester Data Global Business Technographics Infrastructure Survey, 2016 www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 6 DevOps Adoption

Public Sector And Healthcare cling To Manual Audit Utilities And Telecommunications And compliance Procedures Lead The Pack To Drive Business Compliance-wary companies need not fear automation Transformation Data must drive strategy, with services Financial Services Accelerate To compete $ that emphasize connectivity With Fintechs DevOps offers key benefits in fraud detection, time-to-market, and scalability

Media, entertainment, & Manufacturing Transitions Leisure embark On The Beyond Automation To Business Services And construction Journey To Differentiate Show DevOps Momentum Ponder The Leap Into Automation With DevOps Technology is becoming part of the business

Manufacturers improve Retail And Wholesale DevOps Adoption Drive DevOps drives to improve efficiency and insight by Differentiated Interactions With clients customer experience expanding the role of Get ready for the next generation of digital stores technology www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 9 DevOps vs. DevTestOps DevOps DevTestOps

Equal Importance for Dev, Operations and Testing

All actions – Development --> Operations Continuous Testing

Emphasis on Automation & Analytics End to End Automation Continuous Integration + Continuous Deployment (CI/CD) Overall Cost Reduction – Automation, Dev --> Ops Process Reduced from Reduce Cost of Quality Months to Days/Hours

Continuous Feedback & Improvement Fail – First, Fail - Fast, Fail- Often – Early Supported Defects Detection

Maturing Processes and Organizations Focus Collaborative Communication utilizing Dashboard www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 10 Devops in a nutshell

Continuous Testing Automated Unit Build Packing & Testing & SAST Automated • Functional Automation Deployments • Non Functional Automation

Build Dev

Production Virtual Server(s) Deployments CI Server

Hosted Tools Post Production Infrastructure Physical Monitoring & Feedback Automation & server(s) Amplification operations

Centralised data management through Enabling continuous testing automated TDM tools & other mechanisms by overcoming the Leveraging a collaborative for on demand testing dependency factors involved dashboard to monitor application release Data Service activities & health Efficient Tools Engineering Virtualization Dashboard

Third party system Leverage

Target System

Client Layer Web Server App Server Component 1 Component 2 Component 3 Target System Application Under Test UI Ensuring faster feedback for Adopting Quality Engineering issues remediation through Services approach to provide end to end continuous infrastructure & App Component Continuous Testing automated functional and non – functional test coverage for application monitoring in Continuous Monitoring production & test environments each release Functional Automation Health Check CI Platform Non - Functional Automation Application Infrastructure {Through scripts} {Through metrics configuration} Ensuring quality during rapid and frequent build deployments by leveraging robust CI tools integrated with test automation frameworks

ROI Factors Cost Elements DevOps Journey Automation Maturity

» Build Reliability » Cost of Maintenance DevOps Build - Deployment – » Uninterrupted Service » Technology Adoption Release Automation Delivery

» DevOps Process Quality Engineering » Org. Change Mgmt. Matured Collaborative Implemented Dual Shift approach for early testing (People, Process and Systems) (Functional and Non Functional) Agile ( Dev, QA and Ops) Approach

» Tools and process for CI Transformed to TDD » Automation Stabilization and CD » Continuous Testing – / ATDD / BDD Adopted CI and » Infra Maintenance CD tools Continuous Feedback

» On Demand Test Envts. » Tool optimization and » End-to-End Automation consolidation and Test Coverage Matured Agile Functional and Non » Quality Check gates Organization Functional Automation » Tools for Automation and » Unified Automation Non Functional Tests Framework » Process Standardization No » Best suited tools Partial Agile Automation www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies in place 13 Key QE elements In Detail

Open Source/Commercial Tools JUNIT/NUNIT 1 1 2 Results Analysis & Manual Analysis IDE Unit Testing Static Code Analysis Vulnerable for false positive (Technology Agnostic) identification removal 3

CI Server

Code Repository Final Summary Build Server 2 Code Commit

» Unit testing is performed using respective open source tools that integrated with the IDE used for code development Secure Coding Guidelines & Standards » Input Validation » CI Server picks the build from the code repository/version controlling » Error handling / Logging » Output Encoding tool once the developer commits the build » Data Protection » Secure handling of credentials » Communication Security » Session Management » CI Server initiates the static code analysis by triggering a request to » Memory management » Cryptographic practices the configured open source / commercial tools

Test Automation Test Management Performance Testing Test Automation Tools Framework

Trigger Automated Performance Testing Scenarios Test Automation Test Execution Tools Performance Testing Tools

Version Trigger Monitoring High Level Build CI Platform Controlling Tool Analysis Report

Security Testing (DAST) Monitoring DAST Tools Tools Automated Results Analysis & Trigger Vulnerabilities Identification Monitoring Tools Final Manual Analysis Summary (False Positives) Report www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 16 TDD blended with DevOps

Automated Configuration Management frameworks DevOps Analytics Platform Feature Spec Continuous file (Ansible/Chef/Puppet) Deployment

Feature broken Continuous Feedback DevOps Specific into functions input to feature CI/CD framework D e vrequirements O p s (Continuous Integration/ Continuous Delivery)

Unit Testing, Integration Add a Developer Function 1- Unit Tests and Performance test Test case Tests in reference to Pass, development under Functionality DevOps practices. Incomplete Function 2- Unit Pass Run Developer

TDD Test case

tests at the unit level Finalized Finalized Feature

Fail

Function 3 - Unit Fail Test case Run the Make code changes Pass, Developer tests Development Ends Functionality complete www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 17 Service Virtualization in AWS Cloud Advantages

✓ Configurations ✓ Accessible from Multiple Geographies (Cloud & On-premise) Legacy Deploy SV tool ✓ Deploy & Bring Up/down to Optimize Resources & Cloud Spend on cloud

✓ Easily Customized as the UI or Business Needs Change Service Service Request Service Service Response ✓ Leverage Docker Containers, enabling Continuous Integration of ✓ Configurations Code Changes and Parallel Test Instances Service Request DevTest Service Response ERP ✓ Reduce CapEx (Capital Expenditure) by Hosting in AWS (Amazon Service Request Service Response Web Services) Cloud ✓T Dynamically Support Infrastructure & Policy as Code, with Fugue… ✓ Configurations Create Virtual Services Databases » Automatically Capture realistic

behavior Service Service Request Service Service Response Target Service » Build VS Image, collection Typical DevTest SV Usage Scenario

services requests and Service SOAP Request responses Test Exercise SUT System Under SOAP Response Virtual Service ✓ Configurations (Manual, Automated) Test (SUT) » Optimize models as desired JMS Publish DevTest Mainframe Playback against Virtual Service

C o n t i n u o u s - Integration vs. Delivery vs. Deployment

Continuous V1.1 Integration

Approve Deploy Continuous Delivery Automatic Deploy Continuous Deployment

Automated Automated

Source Control Build Staging Production Commit Changes Run Build And Unit Tests Deploy to test Environment Run Integration Deploy to Production Environment Tests, Load Test, & Other Tests www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 19 What tools/technologies should QE team get trained

Though Selenium continue to dominate, the With DevOps, The following tools have been adopted faster need for

Web Driver IO Protractor Concourse Microservices Server 1 Server 2 Server 3 REPL is used to build web driver component testing with spring server technologies Brief list of various Test Automation tools that will take higher Service have been in faster adoption precedence

By 2019, Spring Server will dominate the server market https://concourse.ci/ - This https://gradle.org/ http://gulpjs.com/ - Supports is better than Jenkins - Shell Alternate to Maven Jasmine, Mocha - BDD tools - space script - runs on a FLOW. This is better than Cucumber (https://github.com/bartobri/spring- server) Webdriver CS http://webdriver.io/ Selenium automate just about Visual Regression GitHub will become anything with a runner - reports in Allure - very minimum of effort powerful one for integration. the global platform and open source will Concourse as a server. Enables multiple deployment for seamless test execution be the future. www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 20 Operations Stabilization & Alignment with DevOps

Automated Deployments Release Staging Deployments DevOps Enabled Automation QA Smoke Tests Test Lab Dev 1

Rapid Configuration Management Virtual Server(s) Configuration Multiple Nodes across Management technologies CI Server 2

Build Environments Staging Hosted Tools Environment Manage Virtualized & QA Physical Management Maintain Non - Virtualized Monitor Dev server(s) 3

Health Check Continuous Infrastructure Threshold Monitoring Application evaluation 4 www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 21 Continuous Automation & Delivery Pipeline

Test Automation Automated CI Enabled Scripts Results Testing Execution Regression & Risk Based

Automated Performance DAST Requirements Collaborative Dashboard Code SAST Continuous Development

Quality Unit testing Continuous Continuous Automation Coding Continuous Deployment Operations Release Enablement Deploy Apps Continuous Management Monitoring Compile & Build

On Demand Infra

ORCESTRAED CONTINIOUS INTEGRATION PIPE LINE Requirements Document & verify SCM Repo Build Package Deploy Host / Stage Adoption Test Jenki ns

Container as a SCM service using Rancher/Docker DevOps Ops Tools On demand test bed Reporting & Pipeline Monitoring Environment Environment using containers Test Automation Delivery Pipeline Delivery Advance Tools Resources Analytics Resources Enterprise Micro Web & Mobile Private Container Service service Stack Dashboards User Directory Container Registry browsers Tableau www.cigniti.com | Distribution is Restricted. Copyright © 2016 - 17, Cigniti Technologies 22 Collaborative DevTest Platform & How does it help?

Build Integrate Deploy Operate

Release Progression Build Status Test Status Predict Analysis Live Live Dashboard Application Health Release Readiness Quality Metrics Go – No Go Decision Enabler