DevOps: A Testers Best Friend CHICAGO April 18th — April 22th Renaissance Hotel 1 West Wacker Drive Chicago IL 60601 Speaker(s): Ken Robertson & Thomas Janik When: April 21, 2016 Company: American Family Insurance Time: 3:30 pm - 4:30 pm DEVOPS; A TESTER’S BEST FRIEND Press ESC to continue DEVOPS; A TESTER’S BEST FRIEND TOM JANIK – AMERICAN FAMILY INSURANCE DEVOPS MANAGER – RELEASE MANAGEMENT KEN ROBERTSON – AMERICAN FAMILY INSURANCE SQA TEST MANAGER – SPECIALIZED TESTING OBJECTIVE OF SESSION: WALK THROUGH THE AMERICAN FAMILY JOURNEY TO BECOMING A COMPANY THAT OPERATES IN A CULTURE BASED IN DEVOPS IN THE BEGINNING: RELEASE FLOW DEVELOPMENT LEG DEV INT SIT LEG DEV INT QA RELEASE LEG DEV INT QA PERF PROD Deploys done by development teams, could happen at any time No stability requirements, no triage of deployment / environment issues 4 year development cycle using Amgile SDLC (selective Agile) Development “Leg” (flow), SIT “Leg”, 2 month effort in Release “Leg” IN THE BEGINNING: SIT TESTING LOCAL/DEV Development and Unit Testing DEV/INT Functional Testing QA Test Data Creation TDC SIT TDC Elevation 1 Elevation 2 Elevation … n Manual testing Testing led by development teams, lacked integration coordination 2 year SIT cycle with multiple code elevations No formalized automation or performance testing No shared test data, test scenarios, or documented financial testing Limited communication between development and testing – throw it over the wall mentality FORMING: THE DEVOPS REVOLUTION DEVELOPMENT LEG 1 DEV INT QA DEVELOPMENT LEG 2 DEV INT QA DEVELOPMENT LEG 3 DEV INT QA RELEASE LEG DEV INT QA PERF PROD D: Release Coordination, Automation, Performance, and Manual Testing Teams O: Change Management, Build and Deploy, Software Configuration Management, Infrastructure Coordination Teams Specific deployment windows, DENT environments, Floating QA environment Branching and merging strategies begin, targeted deploys to PERF (testing left) 3 parallel development (“Legs”) streams, Monthly maintenance release “Leg” Press ESC to continue FORMING: SIT TESTING CYCLES LOCAL/DEV Development and Unit Testing DEV/INT/DENT Functional Testing INT AIT (Application Integration Testing) Phase 1 Phase 2 Phase 3 QA HIST Test Data Creation QA SIT (System Integration Testing) Cycle 1 Cycle 2 Cycle n... Introduction of DENT and QA HIST tiers Point to point integration testing coordinated by development teams Application integration coordination being introduced 6 month SIT duration – 3 months financial infrastructure testing and 3 months for financials Documented financial testing in collaboration with controllers ACTIVITY: DEVOPS WHAT’S IMPORTANT TO YOU? Using the index card in front of you write down 3 things that are most important to you about DevOps. (1:00 Minute) Gather in groups of #-# at one of the posters on the wall. (1:00 Minute) Come up with something common between everyone in the group and write that on top of the poster as your group name. (1:00 Minute) Come up with one common thing from everyone’s index cards and draw (no words) that one common thing on the rest on the space on the poster. (1:00 Minute) STORMING: A CULTURE CHANGE TO DEVOPS DevOps is the practice of operations and development engineers participating together in the entire service lifecycle: from design, through the development process, and then into production support Specifically DevOps is characterized by: Operations staff making use many of the same techniques as developers for their systems work Developers incorporating more operational concerns in their planning and coding (designing for operationalization) DevOps can be a team, a department, an effort, a movement, but most importantly it is a culture! KEY TENETS OF DEVOPS Operate Plan FEEDBACK Theory of Constraints (“The Goal”) Release Architect The 3 Ways (“The Phoenix Project”): Systems thinking Amplified feedback loops Test Dev Continuous experimentation and learning Lean principles of manufacturing (“TPS: Toyota Production System”) People, then process, then tools (Men, Methods, Machines, Measurements) Collaborative development (Scrum, Agile, Kanban) Environment provisioning, quicker access to new code for testing Continuous deployment, integration, testing, delivery, feedback, and improvement THE NEW DEVOPS CULTURE Automation Specialty Testing Testing Quality Assurance Release Release Management Automation Data Services Team Team DevOps Environment Application Management Reliability Planning (Enterprise) Agile Project Small Increased Speed to Defect Minimal Viable Build Quality Into Tracking Batches Frequency Resolution Product the Process Mapping Architecture, Design & Development Value Stream Stream Value Developer Application Design for Operations Loosely Coupled Micro Branching by Commit Instrumentation Incl. Graceful Degradation Architectures services Abstraction /Toggling The Automated Delivery Pipeline (DevOps) Continuous Environment Deployment Automated Testing Production Feedback Standardization Integration Management Orchestration Deployment Mechanisms Layered Testing (GUI, API, Version Baselined / Automated Backend) Deployment Dashboards/ Build Automation Modelled Deployment Full System Testing Phased Rollout Scorecards Basic Test (incl. Environments Development (Functional, UX Rollback Plans Trends static code quality, Test Data Integration Verification, Regression, APM & Analytics Reports Lean security, Propagation Acceptance Security, Performance, Communication Notifications Principles performance, and Automatic Performance Compatibility, ALDM code coverage) Configuration Production Acceptance) Monitor/Measure Package Infrastructure-as- On-Demand Environment Sanity Tests Everything Deploy to Software Code Environment Application Resiliency Continuous Process Asset Repository Automation Provisioning Testing (Chaos Monkey, etc.) Improvement Governance Checks On-Premises/ Non-Functional Sanity Common Metrics Feedback Cloud/Hybrid Tests Measuring Quality Infrastructure Resiliency Removal of of Removal Mechanism Virtualization Measuring Process Debt Process Infrastructure Loosely Coupled Testing Efficiency Integrations Architectures Visible Status Continuous Continuous Trust Dev in Ops Ops in Dev Common Goals based on Experimentation Improvement Customer Satisfaction Removal of of Removal Collaboration / Break Collective Blameless Openness & Technical Debt Technical Never Done Down Silos Ownership Culture (Foundational) Postmortem Visibility Planning (Enterprise) Agile Project Small Increased Speed to Defect Minimal Viable Build Quality Into Tracking Batches Frequency Resolution Product the Process Mapping Architecture, Design & Development Value Stream Stream Value Developer Application Design for Operations Loosely Coupled Micro Branching by Commit Instrumentation Incl. Graceful Degradation Architectures services Abstraction /Toggling The Automated Delivery Pipeline (DevOps) Continuous Environment Deployment Automated Testing Production Feedback Standardization Integration Management Orchestration Deployment Mechanisms Layered Testing (GUI, API, Version Baselined / Automated Backend) Deployment Dashboards/ Build Automation Modelled Deployment Full System Testing Phased Rollout Scorecards Basic Test (incl. Environments Development (Functional, UX Rollback Plans Trends static code quality, Test Data Integration Verification, Regression, APM & Analytics Reports Lean security, Propagation Acceptance Security, Performance, Communication Notifications Principles performance, and Automatic Performance Compatibility, ALDM code coverage) Configuration Production Acceptance) Monitor/Measure Package Infrastructure-as- On-Demand Environment Sanity Tests Everything Deploy to Software Code Environment Application Resiliency Continuous Process Asset Repository Automation Provisioning Testing (Chaos Monkey, etc.) Improvement Governance Checks On-Premises/ Non-Functional Sanity Common Metrics Feedback Cloud/Hybrid Tests Measuring Quality Infrastructure Resiliency Removal of of Removal Mechanism Virtualization Measuring Process Debt Process Infrastructure Loosely Coupled Testing Efficiency Integrations Architectures Visible Status Continuous Continuous Trust Dev in Ops Ops in Dev Common Goals based on Experimentation Improvement Customer Satisfaction Removal of of Removal Collaboration / Break Collective Blameless Openness & Technical Debt Technical Never Done Down Silos Ownership Culture (Foundational) Postmortem Visibility NORMING: WHO DOES WHAT? The Automated Delivery Pipeline (DevOps) Continuous Environment Deployment Automated Testing Production Feedback Integration Management Orchestration Deployment Mechanisms Layered Testing (GUI, API, Version Baselined / Automated Backend) Deployment Dashboards/ Build Automation Modelled Deployment Full System Testing Phased Rollout Scorecards Basic Test (incl. Environments Development (Functional, UX Rollback Plans Trends static code quality, Test Data Integration Verification, Regression, APM & Analytics Reports security, Propagation Acceptance Security, Performance, Communication Notifications performance, and Automatic Performance Compatibility, ALDM code coverage) Configuration Production Acceptance) Monitor/Measure Package Infrastructure-as- On-Demand Environment Sanity Tests Everything Deploy to Software Code Environment Application Resiliency Continuous Process Asset Repository Automation Provisioning Testing (Chaos Monkey, etc.) Improvement