Devops; a Testers Best Friend

Home , Functional testing, Integration testing, Release management

DevOps: A Testers Best Friend CHICAGO April 18th — April 22th Renaissance Hotel 1 West Wacker Drive Chicago IL 60601

Speaker(s): Ken Robertson & Thomas Janik When: April 21, 2016 Company: American Family Insurance Time: 3:30 pm - 4:30 pm DEVOPS; A TESTER’S BEST FRIEND

Press ESC to continue DEVOPS; A TESTER’S BEST FRIEND

TOM JANIK – AMERICAN FAMILY INSURANCE DEVOPS MANAGER – RELEASE MANAGEMENT KEN ROBERTSON – AMERICAN FAMILY INSURANCE SQA TEST MANAGER – SPECIALIZED TESTING

OBJECTIVE OF SESSION: WALK THROUGH THE AMERICAN FAMILY JOURNEY TO BECOMING A COMPANY THAT OPERATES IN A CULTURE BASED IN DEVOPS IN THE BEGINNING: RELEASE FLOW DEVELOPMENT LEG DEV INT

SIT LEG DEV INT QA

RELEASE LEG DEV INT QA PERF PROD

Deploys done by development teams, could happen at any time No stability requirements, no triage of deployment / environment issues 4 year development cycle using Amgile SDLC (selective Agile) Development “Leg” (flow), SIT “Leg”, 2 month effort in Release “Leg” IN THE BEGINNING: SIT TESTING

LOCAL/DEV Development and Unit Testing DEV/INT Functional Testing

QA Test Data Creation TDC SIT TDC Elevation 1 Elevation 2 Elevation … n

Manual testing Testing led by development teams, lacked integration coordination 2 year SIT cycle with multiple code elevations No formalized automation or performance testing No shared test data, test scenarios, or documented financial testing Limited communication between development and testing – throw it over the wall mentality FORMING: THE DEVOPS REVOLUTION

DEVELOPMENT LEG 1 DEV INT QA DEVELOPMENT LEG 2 DEV INT QA DEVELOPMENT LEG 3 DEV INT QA

RELEASE LEG DEV INT QA PERF PROD

D: Release Coordination, Automation, Performance, and Manual Testing Teams O: Change Management, Build and Deploy, Software Configuration Management, Infrastructure Coordination Teams Specific deployment windows, DENT environments, Floating QA environment Branching and merging strategies begin, targeted deploys to PERF (testing left) 3 parallel development (“Legs”) streams, Monthly maintenance release “Leg”

Press ESC to continue FORMING: SIT TESTING CYCLES

LOCAL/DEV Development and Unit Testing

DEV/INT/DENT Functional Testing INT AIT (Application Integration Testing) Phase 1 Phase 2 Phase 3 QA HIST Test Data Creation QA SIT (System Integration Testing) Cycle 1 Cycle 2 Cycle n... Introduction of DENT and QA HIST tiers Point to point integration testing coordinated by development teams Application integration coordination being introduced 6 month SIT duration – 3 months financial infrastructure testing and 3 months for financials Documented financial testing in collaboration with controllers ACTIVITY: DEVOPS WHAT’S IMPORTANT TO YOU?

Using the index card in front of you write down 3 things that are most important to you about DevOps. (1:00 Minute)

Gather in groups of #-# at one of the posters on the wall. (1:00 Minute)

Come up with something common between everyone in the group and write that on top of the poster as your group name. (1:00 Minute)

Come up with one common thing from everyone’s index cards and draw (no words) that one common thing on the rest on the space on the poster. (1:00 Minute) STORMING: A CULTURE CHANGE TO DEVOPS

DevOps is the practice of operations and development engineers participating together in the entire service lifecycle: from design, through the development process, and then into production support Specifically DevOps is characterized by:

 Operations staff making use many of the same techniques as developers for their systems work

 Developers incorporating more operational concerns in their planning and coding (designing for operationalization) DevOps can be a team, a department, an effort, a movement, but most importantly it is a culture! KEY TENETS OF DEVOPS Operate Plan

FEEDBACK Theory of Constraints (“The Goal”) Release Architect The 3 Ways (“The Phoenix Project”):

 Systems thinking

 Amplified feedback loops Test Dev

 Continuous experimentation and learning Lean principles of manufacturing (“TPS: Toyota Production System”) People, then process, then tools (Men, Methods, Machines, Measurements) Collaborative development (Scrum, Agile, Kanban) Environment provisioning, quicker access to new code for testing Continuous deployment, integration, testing, delivery, feedback, and improvement THE NEW DEVOPS CULTURE

Automation Specialty Testing Testing Quality Assurance Release Release Management Automation Data Services Team Team DevOps

Environment Application Management Reliability Removal of Removal of Lean Value Stream Standardization Technical Debt Process Debt Principles Mapping

  

 



Collaboration

Version Build Automation Build Basic Test Basic static code quality code static security performance code coverage code Package Deploy to Software to Software Deploy Asset Repository Asset Governance Checks Governance Feedback Feedback Mechanism Infrastructure Infrastructure Integrations

Trust

Continuous Continuous

Integration

Developer

DownSilos

Commit

Agile ProjectAgile

Tracking

, ,

(

incl

)

. .

and and

Break

Dev in Dev Ops

, ,

Instrumentation



 

Application

Management

Environment

Baselined Baselined Modelled Modelled Environments Test Data Data Test Propagation Automatic Automatic Configuration Infrastructure Code Automation On Cloud Virtualization Loosely Coupled Coupled Loosely Architectures

Batches

Premises

Ownership

Small

Collective

Hybrid

Ops in DevOpsin

/ /

Design Operationsfor

Incl



     

. .

The The Automated PipelineDelivery

Orchestration

Deployment

Graceful Degradation Graceful

Automated Automated Deployment Development Integration Acceptance Performance Production On Environment Environment Provisioning

Frequency

Architecture

Increased

Demand Demand

Common Goals based on

Customer Satisfaction

Culture

Planning

, ,

(

Design Design

Foundational



 



(

Enterprise

Automated Automated Testing

Layered Testing Testing Layered Backend Full System Testing Testing System Full ( Verification Security Compatibility Acceptance Environment Sanity Tests Sanity Environment Application Resiliency Resiliency Application Testing Testing Non Tests Infrastructure Resiliency Resiliency Infrastructure Testing Testing

Speed to Defect

Functional

& &

Loosely Coupled

Development

Resolution

Architectures

Functional Sanity Sanity Functional

(

, ,

)

ChaosMonkey

) )

Performance

)

, ,

)

Regression

(

, ,

DevOps

(

GUI

, ,

etc

Postmortem

API

, ,

Blameless

)

, ,

Experimentation

, ,

Continuous

Minimal Viable

    

services

Micro

Product

Deployment

Production

Deployment Deployment Phased Rollout Phased Rollback Plans Rollback APM APM Communication

& &

Analytics

NeverDone

Abstraction

Branchingby

Build IntoQuality



    



  



Improvement

Continuous

the the Process

Dashboards Scorecards Trends Reports Notifications ALDM Monitor Everything Continuous Process Process Continuous Improvement Common Metrics Common Measuring Quality Measuring Measuring Measuring Efficiency Visible Status Visible

Mechanisms

Feedback

Openness

Toggling

Visibility

Measure Measure

& Removal of Removal of Lean Value Stream Standardization Technical Debt Process Debt Principles Mapping

  

 



Collaboration

Trust

Continuous Continuous

Integration

Developer

DownSilos

Commit

Agile ProjectAgile

Tracking

, ,

(

incl

)

. .

and and

Break

Dev in Dev Ops

, ,

Instrumentation



 

Application

Management

Environment

Batches

Premises

Ownership

Small

Collective

Hybrid

Ops in DevOpsin

/ /

Design Operationsfor

Incl



     

. .

The The Automated PipelineDelivery

Orchestration

Deployment

Graceful Degradation Graceful

Automated Automated Deployment Development Integration Acceptance Performance Production On Environment Environment Provisioning

Frequency

Architecture

Increased

Demand Demand

Common Goals based on

Customer Satisfaction

Culture

Planning

, ,

(

Design Design

Foundational



 



(

Enterprise

Automated Automated Testing

Speed to Defect

Functional

& &

Loosely Coupled

Development

Resolution

Architectures

Functional Sanity Sanity Functional

(

, ,

)

ChaosMonkey

) )

Performance

)

, ,

)

Regression

(

, ,

DevOps

(

GUI

, ,

etc

Postmortem

API

, ,

Blameless

)

, ,

Experimentation

, ,

Continuous

Minimal Viable

    

services

Micro

Product

Deployment

Production

Deployment Deployment Phased Rollout Phased Rollback Plans Rollback APM APM Communication

& &

Analytics

NeverDone

Abstraction

Branchingby

Build IntoQuality



    



  



Improvement

Continuous

the the Process

Mechanisms

Feedback

Openness

Toggling

Visibility

Measure Measure

& NORMING: WHO DOES WHAT?

The Automated Delivery Pipeline (DevOps)

Continuous Environment Deployment Automated Testing Production Feedback Integration Management Orchestration Deployment Mechanisms  Layered Testing (GUI, API,  Version  Baselined /  Automated Backend)  Deployment  Dashboards/  Build Automation Modelled Deployment  Full System Testing  Phased Rollout Scorecards  Basic Test (incl. Environments  Development (Functional, UX  Rollback Plans  Trends static code quality,  Test Data  Integration Verification, Regression,  APM & Analytics  Reports security, Propagation  Acceptance Security, Performance,  Communication  Notifications performance, and  Automatic  Performance Compatibility,  ALDM code coverage) Configuration  Production Acceptance)  Monitor/Measure  Package  Infrastructure-as-  On-Demand  Environment Sanity Tests Everything  Deploy to Software Code Environment  Application Resiliency  Continuous Process Asset Repository  Automation Provisioning Testing (Chaos Monkey, etc.) Improvement  Governance Checks  On-Premises/  Non-Functional Sanity  Common Metrics  Feedback Cloud/Hybrid Tests  Measuring Quality Mechanism  Virtualization  Infrastructure Resiliency  Measuring  Infrastructure  Loosely Coupled Testing Efficiency Integrations Architectures  Visible Status

RELEASEENVIRONMENT AUTOMATIONRELEASE MANAGEMENT MANAGEMENTQUALITY ASSURANCEAPPLICATION RELIABILITY NORMING

DEV INT QA

Release management, Better defined schedules 2 week release activity cycle, Bi-monthly maintenance releases Overlapping release cycles, Same branching strategy as development legs NORMING: QUARTERLY RELEASES

Summer CTP Code Freeze Prod Performance Testing P2 Development and Functional Testing SIT Testing Release Data Creation Functional Final Code Code Freeze DEV m INT QA Freeze

2 2 Weeks

6 Months NORMING: FULL RELEASE SCHEDULE

6 Months DEVELOPMENT LEG 1

SummerSummer Winter

DEVELOPMENT LEG 2 Spring Fall Spring

RELEASE LEG

m mm m m m m M m m m m m M m m m m m M m m m m m M

2 NORMING: TESTING

LOCAL/DEV Development and Unit Testing

DEV/INT/DENT Functional Testing INT Functional Integration Testing Phase 1 Phase 2 Phase 3 QA HIST TDC QA SIT Cycle 1 Cycle 2 Cycle 3

Program level test lead for coordination of functional and integration testing Phased / cycle approach to functional integration and SIT testing 10 week SIT duration with data creation critical path constraint removed Introduction of forced month ends for financial testing TESTING BENFITS FROM DEVOPS

Continuous integration builds Gated deployments based on testing results Automated unit testing Automated smoke testing Automatic rollback for failed deploys Environment stability Quicker, higher quality code delivery Ability to feature toggle and/or deploy dormant code Automated deployment notification and release notes Better merging strategies ease regression testing requirements Exposure into development process NORMING: TOOLS Maven for configuration GIT for code management CA Harvest for build and deploy JUnit / GUnit for continuous integration testing Jenkins for automated deployments CA Dev Test (formerly Lisa) for automated testing Word for requirements (moving toward TFS) Excel for testing plan (moving toward TFS) Excel release plans Puppet for IaaS CA Service Desk for change management and configuration item repository Artifactory for Enterprise Data Management Solution (EDMS) QUESTIONS?

THANK YOU TOOL APPENDIX FOLLOWS APPENDIX A: TOOL RESEARCH

Tool Category Trial Site AccuRev (Micro Focus Software) Configuration Management No borland.com Agile Software Factory (Grid Dynamics) Application Release Automation No griddynamics.com Amazon ECS Container Management No aws.amazon.com Ansible (Red Hat) Configuration Management, Application Release Automation Yes ansible.com Apache Ant Build Management Yes ant.apache.org

Name Description Ansible An agent less orchestration engine that can automate application and systems. Artifactory JFrog’s open source binary repository management product with support for Apache Maven artifacts. Asgard A netflix web interface tool for AWS Cloud Management and application deployment. AWS A cloud computing platform offering ready to use web services which are loosely coupled. Awsbox A featherweight DYI PaaS on top of awscloud EC2 for deploying node apps.

http://www.devopsbookmarks.com/ https://xebialabs.com/periodic-table-of-devops-tools/