I Don’t Believe Your Company Is Agile!
Alex Martins CTO Continuous Testing CA Technologies
October 2017
1 Why Many Companies Think They’re Agile…
They moved some Dev projects from waterfall to agile
They’re having daily standups
They have a scrum master
Product owner is part of the team
They are all talking and walking agile…
And are talking about Continuous Delivery
BUT… QA is STILL a Bottleneck… Even in DevOps Shops
A 2017 survey of self- …of delays were occurring at proclaimed DevOps 63% the Test/QA stage of the practitioners found that … cycle.
“Where are the main hold-ups in the software production process?”
63%
32% 30% 16% 22% 21% 23%
http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf
3 Challenges to Achieving Continuous Delivery & Testing
IDEA Requirements 64% of total defect cost originate in the requirements analysis and design phase1. ? ?
of developers time is spent 80% of teams experience delays in development and QA Development 50% 3 finding and fixing defects2 due to unavailable dependencies X X
Security 70x required manual pen 30% of teams only security scan 50% more time spent on security test scan cost vs. automation10 once per year9 defects in lower-performing teams8 X X X 70% of all 63% of testers admit they 50% of time 79% of teams face prohibitive QA / Testing testing is still can’t test across all the different spent looking for restrictions, time limits or access manual4 devices and OS versions5 test data6 fees on needed 3rd party services3 !
Release 57% are dissatisfied with the time it takes to deploy new features7
! ! Ave. cost of downtime per minute = $4,700 Operations Greater than 50% production outages from configuration management 53% percent of visits are abandoned if a mobile site takes more than three seconds to load What is Continuous Testing? First need to understand what it’s not!
It’s not… Test Automation It’s not… Continuous Integration It’s not… Unit Testing It’s not… done only in the Test environment It’s not… a QA-led activity
5 What is Continuous Testing?
The practice of testing across every activity in the SDLC to uncover and fix unexpected behaviors as soon as they are injected Most Customers Cannot Do Continuous Testing Today, But They Aspire To Get There….
7 If This is Your Current SDLC You are waterfall
X WEEKS X WEEKS X WEEKS
Dev Test
Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations
8 If This is Your Current State You are water-SCRUM-fall
X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS
Sprint 1 Sprint 2 Sprint 3
Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations
9 If This is Your Current State You are not “really” agile
X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS
User Sprint 1 Sprint 2 Sprint 3 Sprint N Stories
Dev and Manual Sprint 1 Sprint 2 Sprint 3 Sprint N Testing
Automated Regression Sprint 1 Sprint 2 Sprint 3 Sprint N Testing
Release
10 What Testing at the Speed of Agile Should Look Like!
In-Sprint everything to achieve “Continuous Testing”
X WEEKS X WEEKS X WEEKS X WEEKS
User Sprint 1 Sprint 2 Sprint 3 Sprint N Stories HIGHLIGHTS
No Sprint-lag effect Dev and Exploratory Sprint 1 Sprint 2 Sprint 3 Sprint N Release in end of Sprint Manual Testing In-Sprint Test Automation
Test Automated Scripting & Sprint 1 Sprint 2 Sprint 3 Sprint N Exectution
Release Release Release Release
11 TESTING REALITY TESTING GOAL
TOOLSET TEST TOOLSET TEST PYRAMID PYRAMID UI Tools UI 5%
API Tools UI Tools 94% API 45% UI
Dev Tools 50% API Tools API 3% Unit Dev Tools Unit 3%
Non-functional: static, performance, security, etc…
*Actual Customer Example of Their Testing Strategy Why are Traditional Testing Activities a Bottleneck? Testing hasn’t changed in the past 30 years
Define an Idea or Write Manual Customer Needs Requirements Processes
Enable Find/Create Automate Tests Identify Interfaces Identify Test Data Create Detailed Create Read Interfaces Test Data (usually Regression) (internal and 3P) Requirements Test Cases Requirements Requirements (real or virtual) Traceability Matrix
Execute Create Defects Fix Defects Test Cases Achieving Speed Through Continuous Testing…
Define an Idea or MODEL Customer Needs Requirements
NOW ALL AUTOMATED Enable Find/Create Automate Tests Identify Interfaces Identify Test Data Create Detailed Create Read Interfaces Test Data (usually Regression) (internal and 3P) Requirements Test Cases Requirements Requirements (real or virtual) Traceability Matrix
Execute Create Defects Fix Defects Test Cases A Different Perspective On An Old Challenge Too Many Unnecessary Manual Steps…
Why not skip the intermediate steps?
Create a diagram Convert it to text Interpret text Create app code, tests, data, scripts, etc.
Intermediate activities are impediments to speed and quality
15 4 Pillars of Continuous Testing
Code Correct? Automated Flow? Did developers create Can the application the application code Code Pipeline code flow across correctly? environments without Quality Automation manual intervention?
Customer Application Are users perceiving value in the Experience Quality Did developers create application delivered? the correct features?
User Value? Features Correct? THANK YOU
17