Drive Continuous Delivery with Continuous Testing
Total Page:16
File Type:pdf, Size:1020Kb
I Don’t Believe Your Company Is Agile! Alex Martins CTO Continuous Testing CA Technologies October 2017 1 Why Many Companies Think They’re Agile… They moved some Dev projects from waterfall to agile They’re having daily standups They have a scrum master Product owner is part of the team They are all talking and walking agile… And are talking about Continuous Delivery BUT… QA is STILL a Bottleneck… Even in DevOps Shops A 2017 survey of self- …of delays were occurring at proclaimed DevOps 63% the Test/QA stage of the practitioners found that … cycle. “Where are the main hold-ups in the software production process?” 63% 32% 30% 16% 22% 21% 23% http://www.computing.co.uk/digital_assets/634fe325-aa28-41d5-8676-855b06567fe2/CTG-DevOps-Review-2017.pdf 3 Challenges to Achieving Continuous Delivery & Testing IDEA Requirements 64% of total defect cost originate in the requirements analysis and design phase1. ? ? of developers time is spent 80% of teams experience delays in development and QA Development 50% 3 finding and fixing defects2 due to unavailable dependencies X X Security 70x required manual pen 30% of teams only security scan 50% more time spent on security test scan cost vs. automation10 once per year9 defects in lower-performing teams8 X X X 70% of all 63% of testers admit they 50% of time 79% of teams face prohibitive QA / Testing testing is still can’t test across all the different spent looking for restrictions, time limits or access manual4 devices and OS versions5 test data6 fees on needed 3rd party services3 ! Release 57% are dissatisfied with the time it takes to deploy new features7 ! ! Ave. cost of downtime per minute = $4,700 Operations Greater than 50% production outages from configuration management 53% percent of visits are abandoned if a mobile site takes more than three seconds to load What is Continuous Testing? First need to understand what it’s not! It’s not… Test Automation It’s not… Continuous Integration It’s not… Unit Testing It’s not… done only in the Test environment It’s not… a QA-led activity 5 What is Continuous Testing? The practice of testing across every activity in the SDLC to uncover and fix unexpected behaviors as soon as they are injected Most Customers Cannot Do Continuous Testing Today, But They Aspire To Get There…. 7 If This is Your Current SDLC You are waterfall X WEEKS X WEEKS X WEEKS Dev Test Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations 8 If This is Your Current State You are water-SCRUM-fall X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS Sprint 1 Sprint 2 Sprint 3 Agile Team: Design / Development / Unit Testing Integration & Central QA Release & Operations 9 If This is Your Current State You are not “really” agile X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS X WEEKS User Sprint 1 Sprint 2 Sprint 3 Sprint N Stories Dev and Manual Sprint 1 Sprint 2 Sprint 3 Sprint N Testing Automated Regression Sprint 1 Sprint 2 Sprint 3 Sprint N Testing Release 10 What Testing at the Speed of Agile Should Look Like! In-Sprint everything to achieve “Continuous Testing” X WEEKS X WEEKS X WEEKS X WEEKS User Sprint 1 Sprint 2 Sprint 3 Sprint N Stories HIGHLIGHTS No Sprint-lag effect Dev and Exploratory Sprint 1 Sprint 2 Sprint 3 Sprint N Release in end of Sprint Manual Testing In-Sprint Test Automation Test Automated Scripting & Sprint 1 Sprint 2 Sprint 3 Sprint N Exectution Release Release Release Release 11 TESTING REALITY TESTING GOAL TOOLSET TEST TOOLSET TEST PYRAMID PYRAMID UI Tools UI 5% API Tools UI Tools 94% API 45% UI Dev Tools 50% API Tools API 3% Unit Dev Tools Unit 3% Non-functional: static, performance, security, etc… *Actual Customer Example of Their Testing Strategy Why are Traditional Testing Activities a Bottleneck? Testing hasn’t changed in the past 30 years Define an Idea or Write Manual Customer Needs Requirements Processes Enable Find/Create Automate Tests Identify Interfaces Identify Test Data Create Detailed Create Read Interfaces Test Data (usually Regression) (internal and 3P) Requirements Test Cases Requirements Requirements (real or virtual) Traceability Matrix Execute Create Defects Fix Defects Test Cases Achieving Speed Through Continuous Testing… Define an Idea or MODEL Customer Needs Requirements NOW ALL AUTOMATED Enable Find/Create Automate Tests Identify Interfaces Identify Test Data Create Detailed Create Read Interfaces Test Data (usually Regression) (internal and 3P) Requirements Test Cases Requirements Requirements (real or virtual) Traceability Matrix Execute Create Defects Fix Defects Test Cases A Different Perspective On An Old Challenge Too Many Unnecessary Manual Steps… Why not skip the intermediate steps? Create a diagram Convert it to text Interpret text Create app code, tests, data, scripts, etc. Intermediate activities are impediments to speed and quality 15 4 Pillars of Continuous Testing Code Correct? Automated Flow? Did developers create Can the application the application code Code Pipeline code flow across correctly? environments without Quality Automation manual intervention? Customer Application Are users perceiving value in the Experience Quality Did developers create application delivered? the correct features? User Value? Features Correct? THANK YOU 17.