Agile Test Automation Strategy, V2.Pptx

MJ Half-day Tutorials 10/3/16 13:00

Test Automation Strategies for the Agile World

Presented by:

Bob Galen

Velocity Partners

Brought to you by:

350 Corporate Way, Suite 400, Orange Park, FL 32073 888---268---8770 ·· 904---278---0524 - [email protected] - http://www.starwest.techwell.com/

Bob Galen Velocity Partners

An agile methodologist, practitioner, and coach, Bob Galen ([email protected]) helps guide companies in their adoption of Scrum and other agile methodologies and practices. Bob is a principal agile evangelist at Velocity Partners; president of RGCG; and frequent speaker on software development, project management, software testing, and team leadership. He is a Certified Scrum Coach, Certified Scrum Product Owner, and an active member of the Agile and Scrum Alliances. Bob published Scrum Product Ownership–Balancing Value from the Inside Out.

Test Automation Strategies for the Agile World

Bob Galen President & Principal Consultant RGCG, LLC [email protected]

Introduction Bob Galen n Independent Agile Coach (CEC) at RGCG, LLC n Director, Agile Practices at n Somewhere ‘north’ of 30 years overall experience J n Wide variety of technical stacks and business domains n Developer first, then Project Management / Leadership, then Testing n Senior/Executive software development leadership for 20+ years n Practicing formal agility since 2000 n XP, Lean, Scrum, and Kanban experience n From Cary, North Carolina

Bias Disclaimer: Agile is THE BEST Methodology for Software Development… However, NOT a Silver Bullet!

1 Outline

n Traditional Automation – Business Case & ROI n 3-Pillars n Agile Test Automation Pyramid n Agile Automation – Business Case & ROI n Implementation Strategy n Communication n Wrap-up

Let’s start with… Traditional Automation Strategy n What are your current strategies towards:

q Test Automation

q Frameworks

q Tooling

q Maintenance

q ROI

q Team structure n Get together in “pairs” and chat about this for 20 minutes. List your approaches and tactics. n Then leverage SWOT or FFA analysis to capture your current position. Then we’ll gather your results…

2 Let’s start with… Traditional Automation Strategy n SWOT – Often used to analyze strategic direction or positions. List the “position”, then identify aspects of:

q Strengths

q Weaknesses

q Opportunities

q Threats Often in Quadrants

n Force Field Analysis – again, list a position

q Forces (FOR) è

q Forces (AGAINST) ç

Traditional Automation Business Case & ROI

3 It’s simple really…

n Manual testing is:

q Slow

q Labor intensive

q Not intellectually stimulating

q Error prone

q Iterative

q Did I say slow? n And we’re testing software…right?

q So we ought to be able to automate the testing.

It’s simple really…

n Automated tests are:

q Fast

q Low/no errors

q Free after initial development

q Run continuously

q Did I say fast?

q Did I say ‘free’?

n And we’re testing software…right?

q So we ought to be able to automate ALL of the testing.

4 Test Automation ROI Sample Points – Manual Testing

ü Capture size of the team ü Cost per tester – usually hourly ü # of Test cases to be run ü Time to run them ü Cyclical nature of the testing cycles (regression, integration, system, etc.) ü Configurations ü Number of tests run per release ü Number of releases per year n Do the math to come up with iterative testing run-time investments…and costs

Test Automation ROI Sample Points – Automation Development n Then contrast that against the cost for automation: ü Time / cost to establish automation infrastructure ü Time / cost to establish automated tests ü Time / cost for ongoing maintenance and results analysis ü Cost for automation tooling – depends on ‘type’, ex: Keyword Driven ü Cost per automation developer vs. manual tester ü Automation replacement target for manual test cases

n Do the simple, discrete math to come up with cost & time to replace manual testing run-time and ROI savings

5 Traditional Business Case

n Run more tests, faster and with fewer people (lower costs)

n VALUE being driven by: Running tests!

n Did I say with fewer people (testers)?

q We can hire more developers with the cost savings! And get MORE done…

q We can even reduce the size of the bathrooms J

http://www.aspiresys.com/testautomationroi/index.php#ROI

6 http://www.aspiresys.com/testautomationroi/index.php#ROI

Sample of Test Automation ROI calculators

n Aspire Systems seems to have pulled down their link, here are other samples…

q http://sourceforge.net/projects/ta-roi/

q http://www.automatedtestinginstitute.com/home/index.php? option=com_content&view=article&id=58:roi- calculator&catid=37:calculators&Itemid=65

q http://www-01.ibm.com/software/rational/offerings/testing/roi/tool/ ROI_Rational.html

7 But often there is underestimation of risks… So the ROI was always a ‘fuzzy’ estimate

§ Underestimating the nature of a software project § Architecture & design, complexity, risks, unknowns, ambiguity, etc. § Underestimating parallel software projects (product + automation) complexity § Integration, dependencies, coordination § Underestimating ongoing support & maintenance § Repairs & updates § Designing and adding new tests § Impact of new product technologies § Underestimating skill levels for automation development

It’s also very dangerous! Will Robinson…

§ It’s commoditizing your testing activity. Could you imagine – § Measuring developer value by LOC…and doing ROI? § Measuring architectural integrity by complexity analysis…and doing ROI? § Measuring product owner value by number of backlog elements…and doing ROI? Based on some sort of automation introduction? Of course not. It’s insane.

§ Then why do we do it with tests…and testers?

8 Traditional Test Automation Tools Just a quick stop…

n We’re all using them, so they must be good and the approaches must be ‘correct’…right?

n However, there are challenges—

q Training

q Size

q UI-centric

q COST

q Can’t test everything (Back-end database code)

q One-size fits all strategy

q Limited developer / whole team usage

Typical Deployment Level of effort MORE

Typical Time & Budget Initial design Investment & Architecture LESS

é Initial Test Level of Effort - Case Automation Investment ê Technology Disruption

Ongoing Maintenance

9 Now it may sound…

n Like I’m opposed to test automation. Truly, I’m not. I think automation rocks! Or like I’m opposed to gaining a ‘return’, absolutely not.

q Automation is clearly a foundation element for increased quality and going faster – releasing more often

n But, this traditional model or approach of focusing on ROI…is sort of insane

3-Pillars of Agile Quality & Testing

10 3-Pillars Genesis

n I’ve learned that “Balance” is important n A sad tale of:

q Thousands of ATDD testing; Gherkin run amok

q All of them are working; continuously testing; increasing “coverage’ and life is Good! n BUT

q These same teams couldn’t write a cohesive User Story to save their life

q So, where were the Acceptance Tests coming from?

3-Pillars of Agile Quality

Development & Test Software Testing Cross-Functional Team Automation Practices • Risk-based testing: • Pyramid-based Strategy: Functional & Non-Functional • Team-based Pairing (Unit + Cucumber + Selenium) • Test planning @ Release & • Stop-the-Line Mindset Sprint levels • Continuous Integration • Code Reviews & Standards • Exploratory Testing • Attack technical • Active Done-Ness infrastructure in the Backlog • Standards – checklists, templates, repositories • Aggressive Refactoring of • Visual Feedback – Technical Debt Dashboards • Balance across manual, exploratory & automation • User Stories, “3 Amigo” • Actively practice ATDD and based Conversations BDD

• Whole Team Ownership of “Quality” • Knowing the Right Thing to Build; And Building it Right • Healthy – Agile Centric Metrics • Steering via: Center of Excellence or Community of Practice • Strategic balance across 3 Pillars; Assessment, Recalibration, and Continuous Improvement

11 Foundation of the 3-Pillars

• Whole team view includes building it right, • Whole Team Ownership of everyone tests, everyone demo’s, etc. “Quality” • Focus on features/stories, confirmation, • Knowing the “Right” thing to conversation, and getting them staged Build AND Building it “Right” properly OVER testing

• Healthy – Agile Centric Metrics • 4-tier metrics: Quality, Value, Prediction, Team

• Steering Required – CoE or CoP • Agile strategies need light-handed “steering”; establish a CoE (heavier weight) or a CoP • Strategic balance across 3 (lightweight) Pillars; Assessment, Recalibration, and Continuous • Consider finding an assessment framework Improvement and then tying it to your strategy measurement, recalibration, and continuous improvement.

• Make the foundation visible thru information radiators and metrics

3-Pillars of Agile Quality

A central part of agile adoption is focusing on CI, 3- Development & tiered Automation development, and Dashboards to Test Automation begin incrementally building coverage for faster feedback on changes. • Pyramid-based Strategy: (Unit + 100% automation is NOT the Goal! Cucumber + Selenium) In the interim, Hardening or Stabilization Sprints and • Continuous Integration having a risk-based Release Train concept help

• Attack technical It’s important that Test or QA not ‘own’ the tooling or infrastructure in the all of the automation efforts. The strategy can come Backlog from QA, but the tactical automation development is best left to the team. • Visual Feedback – Dashboards Mature teams invest in Automation, Tooling, and Technical Debt reduction as part of Done-ness and • Actively practice ATDD continually add it to their backlogs and BDD

12 3-Pillars of Agile Quality

Exploratory Testing (SBET with pairing) can be an incredibly effective way to establish a whole-team, Software Testing collaborative view towards quality and testing. It also emerges new tests. • Risk-based testing: Functional & Non- Leverage ‘plans’ as a whole-team collaboration- Functional conversation mechanism; at Sprint and Release levels. • Test planning @ Release & Sprint levels Do not measure testing or tester progress; instead, measure throughput, output, sprint outcomes, and • Exploratory Testing done-ness escapes at a team level.

• Standards – checklists, You need a balanced test team; not everyone needs templates, repositories to be able to program. But everyone needs to be passionately skilled testers with curiosity. • Balance across manual, exploratory & Agile testing is a Risk-Based play in every Sprint and automation across a release sequence.

3-Pillars of Agile Quality

Cross-Functional One of the hardest areas to get ‘right’ culturally. It Team Practices needs leadership alignment from Quality/Testing to Product to Development and a consistent voice of

whole-team approaches. • Team-based Pairing

This is where LEAN Thinking lives, where whole- • Stop-the-Line Mindset team collaboration happens, where professionalism and craftsmanship are held dear. • Code Reviews &

Standards I like the view of testers becoming the VOC, champions of quality, and consistent questioners of • Active Done-Ness what is being build. Are we solving the right problems…as simply as possible. Notions of Minimal • Aggressive Refactoring Viable Product / Feature help with focus. of Technical Debt

And yes Virginia, there ARE standards, templates, • User Stories – 3 Amigo based Conversations and a focus on x-team consistency!

13 Agile Test Automation Pyramid

Agile Test Automation Pyramid n Coined by Mike Cohn, ~ 2005 n Establishes the validity of turning Traditional Automation – upside down

n Invests:

q Most effort (%) in Unit Tests

q Moderate effort (%) in Middle-tier tests (business logic, API, component, feature)

q Least effort (%) in UI-centric, top down tests n Granularity of the tests is part of the strategy

14 Agile Test Automation Pyramid Often: n Tied to Definition-of-Done from a maintenance perspective

q You break it, you fix it n Percentages vary; intent does not n Whole-team ownership of automation

q Although it skews at either end of the pyramid n Run as much as possible

q Check-in, Overnight, Cyclically

q Prime Directive = Real-time Feedback

Agile Test Automation Pyramid

+80%, UI Centric Automation

0-10%, Service or Middle Tier Automation

0-10%, Unit level Automation Traditional Agile 0-10%, UI Centric Automation

20-40%, Service or Middle Tier Automation

50-60%, Unit level Automation

15 Agile Test Automation Pyramid Mike Cohn; Lisa Crispin & Janet Gregory http://behaviordrivendevelopment.wikispaces.com/Testing

Definition of Done iContact example

n As part of story design, consider 3-levels of automation required for solid implementation:

q Unit

q Cucumber

q Selenium n If the team warrants the automation has value (ROI) then automate it n Implement those automated test cases WITH the story n Demonstrate them in the Sprint Demo

16 Definition of Done iContact example

n Previous sprint automation needs to continue to work; if you break it…then Fix It! n Previous CI/CD – automation wiring needs to continue to work; if you break it…then Fix It! n All appropriate (RBT) Acceptance Tests should be regressed so that we haven’t “lost value”

n And in Readiness Criteria

q Automation research spikes and architectural look-ahead

Brainstorm… Agile, Multi-tiered Automation n Get together in small groups of 4-6 to discuss

n Take a few minutes and think about your current automation approaches:

q Tooling, approaches & strategies, strengths, weaknesses, opportunities, maintenance challenges, future technology, etc. n What sorts of adjustments would you need to make to take this approach? n What would be the largest challenges in taking this approach? How might you overcome them? n Do you “buy” the whole-team view to automation?

17 Agile Test Automation ROI

What agile has exposed to me… wrt/Automation & Testing n Testers vs. a Whole Team view

q Build teams in ratios – developers vs. testers

q Instead invest in cross-functional teams composed of the skills to design, construct, test, and deliver excellent products

q Only testers write test automation

q The entire team writes test automation and, oh by the way, can test as well

q Only testers test the application

q The whole team participates in writing / running automation, functional testing, exploratory testing, regression testing, etc.

18 What agile has exposed to me… wrt/Automation & Testing n Traditional vs. Open Source Tools

q Instead of leveraging singular, UI-centric tools

q Leverage multiple tools at all-tiers of your application stack

q Higher cost

q Lower cost and a sense of “Community”

q Developers won’t use them; costs

q Everyone is leveraging the same tool-sets; AND integrating them in Continuous Integration and Continuous Deployment

What agile has exposed to me… wrt/Automation & Testing n People are your primary value proposition

q Instead of looking at testers as a commodity

q Consider good testers to be as valuable as any other team member

q Instead of thinking that testing is a by-rote exercise with little intellectual pursuit

q Testing is a profession. Testers are valuable craftspeople. Testing is an intellectual exercise.

q Instead of thinking the value is in the test cases

q The value is in the testers (people) continuously reevaluating the right things to test at the right time…for valuable feedback

19 http://outrage.typepad.com/crisisanalysis/2010/11/the-tyranny-of-roi.html

The NEW Test Automation Business Case n So, don’t

q Count pennies

q Count heads

q Count cycles n Work as part of a TEAM

q Focus in on testing continuously – just the right things at just the right time

q Providing feedback, transparency, and reaction

q Allowing your teams time to learn and improve

q Solving your customers’ problems; delivering value

20 The NEW Test Automation Business Case n Run more tests, faster, unattended, continuous feedback, learning & adjustments

q Yes, saving time. Now what to do with it???

n Drive your new VALUE by:

q Early feedback for adjustments—risk avoidance, learning, solving real problems

q Having more time to continuously refine your test cases (automated and not)

q Having more time to focus on building Quality into your products

q Creatively testing the application (Exploratory Testing, Test Idea Generation, Test Design)

So, with the additional time… INVEST in your testers & quality ü In building In Quality ü In attacking the customers REAL problems ü In better testing ü In more balanced automation ü In team training; in slack time ü In better collaboration ü In doing more than thought possible ü In creative solutions ü In adapting to new technologies ü In having FUN

21 How do we “Sell” this change? n You don’t sell it from a testing perspective. Nor from a bean-counting perspective.

n You allow the results to begin speaking for themselves as we begin focusing on and talking about—

q How one builds truly innovative and creative software teams

q That understand and solve customer problems

q With simple solutions that fundamentally work

q And do so iteratively & quickly, while nimbly adjusting to customer feedback

But Bob…

n That doesn't work in the “Real World”!

n My boss would never go for it without ROI justification and hard measures

n Testing IS a commodity

n We’ve done automation perfectly – seeing none of the issues/risks you’ve pointed out

22 But Bob…

n Think about what I’m saying; taking the time and…

q Invest in people

q Invest in better quality & testing

q Invest in better product solutions to customer problems

q Invest in improved workflow and delivering high customer value WOW – how crazy is that?

n Meet in small groups of 2-3 (pro / con) and chat for 10 minutes about the “sides” and the possibility for change?

n Then we’ll debrief…

Agile Test Automation Implementation Strategy

Automation is one of the foundational investments of effective Agile Adoptions. It’s not a choice, it’s an imperative! It’s a cost of doing business.

23 Implementation Strategy

Typically a 5-Phased Strategy

1. Strategy – This session: 3 Pillars + Pyramid + Team

2. Team Formation 3. Tools Selection 4. Training & Infrastructure Development 5. Automation Development 6. Care & Feeding

Phase 2 Team Formation n GOAL…whole team ownership n Skill-set discussion; SDET vs. Test Professional? n May influence

q Development focus on Unit Testing

q Automation team to “gain momentum”

q Agile execution (Scrum, Kanban, transparency, demo, etc) n Connect the dots architecturally n Bias: I like to lead automation with a Test Automation Architect

q Rather than with someone from the development team

24 Phase 3 Tools Selection n Open Source vs. Commercial n Singular, all-purpose vs. Tool-kit

q Cobble together or Integrate disparate tools n Always perform a “Bake off” within your teams n If Open Source, consider support implications n Connect to development tooling

q Including CI and CD n Additional Considerations: Test data, virtualization, DevOps n Be prepared to iterate, learn, and possibly fail

Phase 4 Training & Infrastructure n Design, coding, and documentation standards n Templates n Community of Practice

q Example code, standards, forum, collaborative groups n Models:

q Keyword, Data-driven, Table-driven, DSL, Model-driven, others

q Test Suite; Hierarchy, Naming n Commercial: invest in training and/or consulting n Open Source: hire and/or acquire direct expertise

q Also can build the expertise

25 Phase 5 Automation Development n Move it to your teams ASAP n Make it a part of Definition-of-Done

q It should become a natural outcome and not a choice or question n Complete it within each sprint

q SAFe as a goal model for completing work in the same sprint!

q Salesforce as well n It needs to be a negotiated part of your Product Backlog n Have % targets; n For every story:

q The TEAM decides the requisite automated tests at each level of the pyramid?

Phase 5 Automation Development Attacking the Pyramid

n Usually I attack the Unit-level first

q Development team engagement; sheer #’s

q Baseline quality improvement

q Integrated with CI/CD; fast feedback n Then select either middle or UI tier; context based

q Tools selection

q Training

q Framework development

26 Phase 5 Automation Development Attacking the Pyramid

n Complete one layer nearly completely before moving to the next layer

n Unit Test Strategy – Don’t dig the “hole” any deeper

q Legacy vs. New

q Encapsulate vs. Proper unit tests

q Refactoring implications

Phase 6 Care & Feeding n Definition of Done & Product Backlogs n Automation evolution roadmap

q Merge with your architectural look-ahead

q Merge with your Product Roadmap

q Factor in ongoing automation investments n Refactoring & Repairs as part of your Technical Debt handling n Consider it at the same level of investment as your application code n Stop-the-Line n Commitment

27 Communication

n Leadership Vision

q High-level Plans, Strategic sharing with your teams

q Leadership collaboration (Development + QA)

q Intent to doggedly pursue automation n Information Radiators n Roadmaps & Product Backlogs n Sprint & Release Reviews

q Even though it’s hard to show “working software”

q Let the “process” transparently communicate your progress, impediments, plans, challenges

Let’s end with… Agile Automation Strategy n Based on what you’ve heard, discussed, learned…

n Get together in “pairs” or small groups and chat about this for 20 minutes. List your strategic approaches. Then leverage FFA to capture your driving forces FOR and AGAINST moving towards Agile Automation n Most importantly, focus on strategies to overcome your obstacles.

n Then we’ll gather your results…

Wrap-up THANK YOU!

§ I hope I’ve added to your previous assumptions and understanding § Inspired you to change your view towards Automation ROI and investment § Introduced you to models for attacking Agile Automation Strategy and ROI

§ Final questions or discussion?

29 Contact Info Bob Galen Principal Consultant, RGalen Consulting Group, L.L.C.

Experience-driven agile focused training, coaching & consulting

Cell: (919) 272-0719 [email protected] www.rgalen.com @bobgalen https://www.linkedin.com/in/bobgalen

Podcast on all things ‘agile’ - http://www.meta-cast.com/

References

30 Links

n http://www.agileengineeringdesign.com/2012/01/7- deadly-sins-of-automated-software-testing/

n Link to PDF of Pillar-1 chapter n Links to “old” STP 3-part automation series n Link to Mary Thorn’s – Agile Test Manager article n + other materials, join the mailing list for download password at - http://goo.gl/ORcxbE

Open Source Tools

n Unit Level

q x-Unit

q Junit, nunit, phpunit, n ATDD

q FitNesse, Robot Framework n BDD

q Cucumber, JBehave n UI

q Selenium, Watir, HP-UTF

31 …DD much from the XP space n TDD - Test Driven Development

q Test-first; Unit Tests as a design aid

q Debate around creating the unit tests “first” n ATDD - Acceptance Test Driven Development

q User Story – Acceptance Tests

q Executable Requirements n BDD - Behavior Driven Development

q jBehave, Cucumber; Gherkin – Given, When, Then

q More useful phrasing n Tests as functional documentation n Product Owners, Customer – writing tests

Agile Testing Quadrants Brian Marick; Lisa Crispin & Janet Gregory

Business Facing Automated & Manual Manual

Functional tests Exploratory testing Story tests Scenarios Examples Usability testing Critique the Product the Product Critique Prototypes UAT Simulations Alpha / Beta Q2 Q3

Q1 Q4

Unit tests Performance testing Supporting the Team SupportingtheTeam Component tests Load testing API tests Security testing Non-functional requirements

Automation, Automated & Tools, and Manual Technology Facing Manual