White Paper

DevOps and Testers

DevOps is part of an overall approach that organizations use to deliver software frequently and with high quality. The most obvious outcome of successful DevOps implementations is the reduction in the time it takes for software changes to transition from an idea to production.

What Does DevOps Mean If you are an experienced DevOps Automated tools and processes practitioner, we hope that you are used in system configuration, for Testers might still find the article useful. If the build process, testing, the Background you are not a tester, we hope that deployment to test, staging and The DevOps movement (for want you will at least see the tester’s production environments, post- of a better label) is progressing perspective. deployment monitoring, evaluation, rapidly. Like many other and operations. movements in the industry, the What Is DevOps? speed of adoption accelerates Is DevOps Just About faster than the definition of the Simplistically, DevOps is a label movement itself. DevOps is still to describe an ecosystem in Tools? not well defined and the nuances which development teams and At one level, the goal of DevOps of culture, the emergent capability systems operations teams work is to eliminate bottlenecks in of new technologies, and range of more closely together. In the the delivery pipeline through (mostly successful) case studies so-called delivery pipeline, from automation. But the automation means that the issues at hand are committing source code to putting of staged processes still still widely debated.1 code into production, developers requires governance. Most accommodate and automate some automated processes are not Depending on who you talk to, of operations activities. Operations really autonomous; they cannot DevOps can be a solution to a has more visibility into, and some complete their tasks without problem or a goal in itself. In some influence over, the activities of human intervention in maintenance businesses, the goal is going developers. The motivation for or in handling exceptions. A fully digital, and DevOps is part of the this is primarily to speed up the automated DevOps process is overall approach to delivering deployment and implementation meaningless without consideration frequently and with high quality. of software. Bringing Dev and of the human factor. Although tools But in the marketing of DevOps- Ops closer together in an agile do a lot of heavy-lifting, it is the related technologies and services, team effectively implements agile people running the process that this goal can be obscured. The operations. make it work or fail. challenge of making the cultural change (or more concretely, The most obvious outcome of Is DevOps Just Dev and Ops making the behavioral change) successful DevOps implementations Working More Closely with the Aid required for success is frequently is the reduction in the time it takes of Tools? underestimated. for software changes to transition No, DevOps is not that either. from an idea to production. When a Often, the testers involved in and developer says a software change is The hand-offs between automated affected by DevOps are new to done, the transition to production is processes often involve other the whole idea. This article is an performed with the aid of pervasive processes, usually testing of one introduction to DevOps for these automation. kind or another. testers and is a discussion of its impact on test practices. 1 “What is DevOps,” The Agile Admin, http://theagileadmin. com/what-is-devops/ DevOps and Testers White Paper

If there is one mantra that best If there are manual interventions, It might be painful describes the drive towards typically between the automated DevOps it is, “If it hurts, do it stages in the DevOps process, we for you, as a tester, to more.” It might be a bit of a cliché, will see these as the pain points. trust developers and but use that as the context for They are bottlenecks, the causes implementing and improving of delays, and the potentially less automation to do the DevOps test practices. reliable and error-prone aspects of testing job properly. If the process. it hurts, you must do it If It Hurts, Do It More Manual testing is painful. Yes, you more often. The difficulty or pain we experience might love exploratory testing; when doing a particular job and you might fear that only you, influences us adversely. If we do not as a human, can find those gnarly like to do a task, we tend to put it bugs that automation will never Is DevOps Just About off. When we finally take the task find, that you as the tester are the on, it is more painful. This is true only person trustworthy enough to Tools? (cont.) for visiting the dentist, cleaning the prevent disaster happening. Automated tests are created garage, integrating software, doing by developers and testers. The testing, and so on. It might be painful for you, as a output of these tests is focused on tester, to trust developers and providing sufficient information to Our experience is commonly that automation to do the testing job other processes, or just as often the less frequently we perform a properly. If it hurts, you must do it to people, to make transitions task, the more traumatic the task more often. between stages in the pipeline. is when we actually do it. Martin Fowler suggests three reasons why Tests, Automation, and Testers and developers who the frequent or even continuous test provide the assurance that execution of certain tasks reduces Trust the DevOps process delivers the pain.3 There is much debate around the successfully and reliably. meaning of checking and testing4, • First, larger, more complex tasks and the reliance we can place are hard to plan, manage, and What is DevOps, Really? on testers, on checks, and on control. Breaking up large tasks automation5, 6. The question is posed and makes them easier to do, less discussed at length in an excellent risky to do and, if something Can we place all our faith in post on the Agile Admin, “What is does go wrong, easier to unwind. automated checks? 2 DevOps” . DevOps is an evolving, • Second, many tasks (and testing We certainly need more emergent discipline. So the is the shining example) provide sophistication than that. But definition of DevOps is still not feedback. Feedback, if received we can, for the purpose of this settled. Perhaps it never will be. early and often, helps us to article, at least separate tests and rapidly address problems before What does that mean for testers? It test execution activity into four any further time is wasted. means that there is still no one true components. way and that your role in a DevOps • Thirdly, if we practice an activity regime that is evolving (and every more frequently, we get better • Checks that can be automated regime is evolving) is not yet fixed. at it. We learn how to do it by developers as part of their There are two main contributions efficiently. We might also see component-level check-in and you can make: opportunities to automate it in continuous integration processes. some way. • Checks that can be automated • You need to pay attention to From the tester’s perspective, the (typically by system testers) to the things that hurt and work to mantra, “It it hurts, do it more,” exercise API-level, link, or end-to- make them less painful. forces us to take the notion of end transactions. • You need to identify the automation in the testing process opportunities and interventions much more seriously. that will add value to the DevOps 4 “Testing and Checking Refined,” James Bach, Michael process. Bolton, http://www.satisfice.com/blog/archives/856 5 “A New Model for Testing,” Paul Gerrard, http://dev.sp.qa/ download/newModel 2 “What is DevOps,” The Agile Admin, http://theagileadmin. 3 “Frequency Reduces Difficulty,” Martin Fowler, http:// 6 “The New Model and Testing v Checking,” Paul Gerrard, com/what-is-devops/ martinfowler.com/bliki/FrequencyReducesDifficulty.html http://blog.gerrardconsulting.com/?q=node/659 DevOps and Testers White Paper

Tests, Automation, and • Which checks need only be run once, manually, as opposed to Certainly, you have to Trust (cont.) checks that need to be retained • Tests that can perform for regression purposes, and are stop believing you are compatibility checks to candidates for automation? the gatekeeper of quality, demonstrate compatibility across • We mentioned the notion of browsers, operating systems, and the last defense, the only trust previously. Another way of platforms. looking at this is to speculate on person who cares. You • Tests that can only be performed how a system could be reliably have to think more like a by a human. tested if there were no late In this article, we offer only a manual testing at all. visionary, risk-identifier, few suggestions on how to make • Imagine an environment where risk manager, pathfinder, these distinctions because every all of the testing was done by environment is different, of course. tools. Would your concerns facilitator, and as a coach be dominated by the fact that and mentor. The more germane question to ask you simply do not trust the is, how does the tester let go of developers to do a good job of 7 late, manual checking? Letting go testing? Moving testing thinking One goal of monitoring is to raise requires proactive effort and trust. to the left should reduce the alerts on failure before users doubt. experience the impact of them. This These areas will be the main focus is rather ambitious, but this is the of your efforts: • If, as a tester, you act more as a pathfinder to identify risks ultimate goal. • Wherever possible, manual and assess them, to select When problems are encountered checks that could be performed tests and to ensure that they in production, the task then is to at a component level should are incorporated into the use the analytics derived from be pushed forward to the development and automation, monitoring to not only trace the developers. As a tester, you might your concerns could be cause and resolve it, but also to suggest these tests in a pairing minimized. refine the test process, automated or whiteboard session. You Certainly, you have to stop or manual, to reduce the likelihood might have to write these tests believing you are the gatekeeper of similar problems in the future. yourself and include them in the of quality, the last defense, the The role of testing and analytics continuous integration regime. only person who cares. You have across the entire pipeline process • End-to-end or user interface tests to think more like a visionary, risk- was introduced and discussed might require automation. These identifier, risk manager, pathfinder, here.8 tests need to be minimized, as facilitator, and as a coach and they tend to be slow to run, mentor. We could call the automated tests brittle, and frequently require in the DevOps process monitoring. maintenance. Consider whether Coupled with monitoring in they need to be run at every Practice, Monitoring, and production, monitoring throughout code check-in or if they could be Improvement the DevOps process an and into reserved for use on larger, less With all of our good intentions in production enlarges the scope of frequent releases only. reducing or eliminating reliance testing. DevOps, therefore, does not diminish the role of testers. • What manual-only tests could on late manual checking, bugs will

be run on components that are still get through. When software is 8 “Thinking Big: Introducting Test Analytics,” Paul Gerrard, not yet integrated into a release released into production, problems http://blog.gerrardconsulting.com/?q=node/630 candidate? Can the manual arise. testing be performed in pairing sessions with developers? Are One of the key disciplines of there alternatives to this testing? DevOps from the operations Could story-boarding or BDD- point of view is a deeper level of style prototyping help? Could UI monitoring. Monitoring happens checks be performed on mock- at every layer, from components ups or wire-frames? and simple transactions in the applications, through integration 7 “How to Eliminate Manual Feature Checking,” Paul Gerrard webinar, http://blog.gerrardconsulting.com/?q=node/622 and messaging and the infrastructure itself. DevOps and Testers White Paper

Conclusion You might be concerned about whether DevOps is here to stay and whether testers should take notice? The answer is simple. Yes.

Why wouldn’t you want developers and operations people talking to each other? Why wouldn’t you want to put more reliable builds and deployments into test and production?

Why wouldn’t you want the best of technology to support more accurate, efficient, and informative pipelines?

DevOps is a good thing but not always easy to achieve. Needless to say, it requires cultural change and that isn’t always easy. For the tester, DevOps gives us greater influence in the early stages of projects, forces us to think more seriously about automation in testing, information provisioning, and decision-making. Testers need to embrace DevOps because it provides opportunities to be proactive, and to gain more authority and respect in our project teams.

For more information, please visit our website at broadcom.com/continuous- testing.

For more product information: broadcom.com Copyright © 2020 Broadcom. All Rights Reserved. The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. Broadcom, the pulse logo, Connecting everything, CA Technologies, and the CA technologies logo are among the trademarks of Broadcom. CS200-181159-0520 May 4, 2020