GOING ALL in with DEVOPS by Andrew Agerbak, Kaj Burchardi, Steven Kok, Fabrice Lebegue, and Christian N

GOING ALL IN WITH DEVOPS By Andrew Agerbak, Kaj Burchardi, Steven Kok, Fabrice Lebegue, and Christian N. Schmid

or too many companies, moving to In many cases, there is. Agile does a good Fagile software development is like job of breaking down silos early in the soft- finding the perfect new strain of grass ware development process. But it can seed—after months of searching—and achieve only so much on its own. To turn then planting the new seeds in your old themselves into digitally ready competitors, backyard. Your lawn may ultimately look a companies have to rethink the entire soft- little better, but it will take longer and the ware development life cycle. They need results won’t be as great as they would be agile, but they also need DevOps. if you had first removed the hidden tree roots, put in new soil, and rethought the DevOps is an approach that integrates criti- irrigation system. cal late-stage activities—like testing and deployment planning—into the code- Many mainstream companies—in financial writing part of software development. (See services, health care, manufacturing, con- Exhibit 1.) With its emphasis on running sumer packaged goods, and other indus- multiple activities in parallel and on multi- tries—have felt compelled to give agile soft- functional teams, DevOps represents a ware development a try. And they have break from the old “waterfall” model, in waited expectantly for the benefits. In theo- which planning, writing, testing, and de- ry, agile’s assignment of a business leader ploying code were discrete steps managed to development teams ensures that the by separate departments. most important software changes get done first, and its emphasis on short coding While many software companies use some sprints ensures that the changes are imple- form of the continuous software develop- mented quickly. The fact that the quick part ment and release that are the hallmark of doesn’t always happen has been discourag- DevOps, the approach (which continues to ing. It has some agile newcomers wonder- evolve and is starting to be referred to as ing if there’s something they’re missing. DevOps 2.0 or BizDevOps by some of its Exhibit 1 | Where Agile and DevOps Fit in the Software Development Life Cycle

Plan Code Build Test Deploy Release Operate Monitor

AGILE Business units Integrated business • Teams “own” and are responsible for development teams a software-based product or service

IT development DEVOPS Continuous • Daily “builds” of error-free software Application integration development • Automated testing Continuous • Ability to update applications Application delivery automatically across environments maintenance IT operations Continuous • Self-service provisioning of hardware deployment • Automated releases of new software

IT operations Continuous monitoring • Applications self-repair and autonomic operations • Capacity adjusted automatically WHO DOES THE WORK

Standardized and automated • Virtualization enables server and storage constraints to be addressed automatically IT infrastructure infrastructure management • Modular architecture allows services to be provisioned quickly and cheaply

Source: BCG analysis.

more advanced practitioners) is a lot newer Today, virtualization makes computing and outside the technology and internet services storage capacity available with less opera- industries. Some traditional companies, tional complexity than before and at far notably in financial services, have some lower cost. But we still see companies tak- DevOps pieces in place, such as automated ing a month and 50-plus emails just to testing and mechanisms for provisioning provision hardware and gather all the nec- hardware quickly. (See “Leaner, Faster, and essary permissions. Better with DevOps,” BCG article, March 2017.) But the scope of these practices is Likewise, a control requiring multiple go- limited. The companies haven’t made the live approvals for new software may have overarching changes that would allow them been justified when there were only a few to capture DevOps’ full range of benefits. software updates a year and each one in- volved a critical part of a monolithic system. But it shouldn’t require two dozen Making DevOps Work people to approve a minor tweak—like To get the most out of DevOps, companies changing the color of the screen users see. must make changes in controls and governance, IT organization roles, and operating With software now a key way of addressing models. fast-changing business and customer needs, the prolonged delays caused by controls Rethink controls and governance. Most big that have become irrelevant put companies companies’ approach to developing and at a fundamental competitive disadvan- releasing software reflects controls they put tage. The delays can pose a reputational in place years ago to maintain quality and risk and even a survival risk if a company avoid costly mistakes. The controls may is the target of a cyberattack. (See “Devel- have made sense at the outset. But as the op a Cybersecurity Strategy as If Your Or- pace of technological change has accelerat- ganization’s Existence Depends on It,” Oc- ed, the controls have lost their relevance. tober 2017.) Now they’re just obstacles. Governance is another area that requires For instance, a control about infrastructure adjustments in the move to DevOps. This provisioning—one of the hurdles that must includes adopting new approaches to fund- be surmounted before a development team ing. In agile, funding isn’t allocated on a can begin its work—may have been imple- project basis: for a set period of time, mented in the days before virtualization. against a defined set of deliverables. In-

The Boston Consulting Group | Going All In with DevOps 2 stead, funding is allocated to critical “prod- teams, which are expected to make the ucts”—like a mutual fund company’s “my fixes. There is no one farther down the line account” function or a retailer’s order-and- who would even think of fixing the code, ship system—that require the attention of and no possibility of different departments teams for many months or even years. touching the same code simultaneously DevOps adds complexity by forcing compa- and working at cross-purposes. nies to figure out how to allocate some application maintenance and infrastructure Ultimately, the best test of governance costs within the product funding paradigm. practices is cycle time. If companies can substantially reduce the time between Another area where DevOps should trigger when they plan software and when they re- a governance change is in the decision lease it in a reliable, high-quality form, that rights related to cloud solutions. In the is a sign that their governance processes past, these decision rights belonged to the are working and that they have the techni- IT organization, and no one questioned cal capabilities they need. that. But that’s changing as more business units create software directly using cloud- Redefine the role of the CIO and the IT based tools. organization. If DevOps is to succeed, there must be changes—some subtle, some more We saw questions about such decision dramatic—in the role of the chief informa- rights at a company where digital product tion officer and the information technology development teams were pushing for direct organization. In companies that adopt agile access to an Amazon Web Services account, models, the specifications for new soft- and the IT operations group, concerned ware—and the coding work itself—become about standardization and security, was re- the implicit responsibility of business units. sisting. In truth, there is no single right an- If this relieves the CIO of responsibility for swer to the question of where the decision individual lines of code, in most cases he or rights should lie, for this company or any she still shoulders the larger burden of other. But it is an issue that must be tack- quality. That is, the CIO must still recruit led in DevOps, which redraws the boundar- and train software developers. He or she ies of software development along multiple must also put in place a better delivery dimensions. model, one that includes an operating environment—standards, services, process- DevOps should also prompt a change in es, tools, and infrastructure—that allows how companies deal with buggy software. developers to maximize their productivity. At companies that haven’t fully adopted The CIO must also front-load more activi- DevOps, there isn’t a “you built it, you own ties in the software development life cycle. it” governance philosophy. Instead, if an issue arises involving software that has The term of art for this sort of front-loading been released, IT support teams report it is the “shift left,” referring to how one through a ticket system (such as Service- would diagram various activities on a soft- Now), and the issue becomes the responsi- ware development life cycle chart. In bility of an application maintenance team. DevOps, technical staff that would once But the original developer of the code, hav- have sat in the IT operations function— ing gotten wind of a problem, may go back whose work kicks in later—are moved into in and try to fix it. The net result is that the product development teams, where they sometimes both the development and have a say in how the code is built. There maintenance teams end up working on the should also be input early on from those re- same software, at the same time, resulting sponsible for a company’s data architecture in inconsistencies, integration problems, and cybersecurity. The shift left of activity and stability issues. By contrast, at compa- and expertise is one way all the code that’s nies that adopt DevOps practices, issues being created—often in many different with released software automatically regis- business units—can get to market quickly ter on the backlog of the development and with the necessary level of security.

The Boston Consulting Group | Going All In with DevOps 3 A particularly important CIO responsibility ing and infrastructure service develop- with DevOps is the implementation of an ment. Without these capabilities, continu- optimal infrastructure environment. The ous delivery and continuous integration business-oriented development teams need aren’t possible, making agile’s promised infrastructure-independent platforms so speed and reliability benefits hard to that they don’t have to worry about com- achieve. patibility. In DevOps, managing this and providing the application development Remake the operating model through toolkit are significant parts of the IT orga- automation. There is a huge benefit if a nization’s responsibility. team, instead of going through a cumber- some approval process that might last Netflix, the global streaming video service, weeks or months, can add a feature or plug provides an example of the kind of benefits a dangerous security hole with relatively that can come from embracing DevOps. little organizational oversight, and in the Netflix captures these benefits through the best case with just a few mouse clicks. efforts of a central engineering operations Automation, one of the pillars of DevOps, group (a sort of specialty IT team) whose makes that possible. But the decisions mandate is to maximize the performance of surrounding automation are complicated, newly released software and to make soft- and there is always the chance that a ware development teams more efficient. company will take a while to gain its footing. For this reason, the where and how At Netflix, developers benefit from a com- of introducing automation is a key decision mon set of tools, services, and infrastruc- for any company moving to DevOps. ture management capabilities—a “paved road,” as Netflix calls it—to traverse the A good place to start is with test automa- normally bumpy path to new software cre- tion. In our experience, the benefits of cov- ation. The paved road and the engineering ering more new code through automated operations group have been instrumental testing can on its own justify a move to in helping Netflix release new code—se- DevOps. Consider the ever-present risk of cure, reliable code—to multiple geographic late-stage delays and the costs they create. regions within minutes. With traditional waterfall and even sometimes with agile development, testing takes Assisting and speeding up software deploy- place once the code is complete. Significant ments in this way require IT staff to devel- problems may be discovered just as the op skills they didn’t need previously. For in- code is supposed to go live. By contrast, in stance, enterprise architects must take a the DevOps paradigm, code is developed much stronger hand in defining IT archi- iteratively and tested regularly. This makes tecture strategy, especially with respect to it less likely that coding issues will emerge platform options. And IT organizations at the last minute. (See Exhibit 2.) must adopt technical mechanisms—like containers and microservices—that allow Valuable as it is, automated testing must coding teams to write reusable software be rolled out in stages. Companies should and to do it faster. (See the sidebar start with the parts of their architecture “DevOps’ Technical Underpinnings.”) where they have already started to transi- tion to agile models. After they’ve had IT organizations must also acquire some some success, they can use automation to brand-new technical capabilities. For in- cover more of their code. stance, they must hire or develop quality engineers. These engineers should be em- Some of the companies that have set the bedded in the software development team pace in digital services, such as Google, and should ensure that rigorous testing have reliability targets well above 99%— happens early in the process. The IT opera- meaning that they expect the software they tions staff must likewise acquire or develop release, with the help of automated testing, new expertise, such as reliability engineer- to work immediately and in pretty much

The Boston Consulting Group | Going All In with DevOps 4 DEVOPS’ TECHNICAL UNDERPINNINGS

IT staff must be familiar with various avoid version control issues. Among the technical tools and approaches in order open-source tools used for code reposito- to implement DevOps. Here are seven of ries are Bitbucket and GitLab. the most important. Continuous Integration. A develop- Containers. A type of virtualization that ment practice that promotes single- keeps software running reliably when it source code management and compre- is moved from one computing environ- hensive automated testing, allowing ment to another. By bundling new code developers to add code to a common with everything needed to run it, a repository as often as several times a container makes it possible for software day, in a highly automated way. This development teams to ignore differences ensures that all development teams are in operating systems and underlying using the latest version of an application. infrastructure. Two open-source technol- Open-source versions include GitLab CI ogies that help with containerization are and Jenkins. Docker and Kubernetes. Continuous Delivery. A discipline for Microservices. A programming archi- building software that enables the tecture that gives developers access to software to be moved to a staging area at application functionality at a very any time. Continuous delivery tools granular level. Microservices make it include Bamboo and Jenkins. easier to continually deliver and deploy large, complex applications. Continuous Deployment. A practice that allows tested software to be re- Code Repository. A database contain- leased, sometimes with not much more ing the source code of an application. than the push of a button. Continuous When centralized and actively managed, deployment sharply reduces overhead code repositories improve the consis- and can be an invaluable tool for tency and stability of code, and help resolving issues quickly.

all instances. Google, of course, was built to development teams expect, the tests enable rapid software releases and service should be synchronized with batch process- improvements. Non–digital natives don’t es, including prescheduled data transfers need to ensure reliability on the same scale, and transactions. Since batch processes are but when it comes to digital services they can often designed to take place overnight, the learn from Google and other digitally ad- IT organization may want to run the auto- vanced companies—and they must. After all, mated tests overnight, too. a traditional company with a mission-critical digital application—like a financial services company rolling out a smartphone payment Getting Started feature—can no more afford to release bad Companies can’t just brush aside their cur- software than Google can. rent software development practices and make a wholesale move to DevOps; it in- With traditional companies’ legacy sys- volves too much change and training and tems, such as payroll or enterprise resource would create too much disruption with ex- planning, the dynamics are necessarily a isting systems and products. DevOps needs little different. Companies can still do auto- to be phased in. mated testing of their legacy systems, and in many cases they already do. But in order The first step should be to find an applica- to support the faster release cycles agile tion that has low levels of dependency

The Boston Consulting Group | Going All In with DevOps 5 Exhibit 2 | With DevOps, Bugs Are Spotted Earlier and Fixed More Economically

Plan Code Build Test Deploy Release Operate Monitor

Bug identified by a programming buddy (pair programming) Bug identified Requirement or design defect identified through in production close business and IT collaboration

Design defect or bug identified by having COST OF quality as part of the design process Design flaw identified through Requirements defect identified through DETECTING QA testing ERRORS Bug identified during Bug identified during a test with a real-world user an automated user test independent testing

Bug identified during Bug identified during systems an automated integration testing performance test Bug identified during a QA review or inspection

ERROR DETECTION ZONE USING BOTH AGILE AND DEVOPS ERROR DETECTION ZONE USING AGILE ONLY

Sources: Puppet, 2016 State of DevOps Report; BCG analysis.

with other applications—perhaps a pro- efficiency, in the form of lower costs of re- curement portal for a manufacturing com- work and an overall increase in the number pany or a savings platform for a bank—and of automated processes. After an introduc- run a pilot project to learn the DevOps tory period like this, the company can cre- model and fine-tune the practices. ate a road map to start applying DevOps practices to other software and infrastruc- In the pilot, a team of developers and IT ture platforms and to other parts of its tech- engineers lays out a technical plan—estab- nology environment. The road map should lishing a central code repository and creat- include a decision about the suite of tools ing a testing framework so testing automa- to be used and the sequence in which tion can start. Once this is in place, DevOps will be implemented in other parts continuous integration and continuous de- of the company and for other platforms. livery can begin. These processes make it possible for the development team to focus on writing code and not on manually DevOps and the Customer checking for bugs and functionality The example of a European travel compa- problems. ny helps demonstrate why DevOps is turn- ing into a must-have. At companies with complex legacy systems, continuous integration and continuous de- The company was unable to make pricing livery are two separate phases. By contrast, updates to its core booking system at the at digital natives, both approaches are core height of its main selling season. Previous to software development, and a digital na- updates had exposed the fragility of the tive may already be thinking about other system, and business managers had im- ways of enhancing the software release posed a policy of no changes during peak process. This explains why developers at periods. the most digitally adept companies often see their code fixes go live within days, There was nothing unusual about the com- hours, or even minutes. pany’s monolithic software infrastructure or the policies to accommodate it. Howev- The DevOps pilot needn’t go on indefinite- er, the deliberate approach to software de- ly. Within six months, it should be possible velopment had left the company unable to to see benefits. These typically take the respond, at the most important time of form of agility, which translates into more year, to new pricing or product proposi- software releases per week; quality, which tions from competitors. If a seven-day trip stems from increased testing coverage; and to Belize was suddenly being discounted to

The Boston Consulting Group | Going All In with DevOps 6 $1,800 on other travel websites, it would Sooner or later, most companies are going still be going for $2,100 on the company’s to find themselves in a similar position. site. Dynamic pricing updates required a That is, they are going to see that one of software change, but the company’s release their competitors is doing something faster, process limited the speed at which such with fewer security and quality issues, and changes could be made. at lower cost. And they are going to need to take action to narrow the gap. Recognizing that its software development processes were hurting the business, the DevOps is a way to do this. The implemen- company adopted some DevOps practices, tation of DevOps involves organization and including continuous integration. As it did process changes that take place well out of so, the quality of its software releases and sight of most customers. But customers will the overall resilience of its system im- be expecting the benefits. For companies proved to such an extent that management that don’t deliver, there may not be a lifted the change freeze. Thereafter, the second chance. company was able to be much more responsive to competitors’ moves during the industry’s peak selling season.

About the Authors Andrew Agerbak is a director in the London office of The Boston Consulting Group. You may contact him by email at [email protected].

Kaj Burchardi is a managing director of BCG Platinion in Amsterdam. You may contact him by email at [email protected].

Steven Kok is a project leader in BCG’s London office. You may contact him by email at kok.steven@bcg. com.

Fabrice Lebegue is a managing director of BCG Platinion in Montreal. You may contact him by email at [email protected].

Christian N. Schmid is a principal in the firm’s Munich office. You may contact him by email at [email protected].

The Boston Consulting Group (BCG) is a global management consulting firm and the world’s leading advi- sor on business strategy. We partner with clients from the private, public, and not-for-profit sectors in all regions to identify their highest-value opportunities, address their most critical challenges, and transform their enterprises. Our customized approach combines deep insight into the dynamics of companies and markets with close collaboration at all levels of the client organization. This ensures that our clients achieve sustainable competitive advantage, build more capable organizations, and secure lasting results. Founded in 1963, BCG is a private company with offices in more than 90 cities in 50 countries. For more information, please visit bcg.com.

The Boston Consulting Group | Going All In with DevOps 7