Transforming Entertainment Through Devops

VIEW POINT

TRANSFORMING ENTERTAINMENT THROUGH DEVOPS

Pressure to rapidly release new products revealed cracks in one of the largest U.S. broadcasters’ software development processes. The company ultimately took the bold step to adopt Agile and DevOps. This required a fundamental shift in processes, technology and culture. But the benefits were worth it. less risk and better quality. Cross-functional collaboration, which is the bedrock of Transforming DevOps, nurtures innovation in developing business solutions (see Figure 1). entertainment through DevOps Figure 1. DevOps model — breaking silos to form a network One of the world’s largest Architecture entertainment companies wanted to offer enhanced viewing experiences to REQ. BACKLOG its customers. But it was taking months to translate new business ideas into live applications that could support this vision. And even then, the end Business product was mired in quality issues. Yet the business turned this around within two years. The company radically redesigned its operating model. It adopted Agile and DevOps Production practices that reduced environment Dev & QA provisioning and software deployment Collaboration Ops cycles from three months to 26 seconds, saved 85% of testing efforts through automation and increased sales by 55%. What are Agile and DevOps? In essence, Agile and DevOps are both approaches that reduce time and Release management increase quality by collapsing down Source: Infosys the traditional chain of events in a software development project, and encouraging collaboration and rapid DevOps vs. traditional waterfall model decision-making between the different people involved. The waterfall model, a term used to describe the traditional More specifically, Agile is an iterative approach to developing software, involves a sequential set software development approach of processes to design, develop, test and launch a product. wherein large functionality is Each team finishes its part of the project and passes it on to the broken down into smaller parts and next team to continue. There is little communication between developed by teams comprising the teams. This can become tricky in large projects since there experts from different areas (business, is no scope to incorporate rapidly changing requirements in development, testing, etc.). The teams such environments. Issues found in the later stages also lead are equipped to swiftly respond to dangerously long delays and high rework costs. to changes at any point in the development cycle. DevOps, on the other hand, insists on cross-functional teams DevOps is a set of practices that and therefore more interaction between developers, testers and integrate the development and designers. It involves shorter cycles of development, testing and operation activities of software deployment. It is driven by the “fail fast, early and often” mantra. development processes to enable With a continuous feedback mechanism, modifications at any faster development and deployment stage can be easily fixed and hence present fewer risks. of software, with higher efficiency,

External Document © 2019 Infosys Limited used traditional ways of content It seemed impossible to address these Transforming the viewer distribution, were affected. This issues using the existing monolithic experience through company wanted to get into the OTT architecture and processes. This is streaming service business as soon as what drove the company to consider Agile and DevOps possible. It intended to deliver rich and the radical transformation of adopting The subject of this case study is one unified experiences across multiple Agile and DevOps — a completely of the world’s largest entertainment devices and browsers. It also hoped different approach to what had been companies, distributing satellite to improve its website — a critical tried previously. The objective was television services in the Americas. channel for online sales, OTT content to break down silos and empower At the start of this decade, and self-service capabilities. teams to work together in order to over‑the‑top services began to gain deliver high-quality applications faster, However, there were numerous popularity. With OTT services, media efficiently and in cost-effective ways. barriers to achieving this. The time content could be directly distributed The ultimate outcome of this was to: to turn an idea into reality was too to viewers via the internet, bypassing long. Very few people involved had • Develop next-generation TV the broadcast platforms. More people a complete understanding of the viewing experiences. shifted from television sets to devices projects they were supporting. There • Implement new business ideas and such as mobile phones, tablets and was no collaboration between teams offers quicker. laptops to access media content, and and no team would take initiatives • Minimize testing and deployment advertising sales began to move to to fix the issues when the project efforts to support more projects. online platforms. encountered roadblocks. Thus, the • Reduce turnaround time through The likes of Netflix and Amazon were costs of projects spiraled and the close collaboration between already leading in this domain, and quality of the resulting products different teams. the sales of the company, which still were poor. • Mitigate risk by failing fast and failing early.

The three pillars of transformation Infosys was engaged to assist in the transformation journey. Given the scale of the transformation, a three-pronged approach to disrupt the existing process, technology and culture was designed to achieve the goals. Process transformation: Evaluation of existing model through the DevOps maturity model, identifying areas of improvement through value-stream mapping and implementing DevOps practices across the entire IT portfolio. Technology transformation: Architectural transformation, moving from a monolithic waterfall approach to a lightweight architecture made up of separately creatable and deployable parts. Cultural transformation: Reconfiguration of the operating model to increase collaboration across business, development, operations and quality teams; automation of the tracking system; promotion of a DevOps culture; and a minimization of efforts where possible.

External Document © 2019 Infosys Limited With people, technology and process involved early in the development Continuous integration innovation as the guiding principles, cycle to identify issues and enable This is a practice in which code Infosys helped the company define a quick rollback in case any problems developed by many developers is new operating model and develop a are identified. integrated with a shared repository large-scale DevOps and Agile adoption At first this process went smoothly, several times a day. The codes thus program across the entire portfolio. but quickly challenges started to arise. integrated are automatically built With implementation across multiple and tested. Any changes or issues Process transformation applications, each team developed its are relayed back to the developers own way of doing things. The tools, immediately, who then work on the The first task was to evaluate existing processes and practices were not next version. This is a “fail-fast-and- practices in order to identify existing common across applications. The fail-often” model. It ensures problem shortcomings and challenges. teams were drifting away from the detection at early stages to avoid A team then had to develop a DevOps important principles of Agile and major setbacks later.1 model that would meet the set goals. DevOps: collaboration, efficiency and The company, together with Infosys, reusability. This started to increase Continuous deployment first created a DevOps maturity model costs. and evaluated the company using This requires that each change in To tackle this, a central governance the system is easily deployable criteria associated with it. The DevOps team was constituted immediately. maturity model is a matrix with five through automation tools. The code, This team was composed of experts which is continuously delivered, is criteria: culture; design; build and from the entertainment company deployment; reporting; and testing. immediately tested by the end users. and Infosys. It brought together Their suggestions flow back to the It also has five maturity levels: crawl, the synergies between different walk, run, sprint and fly. developers, who can then incorporate application teams and increased the changes in next version. The company also conducted a reusability. Having all the teams aware This enables continuous improvement value‑stream mapping exercise linked of what the others were doing meant of the product instead of having to to the DevOps maturity assessment. that if one team had faced a challenge wait until the end of the project to The value-stream mapping exercise previously, the other teams could realize what went awry, at which point assessed the current state of processes benefit from how that team had solved it might be too expensive to correct. used to develop a product, identified it instead of starting from scratch. the improvement areas and suggested Pipeline as code delivery areas where processes could be changed without reducing value. Technology The practice of using CI and CD to develop and deploy code resembles Cross-functional scrum teams were transformation a pipeline, as shown in Figure 2. It has created, dissolving boundaries Adoption of DevOps and Agile meant a continuous feedback mechanism between the development, business, shifting from traditional technologies for continuous improvement. operations and quality teams. to a new technology stack relevant for This pipeline gets triggered when Each team was assigned the complete Agile and DevOps. Automation is the any code is developed or changed. ownership of an application — from foundation of DevOps. Infosys enabled The code quality is checked, unit development to deployment. automation by deploying continuous tested and integrated with the A “shift left” testing approach was integration, continuous deployment, repository. That version is then tested introduced. Shift left is a practice pipeline as code delivery and dual and auto-deployed. This is the basis of in which the testing team gets environment deployment practices. automated one-click deployment.

Automated Automated Automated Automated Automated functional and Development code qaulity build deployment Monitoring check testing regression testing

Continuous feedback mechanism

Source: Infosys

Dual environment deployment the code. Infrastructure as a code was Prior to Agile and DevOps adoption, practices implemented. IaC helped DevOps each team had a different way of teams automate the management of working. To bridge the gap between This is a strategy for releasing software multiple versions of code and different business and IT, the role of strategic code that eliminates downtime. In this environments (development, test, etc.) product analyst was created. Strategic approach, two identical hardware that the DevOps model entailed. product analysts were responsible for environments are set up, say blue the overarching product and ensuring A dashboard was created to monitor and green environments. One of the that communications happened across the entire pipeline, providing a environments is active at any stage different relevant teams. Agile coaches detailed view of progress at each while the other is idle. New code from Infosys trained the teams to step. A vendor-agnostic, on-demand is released in the idle environment help them adapt to the new way of cloud infrastructure was used to and thoroughly tested. Once the working. Workshops were conducted reduce dependency on a single results are satisfactory, the user to promote the DevOps culture. traffic is routed to this environment. cloud service provider. The company If any issues are encountered, traffic could use and pay only for the cloud An entirely new set of skills was is immediately redirected to the resources it consumed. This ensured required to make all of this work. idle environment that was working higher utilization, flexibility and cost- For example, the teams now had correctly. This ensures there is no efficiency. full-stack developers and DevTesters, downtime for the application, either as well as SQA testers who focus on during code deployment or in the face risk-based testing after supporting of any other issues. Cultural transformation automation of repetitive tests. Many In the traditional model, the teams people in the company were reskilled Tools and infrastructure were unaware of the work of other for these roles, and in some cases teams. Agile and DevOps disrupted experts were recruited. Shared DevOps In this enterprise, CI/CD practices this model of working in silos. teams were created, and experts in were used to implement an integrated Cross‑functional teams were created, DevOps and Agile were hired on a pipeline to achieve one-click with experts from different areas such temporary contract basis to “handhold” deployment. Many tools were used as business and development testing the teams until they gained expertise. for automation and for streamlining for each application (see Figure 3). the processes. Jenkins, an open- They were in charge of the entire source automation engine, acted as process up to delivery, which brought an orchestrator for the automation a sense of ownership to the teams. of the build, test and deployment The end-to‑end expertise in a single processes. Selenium was used for team meant a seamless development testing automation. Automated tests cycle. Any concerns, issues and could be run on multiple browsers roadblocks could be immediately and machines in parallel. Docker was resolved. The whole team was now used for automating deployment of aware of the status of the application.

CROSS-FUNCTIONAL TEAMS

Project-1 Project-2 Project-3 Project-4

Business team

Design team INDEPENDENT

Development team

Testing team TEAMS

Deployment team

Operational team

Source: Infosys

However, implementing this change trust and sense of ownership that configuration, installation), which took was not easy. came with this culture guaranteed that around three months with traditional they would not fall back into the older methods, shrank to a staggering 26 “Changing mindsets to adopt the way of working. seconds. Automating regression test DevOps and Agile way of working execution saved 85% of testing effort. was the hardest of all challenges. Technical and process challenges were Impact of easier in comparison,” says the project Stronger — better quality manager who handled this project. transformation Fail fast, early and often helped There was initial resistance to the Though the path of transformation detect the defects early in the change and disruption that DevOps was fraught with many challenges, development. This prevented defects brought in. “Fail fast and often” was not the project was a success. With the from percolating to the later stages. an easy idea to inculcate. People were dual environment deployment The reduction in rework efforts and threatened by the changing roles and strategy, there was no downtime quality issues resulted in annual skill requirements. in the live environment even if savings of $500,000. To alleviate the concerns, Infosys there were issues in the back end. conducted multiple workshops, Services worked seamlessly even More — increased revenue during the times of highest number demonstrations and value articulation The number of product releases of visitors to the website. This meant sessions for development and increased multifold after DevOps more revenues (downtimes previously operations teams. This ignited curiosity implementation. With more features had cost hundreds of thousands to and made adoption easier. Cross- and products came more sales. Peak the company). functional demos and team-building day sales revenue increased by 55%, activities helped in fostering trust. with the Agile and DevOps approach Faster — improved cycle time Overall, “show and tell” succeeded in being an important contributing assuaging fear and changing mindsets. Deployment times decreased factor. There were no leakages or With practice, people started significantly. Environment provisioning failures, which generally happen appreciating the benefits of the Agile and deployment (the process of when the websites or services are and DevOps culture. The transparency, implementing the code — testing, not accessible to users. The number

External Document © 2019 Infosys Limited of releases per year doubled every Charting out the Agile and DevOps co-located as much as possible and passing year, with more than 300 implementation plan, setting up very well connected. If not structured product features delivered in 2018. the infrastructure and technology, correctly, miscommunications and forming teams with the right skill backlogs can form, leading to dire The necessity for faster and more set, and changing the culture can consequences. efficient creation of high-quality take a couple of years and millions of software led to a revolution in the It is always easy to show success dollars. So support and confidence of way software is developed and in small pilot projects. However, leadership and buy-in from teams are delivered at this entertainment giant. scaling Agile and DevOps in a large extremely crucial. It is not alone. DevOps adoption has organization can be complicated. surged in the past few years, with Companies might be tempted to pick Changing technology, processes and 56% of global organizations now only convenient practices from Agile ways of work in large-scale projects implementing or expanding DevOps and DevOps, and skip the difficult can take time and can be complex. initiatives, according to a recent parts. But Agile and DevOps work Any failures or roadblocks can have Forrester survey.2 only if they are adopted holistically. very large and negative impact on Piecemeal adoption will ultimately the projects. While DevOps and Agile practices result in more processes without come with a lot of promises, they also any benefits. Also, teams should be demand lot of investment and time.

It is important for organizations to understand the following before embarking on this journey:

• Appraise the current situation and the challenges. • Clearly describe what the goal is that you hope to achieve. • Assess how DevOps fits into this picture. • Identify the places where DevOps and Agile can add value. • Make a detailed plan of the DevOps adoption program. • Continuously monitor progress so that problems can be addressed as early in the program as possible. • Identify the right tools and frameworks to ensure that automation can work smoothly. • Use Agile coaches and DevOps experts to train teams and project managers. • People are the most important pillars of any project. For successful Agile and DevOps adoption, cultural change involving everyone from top leadership to each team member is extremely important. Equip and support them well so that they can adapt to the new way of working.

References 1 “What is a Continuous Integration and Delivery Pipeline, and Why Is It Important?,” Codefresh, Dec 20, 2016, https://codefresh.io/continuous-integration/ continuous-integration-delivery-pipeline-important/ 2 “Five Key Trends To Benchmark DevOps Progress,” Forrester, January 31, 2019, https://www.forrester.com/report/Five+Key+Trends+To+Benchmark+DevOps +Progress/-/E-RES120481#figure1

Authors Alok Uniyal Rachana Hasyagar Vice President and Head of IT Process Consulting Consultant – Infosys Knowledge Institute [email protected] [email protected]

External Document © 2019 Infosys Limited About Infosys Knowledge Institute The Infosys Knowledge Institute helps industry leaders develop a deeper understanding of business and technology trends through compelling thought leadership. Our researchers and subject matter experts provide a fact base that aids decision making on critical business and technology issues. To view our research, visit Infosys Knowledge Institute at infosys.com/IKI

For more information, contact [email protected]

© 2019 Infosys Limited, Bengaluru, India. All Rights Reserved. Infosys believes the information in this document is accurate as of its publication date; such information is subject to change without notice. Infosys acknowledges the proprietary rights of other companies to the trademarks, product names and such other intellectual property rights mentioned in this document. Except as expressly permitted, neither this documentation nor any part of it may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, printing, photocopying, recording or otherwise, without the prior permission of Infosys Limited and/or any named intellectual property rights holders under this document.

Infosys.com | NYSE : INFY Stay Connected