Mainframe agility A Deloitte DevOps experience Contents

Foreword 01 Introduction 02 Agile innovation loop 03 Today’s mainframe development challenges 04 Mainframe DevOps 05 DevOps for z Systems mainframes 06 Plan 08 Code 09

IDE capabilities Team orchestration and build automation capabilities RTC-build vs. JCL-build capabilities RTC-build automation (scheduling) RTC’s Jenkins plugin Test 12

Unit testing and automated testing capabilities Deploy 14 Additional possibilities and hybrid models 15 Areas of concern 16 Appendix 17 Mainframe agility | Introduction

Foreword

It’s no secret that mainframe release Now a new stack of modernized IBM® processes tend to be very complex, z Systems® mainframe DevOps tools offer manual, and cost inefficient ordeals, taking increased speed of continuous integration hours and even days to complete. In most (CI) and continuous deployment (CD) on cases development and deployment mainframe assets. The tools improve processes have not changed for decades. software code quality through shift-left As organizations with mainframe assets testing. An automated delivery pipeline and infrastructure move towards becoming accommodates more frequent automated truly agile, they need to tackle the challenge testing, which allows the code to go through of manual software development and more change iterations. deployment, as well as rising MIPS (Millions of Instructions per Second) costs associated DevOps can also lower costs because with automation testing on target hardware. of fewer quality problems and software rollbacks. Ensuring new feature The challenge? Mainframe applications lack development can happen as smoothly modularity, which makes the development and quickly on the backend system as process painfully slow. Even small code on the frontend, giving organizations the changes can be risky when a mainframe advantage they need to be responsive to infrastructure is running live. their respective markets. Mainframe agility | Introduction Mainframe agility | Introduction

Introduction Agile Accelerating delivery Plan Develop innovation loop • Teams are small, multi-functional, • Application features are sliced to small focusing on the end-to-end delivery batches (less than a week) to allow responsible for building, testing, and for minimum viable product (MVPs), The IBM relationship with mainframes impressive measures: IBM’s mainframes In a traditional waterfall model, teams of deploying their applications enabling rapid development started in 1952 with the IBM 701 designed have a mean time between failures architects and developers analyze, design, • Product backlog is frequently updated • Following trunk-based practices by specifically for the Department of Defense. measured in decades. code, test, and deploy in a sequential to reflect client feedback frequent code commits for better Today, mainframes remain a central figure manner via releases. It takes too long. In • Teams continuously optimized for both stability and higher throughput in the modern business world. Despite Mobile and web 2.0 applications are contrast, agile development (as shown in team and individual productivity • Developers get daily feedback from developers continuously migrating towards redefining how value is delivered to Figure 1) is performed through a series of • Operational and security aspects are acceptance, performance, and cloud-based architecture, mainframes are customers. They’ve transformed the adaptive iterations by analyzing, designing, aligned to the business domain integrated tests still leveraged by 92 of the top 100 banks industry ecosystem. And, they are developing, and testing feature sets almost worldwide, 23 of the top 25 US retailers, constantly evolving at a faster clip than in real-time. It happens at a more granular Deploy Test all 10 of the world’s 10 largest insurers, IT has ever experienced before. These level, module by module, feature by feature, and 23 of the world’s 25 largest airlines applications require a fast and nimble within each cycle. During an iteration, it is • Deployment is frequent and automated • Code commits trigger automated [1]. Companies in every industry segment deployment environment to respond to imperative that both internal and external and code promotions are standardized performance and unit tests to test the are deploying new mobile apps that have fast-paced business demands. Since many stakeholders provide feedback and ensure to avoid high risks validity of the branch (code commit is mainframes as the middleware and the of these web/mobile applications integrate the features meet their needs. The major • Blue-green deployment style, canary rejected if sanity fails) backend. Why? with existing CICS® and IMSTM backend advantages of the agile innovation loop are: releases are followed to minimize • Teams have access to production like programs, there is a need to step out of production downtime Monitor data for testing and can condition the Mainframes are built specifically for mission- the waterfall school of thought of time- • Faster feedback cycle • Examining how to manage independent test data within the pipeline critical applications, such as processing consuming releases and go with a more • Early problem identification services/applications as products, • Regular ad-hoc testing is performed banking transactions, where both reliability agile approach, seamlessly integrating “There are 6,900 tweets, 30,000 • Flexible prioritization enabling cross functional alignment and production failures are and security are paramount. The systems mainframe applications with newly • High potential for customer satisfaction and ownership to cultivate strong systematically induced to resiliency are designed with redundancies and developed software code in the mobile/web Facebook likes and 60,000 alignment to business domains backups built in. If a component fails, it and cloud domain. With an agile innovation loop for mainframe can be swapped out without affecting the software and solutions development, just Google searches per second. Monitor system. This innovative model has garnered imagine the outcomes: collaborative cross- The mainframe CICS runs 1.1m functional teams, early delivery, frequent • Dashboard statistics are readily available to the team to enable releases, rapid and flexible response to continuous monitoring transactions per second, which change, and continuous improvement. To • Daily automated security checks are run to ensure overall system achieve such high performance will likely health equates to 10bn per day.” require a transformation that isn’t free • Continuous logging is enabled and available to developers for of risks or challenges. The first step to debugging Rob Lambs minimizing potential issues is identification. Figure 1: Agile • Periodical snapshots and backups are done to ensure advanced IBM Hursley Laboratory Director [2] innovation loop rollback capability

4 5 Mainframe agility | Introduction Mainframe agility | Introduction

Today’s mainframe Mainframe DevOps development challenges Continuous integration

Despite these challenges, mainframe As with any transformation, the rewards Attracting talent Continuous editing software applications shouldn’t be thought must outweigh the risks for the initiative to The original mainframe generation is slowly Due to the long shelf life of these systems, of differently than frontend development. make sense. Understanding the obstacles retiring. With universities rarely teaching code has been refactored and migrated The same methodologies that are leads to better decisions and, ultimately, to mainframes and a steep learning curve to numerous times, leading to poor quality and applied to speed up facilitated software better outcomes. Here are a few challenges contend with (three to five years to learn even dead code. development and deployment on the cloud with today’s mainframe development. basics of mainframe development), finding can be applied to mainframe software the right people with the right experience Lack of automation development and deployment. What are Version Build Test Release Continuous Friction between development layers can be difficult. There is an increasing need Poor documentation and lack of modularity the best practices developers consistently control automation automation automation delivery As organizations try to rapidly build and in mainframe development environments in existing codebase makes testing and define? expand new software solutions that tie to shift towards the use of more modern quality assurance an uphill task. Single Compile and build Execute unit, Packaging, Release of changes back-office (system of records apps) to tools that are attractive to a new generation • Automate everything source code application without integration, service deploying, and can be done by the front-office (system of engagement apps) of developers if organizations are going MIPS costs • Track and plan everything repository human intervention tests without post-deployment push of a button they naturally incur friction. Different to sustain or increase current rates of Mainframe performance is measured in • Version control everything human intervention testing without approaches, processes, and even development. MIPS, an older measure of computing • Test everything human intervention technologies exist in these two different power and performance. Organizations • Dashboard and monitor everything Lost code layers of software development and the with mainframe assets and infrastructure End-to-end SDLC agility frequency of releases across these tiered Lack of proper source code management need to tackle the challenge of rising MIPS As a software delivery methodology and systems is introducing new challenges. The during workforce transitions can result costs associated with automation testing practice, an agile innovation loop using rapid development and deployment of top in mission critical binaries in production. on target hardware if they want to become DevOps tools for mainframes can help layer web and mobile apps doesn’t always Without an accompanying codebase, more agile. This is especially important bring frontend and backend IT operation align with the slower moving development making changes is nearly impossible. considering that mainframes continue to be teams together along with testing and Proactive monitoring (including APM) of bottom layer mainframe applications. the tool of choice for executing high volume QA teams, project management, and Mainframe app changes simply aren’t Lack of modularity application transactions in a robust and even customers in some cases. It enables and configuration management keeping up with the fast pace of web and By nature, mainframe applications lack reliable fashion. smoother development and faster work mobile app development change. modularity, which makes the development while maintaining quality and integrity. As process slow and small code changes risky. a start, a basic minimum viable DevOps Mainframe release processes are also very pipeline, like the one shown in Figure 2, complex, manual, and cost inefficient, often can drastically improve collaboration. More taking hours or days to complete. advanced and mature DevOps pipelines can also enable continuous monitoring and incorporate constant feedback from all stakeholders including customers. Figure 2: DevOps pipeline for end-to-end software development pipeline

6 7 Mainframe agility | DevOps for z Systems mainframes Mainframe agility | DevOps for z Systems mainframes

After studying and analyzing the capabilities overall benefits of such pipeline for a z/ peripherals such as DB2®, CICS® application DevOps for of the mentioned tools and their potentials, OS mainframe ecosystem. Figure 10 in code, COBOL code, ISPF, and UCD agents an attempt was made to integrate each the appendix section shows a high level for continuous code integration and code silo component into a basic DevOps architectural overview of the DevOps tools deployment. Here’s what was assessed. z Systems mainframes pipeline (see Figure 3) to evaluate the integrated with a z/OS system and its

Organizations have the potential to achieve Table 1 shows a mapping of these tools against common best practice DevOps capabilities: CODE BUILD TEST DISPLAY a better and more agile way of mainframe software development and deployment—one IBM DevOps offerings for z Systems that increases collaboration and automation IDz RTC zUnit UCD Mainframes vs. core DevOps capabilities by using a DevOps pipeline and tools. To test Jenkins this point-of-view, Deloitte evaluated IBM’s Code development X . DevOps capabilities for z/OS® Mainframe Collaboration X X Systems against today’s best DevOps IBM UrbanCode UrbanCode practices. Build automation X X Rational deploy deploy Continuous integration/development X X Team Concert A stack of modernized DevOps pipeline IBM and tools products by IBM for z Systems Continuous testing/test automation X X Rational mainframes were installed and configured Team Concert Continuous deployment/delivery X in a sandbox environment. First, a brief assessment of the following IBM software Continuous monitoring ? suite of products was performed: Maintenance zUnit developers • IDz (IBM Developer for z Systems), an IDE based on Eclipse that supports z/OS IBM IDE and RTC work together systems software development. in these cases as RTC module TM DEPLOYMENT ENVIRONMENTS • IBM’s RTC (Rational Team Concert ), is installed on top of IDz essentially a team orchestration and collaboration tool with advanced features for workflow assignment and tracking DEVELOPMENT TEST QA DISASTER RECOVERY PRODUCTION as well as debugging and building automation. Multiple dev environments Multiple test environments Multiple quality assurance Multiple backup and Multiple production • zUnit Test, a suite of capabilities within environments recovery environments environments RTC that enables automated unit testing. • UCD (Urban Code Deploy), a package zDT: z Systems development and test environment (Former RDT) management tool that enables fast and Table 1: IBM DevOps offerings for z Systems mainframes vs. core DevOps capabilities reliable code and solution deployment. Figure 3: Proposed DevOps pipeline for software solution delivery on z Systems mainframes

8 9 Mainframe agility | DevOps for z Systems mainframes Mainframe agility | DevOps for z Systems mainframes

Plan Code

For decades, mainframe application As the mainframe teams move toward Sometimes experienced developers get Applying modular coding and architectural developers have mostly worked in waterfall releasing software and code in more comfortable using ISPF “green screen” practices and refactoring mainframe environments with blue/green screen frequent iterations, it will be important to editors and resist change, ignoring applications into smaller independent access terminals. Mainframes are core understand their current development and the productivity and quality gains of components with managed helps systems that handle major business release process and challenge routines modern IDEs in finding defects earlier organizations improve the speed at which transactions for organizations. Embrace that may slow them down. Each and debugging code once defects are they deliver. Keep in mind, not every them! Investment in mainframes, be it the organization should establish a found. In addition, organizations willing deployment would need all the modules people, process, or technology, is key to standardized application delivery process to invest in customer-facing application every single time. sustainable growth. Make the shift from a for both modern distributed teams and improvements often resist using resources waterfall environment to an agile innovation legacy mainframe teams. to modernize legacy applications. The Next, the tools and capabilities were loop environment to speed up and challenge is that traditional mainframe examined including: invigorate the process. application development lifecycle tools were not designed for small, fast, automated • IDE capabilities To enable such a cultural change, leaders deployments. When considering the • Team orchestration and build can start by restructuring mainframe teams current market conditions and industry automation capabilities to include select mainframe developers disruption, keeping the status quo becomes • RTC-build vs JCL-build capabilities and pair them with agile, experienced unsustainable. • RTC-build automation developers from other platforms. Doing so • RTC’s Jenkins plugin will not only help promote an agile working Modern IDEs offer faster feedback and mindset with the mainframe development debugging tools where older applications team, but also enable other developers such as ISPF/PDF (Interactive System to understand the aging mainframe Productivity Facility/Program Development code base. With organizations currently Facility), present significant barriers to dependent only on a few developers for getting the next generation of developers their continuous operations, this transfer to work on mainframe application code. of knowledge can hold the key to future Familiar tools with integrated debuggers, high performance as mainframe developers syntax-directed editors, and interfaces begin to retire. make picking up COBOL or PL/1 relatively easy.

10 Mainframe agility | DevOps for z Systems mainframes Mainframe agility | DevOps for z Systems mainframes

IDE capabilities Team orchestration RTC-build vs. RTC-build RTC’s Jenkins

and build JCL-build automation plugin IBM Rational UrbanCode IBM Developer for z Systems (IDz) is a Team Concert deploy common Eclipse-based IDE that provides automation capabilities (scheduling) standardized IDE features for all types of IBM also provides an RTC plugin that Jenkins development, including the z/OS software capabilities allows the integration of the popular open development. IBM has add-ons that allow Traditionally a RTC-build capabilities are not limited to source CI (Continuous Integration) and CD product managers, developers, and testers (JCL) is a scripting language used on IBM personal locally invoked build requests. (Continuous Deployment) tool, Jenkins. to work on COBOL code and JCL scripts, Rational Team Concert (RTC) is a team mainframe operating systems (z/OS) to They also include scheduling mechanisms RTC already provides build automation Figure 4: Jenkins used as an orchestration while allowing users to enjoy a variety orchestration tool that is loaded in IDz, instruct the system on how to run a batch that allow stakeholders to plan and through one-click build requests: personal zUnit engine can further enhance IBM RTC, of features that increase efficiency and enabling project management workflow, job or start a subsystem. One such type organize a series of automated build build for developers on local machines zUnit, and UCD capabilities. productivity. These include: teamwork tracking or orchestration, and of batch jobs is instructing the z/OS to requests at pre-planned times. They through coordination with z/OS mainframe, supports version control or source code create load and binary files after compiling do this by changing the requirements as well as project build requests for the • Smart code coverage, code governance management. COBOL application code. However, RTC of a build definition inside a build entire build stream. RTC build allows • Unit test support has made it easier to make build requests configuration file on a z/OS mainframe automation by scheduling the builds in Jenkins • Visual control flow analysis for both Embedded features within RTC also allow directly through its feature set by creating system. Other open source auxiliary the build definition file. The feature allows data flow and program flow analysis code review in the work item flow. In and configuring JCL instructions under the orchestration and automation tools such developers to configure their own builds • Code review and work flow metrics addition, it provides tools for facilitated hood without the need for the developers as Jenkins can further enhance RTC-build using a build engine of their choice. Jenkins • ADFz Fault Analyzer, reports, and IDz code debugging and helps direct build to worry about the build process. And, if a automation. can also be connected to test automation integration requests either as a personal or project developer or architect chooses to create and orchestration as well as automated CODE & COMMIT BUILD & CONFIGURE SCAN & TEST RELEASE DEPLOY • IBM Application Performance Analyzer build—opposed to the more traditional way their own JCL instruction set, it is possible to deployments through communication with of building COBOL code through JCL scripts submit customized JCL jobs. Urban Code Deploy. See Figure 4. loaded on the z/OS mainframe along with BI monitoring Collaboration Continuous Microservices the COBOL source code. What are some benefits of Jenkins? integration

• Over 100 Jenkins plugins are available Build Cloud IAAS Logging • Integration with over 100 DevOps tools Database • Orchestration of the DevOps toolchain • End-to-end CD-pipeline management Config management Config provisioning Source code Deployment management The main advantage of incorporating an automation and orchestration engine like Networking Testing Jenkins is that it opens up the gates Release Security of a DevOps pipeline to an additional management arsenal of modern DevOps Repo management Provisioning tools. See Figure 5.

Figure 5: Jenkins can open the doors of possibilities to include other DevOps tools.

12 13 Mainframe agility | DevOps for z Systems mainframes Mainframe agility | DevOps for z Systems mainframes

Test

In general, there are few z/OS test DevOps enables a shift-left testing $7,600/defect + Unit testing and environments that are shared across strategy by detecting defects early, law suits, loss of application development teams, so there reducing costs, improving time to market, customers, and automated testing has been very little automated testing of and creating a competitive advantage. damage to brand applications and development projects. capabilities Automated and Manual Cost of fixing Automation plays a key role in shortening part of continuous defects the development lifecycle and ensuring integration process a better quality product. It also identifies zUnit test automation is another piece of validation tests earlier in the development $960/defect the IBM DevOps tool stack. It provides unit UI automation lifecycle, helping developers to better testing capabilities for different modules of filter out and fix the bugs during the a mainframe application. However, the tests development process. Based on new themselves and supporting code should be findings, implementing granular test written in COBOL code using a black box modules for each new feature or feature $240/defect test methodology. modification early in the process can save Once released API testing on cost overruns. Take a look at Figure 6. During quality into production Developers can create a test function Cost of defectsCost of assurance or for each feature that adds in the COBOL the system test application, comparing the outcome of

the unit with the desired or pre-measured Number of test cases $80/defect During build phases Speed of test execution phase outcome, hence the name. The zUnit Integration component provides all the necessary tools During coding to do black box testing right along with and unit testing other layers of testing shown in Figure 7.

Keep in mind, the test automation and unit testing can be added as a separate job in Unit testing Jenkins build automation. Now, let’s move Progression in SDLC on to deployment.

Figure 7: Agile test automation pyramid

Figure 6: National Institute of Standards & Technology (Source: GBS Industry standard study)

14 15 Mainframe agility | DevOps for z Systems mainframes Mainframe agility | Additional possibilities & hybrid models

Deploy Additional possibilities and hybrid models

As continuous integration produces builds configurations are standardized to reduce UrbanCode Deploy is a multi-tier A standardized deployment process Implementing DevOps on mainframes that require constant changes, have large at regular intervals, the artifacts can be risks and vulnerabilities in the deployment deployment automation tool for z Systems consists of a package detailing the group provides a platform to catch up with the dependencies, or cause issues when deployed to a variety of environments, process. that offers out-of-the-box integrations with of deployable artifacts as defined by fast-moving distributed applications hosted deployed to production. such as SIT, UAT, and Stage, and pass Rational Team Concert and pushes artifacts the build engine’s ship list. It is relayed on the cloud. However, mainframes with IBM/ Containerized through appropriate quality gates all the In the intensely shared nature of the z/OS out upon building. An application comprised through UCD Agents running on an all- their tightly coupled monolithic architecture Additional possibilities are available to Docker container app way to production. Continuous delivery environment, deployment processes are of UCD components work together to target machine where the package is to and dependencies across programs still organizations with mainframe assets and facilitates this by automating the delivery highly structured, requiring all changes provide a business function. Applications be deployed. A deployment process set pose a significant challenge for a fast can help them better manage technical IBM/ pipeline. Highly mature organizations to follow a common process. With a include application processes, which deploy consists of component processes deploying risk-free and efficient release process. On debt and work in progress. Take for example IBM z/OS on Containerized have automated deployment capabilities, deployment capability locked in a system the components. Applications also contain UCD components, application processes any given day, a developer making a single an even containerization of monolithic z Systems Docker container app enabling them to push changes as they that does not coordinate well with other environments, which represent the target deploying collections of UCD components, change in any given program can have mainframe applications with the use of come in as opposed to bundling up changes environments, a comprehensive and systems that the application components and generic processes running automated multiple impacts across the application. modern containerization solutions, such or due to strenuous manual deployment automated deployment solution can be are deployed on. tasks on target systems. See Figure 8. This intertwined nature of code slows as Docker. Another possibility? Combine IBM/ Containerized processes. Environment and security difficult to achieve. down the development process and makes COBOL z/OS application development using IBM LinuxOne Docker container app deployment to production a very risky more modern procedural environment on Emperor II process. suites. Do it using RESTFUL APIs or Java APIs, which can help bring additional talent IBM/ Containerized Creating modular applications that can be into the landscape of mainframe application Docker container app modified, built, and released independently modernization and development. becomes increasingly important. Interdependent COBOL programs need Along with the focus on software production Figure 9: Containerization example on to be refactored into smaller individual modernization for mainframes, it may make IBM Mainframes (z/OS & LinuxOne) components, essentially representing sense to encompass the use of two or more with Docker containers themselves as a micro service wrapped additional solutions: 1) containerization to around APIs to communicate with other further enhance and maximize more rapid programs in the application. scale up and 2) continuous mainframe solution development and deployment. SCM BUILD PACKAGE DEPLOY TEST STAGE PROD As refactoring large complex mainframe Containerization of COBOL applications applications can be a challenging process, can provide the ability to rapidly scale up organizations should target the applications the mainframe infrastructure. (Shown in Figure 9.) DEV

Figure 8: Deployment after packaging on test, stage, and production environments

16 17 Mainframe agility | Areas of concern Mainframe agility | Appendix

Areas of concern Appendix

Given the current challenges with Only after carefully examining and References DB2 mainframe development and its widespread considering such areas of risks on a RTC use in various industries, DevOps isn’t case-by-case basis, can an organization Server just an improvement, but a necessity for decide on a DevOps pipeline that includes 1. R. Radcliffe, “Mobile to organizations if they are going to maintain processes and procedures to address above Mainframe DevOps for a competitive advantage. However, it’s calculated areas of concern. Dummies,” IBM, 2015, pp. 09-10. important to consider possible areas of 2. “Digital Transformation Needs concern as well, such as: Mainframes aren’t going away any time Mainframe DevOps,” Forrester COBOL soon. A new stack of modernized IBM z Research, June 20, 2016. Code • Cultural shift. Moving towards Systems mainframe DevOps tools offer continuous delivery can be daunting increased speed and an agile innovation for mainframe shops that have focused loop of continuous integration and on application stability and hyper- deployment on mainframe assets. The tools CICS rigorous change management, rather improve software code quality through both Developers z/OS App than on faster iterative delivery and shift-left and automated testing. Finally, rapid response. Coaching teams in agile there’s a swift, modern way of development methodologies and concepts can help for legacy, backend systems. them make the shift. • Archaic builds. Due to the large size ISPF UCD Agent UCD Server of the application, developers generally build only the changes. It is common to find parts of an application, which have not been rebuilt in decades, and that represents a challenge. Modularization of the applications can significantly mitigate this problem. • Point of diminishing returns. As more and more complexity is added DEV TEST QA PROD to a DevOps pipeline, DevOps teams ENV ENV ENV ENV should consider the costs. Only adopt additional agile and DevOps processes and technologies to the point that they don’t result in diminishing returns on Figure 10: Internal high-level architectural view of integration of z/OS systems DevOps tools with a z/OS mainframe system and its peripherals investment of time and resources.

18 19 Mainframe agility | Appendix

Notes Authors

Arthur Papian Gorji Digital Integration/DevOps Senior Consultant Deloitte Inc. [email protected]

Prem Suresh Kumar Digital Integration/DevOps Consultant Deloitte Inc. [email protected]

Contributors

Robert Miller IBM Alliance Solution Architect DC Specialist Leader, Deloitte Consulting —Technology Alliances [email protected]

21 Deloitte refers to one or more of Deloitte Touche Tohmatsu Limited, a UK private company limited by guarantee (“DTTL”), its network of member firms, and their related entities. DTTL and each of its member firms are legally separate and independent entities. DTTL (also referred to as “Deloitte Global”) does not provide services to clients. In the United States, Deloitte refers to one or more of the US member firms of DTTL, their related entities that operate using the “Deloitte” name in the United States and their respective affiliates. Certain services may not be available to attest clients under the rules and regulations of public accounting. Please see www.deloitte.com/about to learn more about our global network of member firms.

This communication contains general information only, and none of Deloitte Touche Tohmatsu Limited, its member firms, or their related entities (collectively, the “Deloitte Network) is, by means of this communication, rendering professional advice or services. Before making any decision or taking any action that may affect your finances or business, you should consult a qualified professional adviser.

No entity in the Deloitte Network shall be responsible for any loss whatsoever sustained by any person who relies on this communication.

Copyright © 2018 Deloitte Development LLC. All rights reserved.