5G Ready Software Defined Test Automation @Rakuten Japan

#CLUS 5G Ready Software Defined Test Automation @Rakuten Japan

Vijay Raghavendran, vCTO, CX Cloud & Automation Chocks Ramiah, Senior Software Architect Noam Ben Gal, Product Architect, Cloud & Automation @cdafInterest

BRKSPM-2018 #CLUS Agenda

• Customer Imperatives in a 5G Ready Cloud Native Environment

• Continuous Deployment and Automation Framework(CDAF)

• CDAF at Rakuten

• Lessons Learnt – People, Process and Technology

• CDAF Offerings

• Conclusion and Q&A

Questions? Use Cisco Webex Teams to chat with the speaker after the session How 1 Find this session in the Cisco Live Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKSPM-2018 by the speaker until June 16, 2019.

The industry is transforming from a traditional hardware-centric architecture to a software-driven, cloud-native environment based on microservices and open- source applications. Customers need:

Operational Security / Enhanced User Ongoing Revenue Efficiency Governance Experience Growth

Limited Visibility Code Quality Manual Process in Agility & Productivity across SW lifecycle Concerns Complex Environment Issues

IDLE TIME IDLE TIME IDLE TIME IDLE TIME

Functional Code | Integrate | Build defect defect Performance defect Deploy to Production & UA Testing Testing BUILD TEST DEPLOY OPERATE Customer Experience

RE-WORK RE-WORK

Developers Testers Release Team Operations

Hyper Automation of CICD. Agile Software Continuous Test Ease of Use & AI/ML Engine for Software Delivery Automation Efficiency Recommendation

Automate Everything, Everywhere - Pipeline

Code | Integrate | Build Functional Performance Deploy to Production & UA Testing Testing BUILD TEST DEPLOY OPERATE Customer Experience

Developers Testers Release Team Operations

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 Continuous Deployment and Automation Framework(CDAF) • Offers a fully-integrated DevOps platform for automated software integration, delivery & Increase efficiency, reduce deployment OpEx, drive faster innovation, and deliver a superior user • Enables end-to-end Continuous Delivery between multiple vendors experience

• Robotic Process Automation(RPA) from integration to pre-production of critical processes including scope, development, test, defect and release management as well as Industry’s first 5G-ready fully- security compliance. integrated cloud-native software development and automation • Yes/No decision for Agile SW cert based on Cisco Patent pending Engine(In-Development) framework

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 Cisco CDAF methodology and components Continuous Deployment Scope Develop Test Defect Governance Release Deploy Checkout code Execute Officially Deploy a Define user, Defect Security from SCM & automatic tests release the published requirements discovery, filing compliance and artifacts repo, (unit, functional, package containerized and test and tracking code threat and build the service level) image package strategy. prevention project to an execution Management environment Epics, Compiled Test reports Bugs, Track PEN test, Release Solution features, user code from BU Dashboard: Test SAST, Management deployed in Artifacts story, sprints, images SW Release DAST,RASP Dashboard Customer Lab for verification Toolset CDAF CICD Test Automation Framework Plan, Track, Orchestrate, Provisioning, Test and Release Management

Rally Issue/ SLCM Defect GitHub Source Control Management

Jenkins Automation of pipeline Artifactory Test Tools JFrog, Docker Artifactory CXTM, Artifact Management & Repository TM500, IXIA, JFrog/docker Accuver

Istio gateway

Istio Service mesh CDAF Microservices based CDAF Promethe us v2.2 Platform:

ElasticSearch Pod Pod • Zero down time deployment /

Artifactor upgrades Ji Pod y ra • Auto Scalable Pod Pod • Deployable in 20 min on any

Grafa Cloud/HW MariaDB Githu CDAF LDAP Kibana na b v2.0 • Full stack included –

Pod Pod Pod Pod Pod Pod apps/tools, monitoring , + logging & metrics. • Each component is individually scalable Node 1 Node 2 Node 3 Node 3 Node 5 Node N • Self-healing ( moving to OpenStack/AWS/Azure/GCE/Bare-Metal healthy state )

Your network. Our CI/CD Platform and Expertise

Microservices Agile SLCM SW Release Process Agile Defect -Based for Co- Management Automation Tracking Architecture Development AI/ML based /Resolution Open source tools Vendor BU, CX, Multiple package Co-develop and Direct access for - Easy to deploy & vendor interaction versions releases & implement with customer OnPrem integrate tracking multiple vendors resource to BU development

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 CDAF Implementation – Eliminate Process Delay Multivendor product Certification OnPrem testing –Functional & security, Managed by Cisco Packaged for deployment

CDAF CICD Vendor Telemetry Customer CICD Platform Platform

Pre-Prod Test Integration Test Quality Gate Quality Gate Weekly Production Release Network Customer Customer Vendor SCM Customer

FCS Release Function Pack Production Development Support Vendor Customer SCM Customer

Software Defined, Programmable Infrastructure Other MNO

Services Anywhere

Internet Transport Cloud RAN Network

xEdge Datacenter Rakuten Datacenters

Rakuten Hybrid Distributed Carrier-Grade Telcocloud Platform Cloud Software Defined, Programmable Infrastructure

Horizontal - OSS/BSS (InnoEye) Zero touch, automated, fully virtualized cloud native network to offer global services

Horizontal Multi-domain Orchestration (NSO) Other MNO VNF Manager (ESC) CBAM

Macro vCU SAEGW-C SAEGW-U CMS CAS TAS W-SG MRF cGNAT Radio

Smallcell+ vDU MME ePDG VNFM SDNS WRG PNS CRDL PCRF Rakuten Hybrid WiFi Radio Cloud vEMS SaMOG Ultra-EM CSCF RMS M-STORE SDC PRS HSS

Cassandra CRDL Cassandra DRA

Admin Admin CiscoAutomation Horizontal TelcoCloud Software (CVIM)

Hardware Routers DC Fabric Compute

Cisco Nokia NEC Quanta Altiostar Mavenir

- 10 vendors – Cisco, Nokia, Altiostar, InnoEye, Mavenir, Netcracker, Fujitsu, NEC, TechM & Quanta - 104+ Software packages(VNF & other Microservices)

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 Rakuten Mobile Network 100% increase in Spearheading the Industry’s first 5G- time-to-market ready, cloud-native mobile network

Deploying Cisco Continuous Deployment and 100% increase in Automation Framework™ (CDAF) software quality

Rakuten needed a software-defined CICD and test automation framework for our Cloud Mobility Solution. This is to enable similar Improved agility outcomes as Netflix and other cloud native architectures in implementing hundreds of changes through agile software updates. We co-developed through 75% test the Cisco CDAF platform and it’s been a game-changer for us. Cisco automation and Rakuten Engineers have deployed this industry first 5G-ready CICD solution in our R&D & Product Certification Lab, This has helped deliver improved code quality, accelerate multi-vendor co-development and speed up innovation. We are looking forward to seeing higher cost saving in testing and certification without compromising quality. Tareq Amin Continuous CTO, Rakuten Mobile, Inc. Innovation through software delivery automation

Rakuten’s Innovation Center

Continuous SW Multi-Vendor Delivery & Deployment Co-Development

Industry’s First 5G-Ready Vendor Automated CICD Test Rakuten 5G-ready R&D Labs Framework Cloud

Real-time Zero-Downtime Feedback SW upgrades

Customer Experience

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 Scope Management Brings Org Together for Agile Software Lifecycle Management CDAF People AMO Automation, Operations Testing & CICD (Customer Program Team Delivery Team Delivery Team Team & Vendors)

Planning: Stories: Stories: Stories: Collaborate RAN OSS/BSS Build, HLD Test, Process LLD NSO, NDAP Schedule Scripting. Automation Reporting Configs. Defect. M2, M3, M4 RIU, VDU Test_RIU

Release Management GitHub GitHub GitHub Reporting Technology Test Artifactory Manager

Jenkins Jenkins Defect

VDU VCU Feature

AMO Operations Automation Test CICD User-story

Network Setup – Monitoring – Automation of Lab Process HLD Integration & Pre- NMS/OSS Configs Automation Prod Child-story

Test Cases & Test LLD-Configs, Automation of script creation – Code Delivery Topology Software Upgrade manual & automated

Report Generation Test Completion – in Release Report generation Dashboard

Software pulled from vendors

Multiple versions maintained

For Test & Production

Summary: Number of % of Software available Vendor • High vendor adoption Software(VNFs & OS) in Artifactory • OSS for Distribution Cisco 16 95% • Engg Productivity Altiostar 7 90% Nokia 27 51%

Mavenir 5 33% Note: NetCracker(BSS) 0 (Note) 0 • BSS/OSS Development uses CDAF platform for build

Note: Artifactory includes support for HTTP/HTTPS, FTP/SFTP, TFTP/STFTP, SCP

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 Test Management Environment & Framework Dev Test Integration Pre-Production Production Develop / Build / Unit Test Fetch / Deploy / Configure / Load Fetch / Deploy / Configure / Load Test Fetch / Deploy / Configure / Owner: Vendors Test Data / (Acceptance) Test Data / Acceptance Test Monitor KPIs Owner: Vendor Owner: Rakuten Owner: Rakuten Purpose: Purpose: Purpose: Purpose: • Development • Test new release • Execute acceptance tests on • Production service • Executed unit tests • First integration point – One or Integration approved release more vendor SW • Execute acceptance tests Software/Configuration Status: Key Highlights from Rakuten Test Environment• Sprint status • Release (Candidate) status Software/Configuration• Two Status:Environments Software/Configuration Status: • Release builds & final configs Software/Configuration Status: • Feature/Development status • Sprint status • Release status • Feature/Development• Integration builds & in • Approve & Pre weekly/bi-Production-weekly SW & Change Control: • Release builds & final configs development configs in-development configs • Executive approval for Production Change Control: Release Change• Control:Pre -Production• Scrum expensive master for releasing environment – Replica ofChange Production Control: • Component Dev Manager Release Candidate to Pre- Lab / Compute Resources: • Production Release Manager • Managing IntegrationProduction Environment• Vendor owned Reserved Rakuten by LabVendors Lab / Compute Resources: Environment • Release Candidates• Vendor reserved provided Rakuten lab by• Labvendors Portal and Compute after & Acceptance test Lab / Compute Resources: environment Storage, Test Tools Lab / Compute Resources: • Vendor• R&DRakuten Lab schedules• Lab Portal andTesting Compute & in Pre-Production • CelCom Production DC Storage, Test tools Configuration: Configuration: • Must have all solution components Configuration: • Can be one vendor or Multiple • Must be production like • Production vendor for Interoperability (redundancy, scale, etc) Vendors Rakuten Responsibility

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 Automated Test Environment Creation – CDAF & OSS Integration Dept Head Rakuten Lab Mgr. RM Dashboard 1 Feature/Fix

2 SW Download Requester: Advisors: Trigger OSS Vendors, 3 CloudOps Artifactory AMO/PMO 6 AMO/PMO SW Pull & Mirror Dept Head 4 VNF Instantiation 7 7 Trigger OSS on Automated Test Priority from 1 1 5 On Feature/Fix SW Pull & LMP 3 8 8 VNF Instantiation OSS OSS 2 4 Kiba 6 Totsuka Artifactory Artifactory 4 8

et Intern Pre-production Production (VNFs) (VNFs)

GUI / API

Projects Test scripts Test Cases Robot Framework - Description - Procedure - Pass/fail criteria Device & Feature Abstraction Libraries Results

Reporting Execution environment

E2E Test Automation • Automated Test Tag Selection based on SW • Single Test Result Dashboard • Single UX

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 Defect Management Integration with JIRA Service Request • JIRA Integrated • Single UX • Automated Issue creation

RM Dashboard • Single RM Dashboard • LMP DB with all info • Any visual Analytics • Patent pending ML/AI SW approval (yes/no)

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 Cisco Rakuten Co-Development Other New Microservice LMP Core Platform v1.0 LMP Core Platform v1.1 (12 Microservices) (15 Microservices) Core Platform

Cisco CX Development

Merged & Automated Tested New Cisco Secure On LMP Microservice Development Lifecycle(CSDL)

LMP Core Platform Existing version1.0 LMP Core Platform v1.1-v2.0 Semi- (12 Microservices) (15 Microservices) Automated Support Maintenance Rakuten developed Microservice Rakuten

Cisco and Rakuten Co-Development

• Microservices Based – Rakuten Develops

• Merged & Quality conformance with Core Platform

• Examples of Co-Dev in LMP • VPN Access Management • Lab Security Approval • Reservation of Pre-Production Environment • VNF Software Results in a particular period • OSS Integration to manage Test Environment

People, Process, Technology 5G Readiness – People

Scope Develop Test Defect Release Management Management Management Management Management

Business Initiatives are communicated in meetings and discussions (manual)

CDAF Challenges : People and Process • Different tools were used to capture requirements for 72 VNFs • Business expectations (like scalability, resiliency) not available for all features • Agile transformation was new to many vendors • Confusion in taxonomy classification like User stories vs Use Cases

USER STORIES USE CASES

Scoping & • Need agreement with Developers and • Facilitate release planning Sizing customers Level of • Leave implementation details • No fixed format Completeness Language & • Use Conversations for details • Accepted by customers & developers Purpose

Lifespan • Developed in one sprint • Developed in multiple sprints

Technical • Not included • Included Details • Design added in HLD/LLD

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 37 People and Process :Templatized Initiatives Bring organizations from PMO, AMO, Automation, Test, CI-CD teams collaborate: • Ran the daily SCRUM call for VoLTE initiative • Taxonomy agreement • Agreement on start-end date for initiatives, test cycles • Resolve resource contentions (Lab and people)

Scope Develop Test Defect Release Management Management Management Management Management

Continuous delivery from vendors is not tracked for certification

CDAF Challenges : People, Process, Technology

• Images were delivered from vendor to customer using non-standard methods • Non-availability of certified images per VNF mapped to solution release • Quality of code across vendors impacted by too many code drops • Target platform was different from VNF certified platforms

Early Releases  Early code drops

Production Releases  General availability

Certified Releases  Certified from Lab

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 40 Technology : Artifactory as SW Distribution Center • Single repository of images • Security compliance (scan) Cisco Cloud Artifactory (Lab)

Security Scan

SDS Server

IMS pod EPC pod Tools pod

Other Artifactory Instances

Scope Develop Test Defect Release Management Management Management Management Management

Test managers do not have any efficient collaboration and approval with architects

CDAF Challenges : People, Process, Technology

Complexity in test cases with VNF feature mapping approved by architects Non-standard format of test cases from vendors Duplication of test cases across vendors Risk in managing huge amount of test cases with limited resources (lab and time)

Scope Develop Test Defect Release Management Management Management Management Management

Defects are not tracked and resolved efficiently Triaging defects across vendors is complex

CDAF Challenges : People, Process, Technology • Various tools were used to track defects • No automated mapping from defects to features or user stories or releases • Tracking defects from various environments (Integration and Pre-prod) • No real time feedback to vendor engineering when defects found

Scope Develop Test Defect Release Management Management Management Management Management

No objective certification of software in timely manner

CDAF Challenges : People, Process, Technology • DevOps vs CloudOps – Agile releases vs operational readiness • Complexity in tracking release date, versions per environment for 72 VNFs • Too many releases from vendors have to be qualified for certification • Unknown multivendor software behavior in production • Moving from water fall model to agile release model

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45 Process change – Management Process Before CDAF After CDAF(tool) Scope XLS Sheets Initiative(Rally) Test Manual Automated(Cisco TM) Release Release Manual Tracker(CDAF/Rally) Defect Manual Automated(JIRA) Configuration Manual Automated(NMS/OSS) Risk/Issue Manual Automated(JIRA) Image Manual Artifactory Scripts/Code Manual GitHub

Source: Rakuten case study

• Quick start guides and simple videos!!

Tech Influence Before CDAF After CDAF(tool) SCM Tools Unknown GitHub TM500, AccuVer, Test Tools Manual LandSlide Monitoring Tools Manual CVIM_MON, Check_MK User Management Manual OpenLDAP(CDAF) Health of Infra Manual Kibana License renewal Manual Automated(CDAF) Results Management Manual Automated(CDAF)

Source: Rakuten case study

OSS / BSS / IT Systems

User Portal Cross-Domain Workflow Engine

Network Orchestrator Analytics / Security Assurance

Service Provider Data Center Enterprise Networking

Security Collab Cloud IoT

Continuous Deployment and Automation Framework (CDAF)

Customer Network User- Planning Design Business Iteration Stories level Engineering Needs to Planning tracking Customer Requirements Network Operations Customer OSS Prototype Documentation Review with Cisco Advisory Customer Cisco Development Issues DevOps and Agile Joint Governance

Implementation/ Release Code Reviews Production Software Governance provided Install Testing and Evaluation

Operations & Business Support Systems

Business Process Automation (BPA) SW Vendor A SW Continuous Deployment and Automation Framework(CDAF) Vendor C SW Vendor B Code Commit

Network Services Orchestrator (NSO) NFVO Elastic Services Controller (ESC)

Code Build Automated Staging Automated Pre- Automated Production Cloud/VM/ Testing Environment Testing Production Testing Rollout Container Scan/Check Verification Environment Scale, Regression Stress, Hot Deployment

Operations & Business Support Systems

Business Process Automation (BPA) SW Vendor A SW Continuous Deployment and Automation Framework(CDAF) Vendor C SW Vendor B Code Commit

Cisco CloudCenter Suite (CCS) Cisco AppDaynamics (AppD) Cisco Container Platform (CCP)

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53 CDAF: Automating end-to-end Software Delivery Accelerate software development, deployment, integration and management

Analyze Develop CDAF Team Allocation: - Team Lead - DevOps Advisor - Scrum Master - CI-CD Automation Eng - Automation Tester - Production testbed SW Eng Monitor Test - Compliance test Eng - Project manager

Deploy

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54 CDAF: Automating end-to-end Software Delivery Accelerate software development, deployment, integration and management

CDAF Functionality outcomes: Scope Management Epics and US definitions, project onboarding Develop/Delivery Management Artifactory to Release mapping, SW Pull Test Management Pipeline TM Integration, Catalog and inventory of TC Defect Management Pipeline Issue mgmt. integration to US and TC Release Management Pipeline E2E view of SLCM, Tracking, Governance Governance/Security Management Pipeline SSAT/DSAT/PEN integrations CICD Software Pipelines Productized pipelines for: NSO, ESC, BPA, EPC User Management LDAP and RBAC, Vendors Monitoring for Tools Infra monitoring, Apps Monitoring, Action fwd Lab Scheduler Reservation systems integration Single UX for End to End Software Project UI and Dashboard adaptation for customer Onboarding branding Vendor Dashboard Vendor dedicated view Notification capability Email, Collaboration integration Maintenance Support for CDAF Managed solution support for the platform

CAIT Proof of Concept CAIT CAIT

CAIT CDAF Quick Start CAIT CAIT CXTM Automation PipelineCAIT

CAIT Advisory People, tools, process 1|3 year Subscription 3 available sizes

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57 Customer benefits Multivendor product Certification OnPrem testing –Functional & security, Managed by CX Packaged for deployment

Increase Improve Quality Enhance User Faster Efficiency & of Software Experience Innovation Reduce ROI Code

Through end-to- Through the AI/ML Through a fully- Through multi-vendor end process engine for integrated platform software co- delivering improved automation software development and faster recommendations productivity from testing to pre- release of software production

Global experience Technology Unmatched networking and scale visionary leadership

Engineering Professionals Fortune 100 24/7 access 3M certified companies 365 worldwide

12k Engineers, architects, advisors >50M Networking Networks 25k Expert technologies certifications installed

11 Years J.D. Power Certified Technology Service and Support Excellence

• Bring 5G Network slices into life with automation – BRKSPM-2031

• 5G Technology Updates – BRKSPM-2071

• Cisco Cloud Native 5G Architecture, products and microservices infrastructure-BRKSPM-2019

• Automated Security Audit for Cisco Virtualization Platforms – DEVWKS-1428

#CLUS BRKSPM-2018 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60 Complete your online session • Please complete your session survey after each session. Your feedback evaluation is very important.

• Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live water bottle.

• All surveys can be taken in the Cisco Live Mobile App or by logging in to the Session Catalog on ciscolive.cisco.com/us.

Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com.

Demos in the Walk-in labs Cisco campus

Meet the engineer Related sessions 1:1 meetings

#CLUS #CLUS