Session Presentation

#CLUS Design 5G ready distributed Telco DC with Cisco ACI

Abhishek Mande, Principal System Engineer Sonu Khandelwal, Technical Marketing Engineer BRKSPM-2009

#CLUS Session Objectives

At the end of the session, the participants should be able to:

. Understand key requirements for data center fabric in 5G context.

. An approach using ACI how can we address these requirements.

Initial assumption:

. The audience already has knowledge of ACI concepts (Tenant, VRF, BD, EPG, L3Out, etc.)

Out of Scope:

. This session is not about 5G Packet Core and 5G NR.

• Impact of 5G on Telco DC • 5G ready DC solution with ACI • Automation • Distributed DC • EPC deployment • Service chaining • WAN connectivity • Security & Compliance • Operations • Customer examples • Conclusion

Africa, 2 Western Europe, 2 Africa, 2 Western Africa 2 5G Launch Status Europe, 14 Asia-Pacific19 Eastern Europe2 Europe 3 Middle East 5 North America, Asia-Pacific, 19 5G MBB North America4 1 Western Europe14 Asia-Pacific, 2 Region Country Operator Launch date Launched North America, Africa South Africa Vodacom 2018 Aug-18 (pre-com) 4 Africa South Africa Rain 2019 1Q19 Middle East , 2 Middle East, 5 Eastern Europe, Europe, 3 Eastern Europe, Asia-Pacific Taiwan Taiwan Mobile 2019 2 Asia-Pacific China China Mobile 2019 2 * Operators counted once even if mulitiple announcements have been made. Asia-Pacific South Korea SKT 2019 4Q18 Asia-Pacific Australia Telstra 2020 May-19 Region Country Operator Launch date Launched Asia-Pacific New Zealand Vodafone 2020 Western Europe France SFR 2020 Asia-Pacific New Zealand Spark 2020 Western Europe Germany Telekom 2020 Asia-Pacific South Korea KT 2019 4Q18 Western Europe Italy TIM 2020 Asia-Pacific South Korea LGU Plus 2019 4Q18 Western Europe Finland Telia 2020 Asia-Pacific Australia Optus 2019 Western Europe Finland Elisa 2018 Asia-Pacific Taiwan Chunghwa 2020 Western Europe Sweden Tele2/Telenor 2020 Asia-Pacific Philippines Globe 2Q19 Western Europe Sweden Telia 2020 Asia-Pacific Philippines PLDT 2020 Western Europe Switzerland Swisscom 2018 Asia-Pacific China China Telecom 2020 Western Europe Switzerland Sunrise 2019 Asia-Pacific Taiwan FarEastone 2020 Western Europe Switzerland Salt 2020 Asia-Pacific China China Unicom 2020 Western Europe Ireland Vodafone 4Q19 Asia-Pacific Japan DoCoMo 2020 Western Europe UK BT/ EE 2019 Asia-Pacific Japan Softbank 2020 Western Europe UK Vodafone 2019 Asia-Pacific Hong Kong HKT 2020 Western Europe UK Three 2019 Asia-Pacific Philippines Globe 2019 Eastern Europe Estonia Telia 2019 5G FWA Eastern Europe Estonia Elisa 2018 Region Country Operator Launch date Launched Europe Multiple Orange 2020 Europe Multiple Deutsche Telekom 2020 Africa South Africa Comsol 2019 Europe Multiple Vodafone 2020 Africa South Africa Vodacom 2018 Aug-18 (pre-commercial) Middle East Saudi Arabia STC 2H19 Asia-Pacific Australia Optus 2020 Middle East Bahrain Zain 2019 Asia-Pacific New Zealand Spark 2020 Middle East Qatar Ooredoo 2018 Eastern Europe Romania Orange 2018 Middle East UAE Etisalat 2020 Eastern Europe Czech Republic O2 2018 4Q18 Middle East UAE Du 2H19 Middle East UAE Etisalat 2018 North America US T-Mobile USA 2019 Middle East Saudi Arabia STC 2018 May-18 (pre-commercial) North America US AT&T 4Q18 4Q18 North America US Verizon 2018 Oct-18 North America US Verizon 4Q18 Western Europe Finland Elisa 2020 North America US Sprint 1H19 Western Europe Switzerland Sunrise 2020 Source: Ovum_5G_Service_Provider_Tracker_1Q19 #CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Customer Experience is Taking Centre Stage

Mobile ARPU, Multiple Countries

50 45 40 35 30 25 20 15 Source: EU Commission 10 5 0 2015 2016 2017 2018 2019 2020 Consumer ARPUs are Declining or Flat B2B or B2B2x Market Has Future Growth

Low Latency for better QOE and to Enable New Applications, Customer Experience Transformation

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 Service Edge Transformation The Service delivery architecture is evolving as the Service Edge functions are undergoing the following transformations:

Virtualization Decomposition Placement

Centralized, Appliance Based Mobile Gateway Distributed, Virtualized, Decomposed Mobile Gateway vPGW- UPF vPGW- PGW Services CPF

Cell Site CRAN Hub/ CO / Agg. / Regional DC Central DC Cell Site CRAN Hub CO / Agg. / Regional DC Central DC Pre-Agg. MSO / HE /Pre-Agg. MSO / HE

Public Cloud Provider Location Type  Cell Site / C-RAN Hub / CO / Agg / MSO / HE Regional DC Central DC Access Pre-Agg. 10,000’s 100s - 1000s 10’s / 100’s 10+ to Few 10’s (<10)

Terminology from a Services Edge or Applications Placement Perspective

Terminology Edge Regional DC Central DC Public Cloud we’re using Far Edge

EMEAR Frontend Data Backend Data Center Customer SP #1 Center Terminology USA Cloud Platform – Cloud Platform – Cloud Platform - Cloud Platform - SP #2 Deep Edge Far Edge Edge Core

On-Demand Infrastructure Use Cases Video, Live TV Network to User vRAN / Cloud RAN, CUPS/ Decomposed Mobile Packet Core, CUPS based BNG, (v)OLT, vCMTS with R-PHY, Gi-LAN Services etc. Connected Vehicle to Network, User User to Network Generated Live TV Operator Branded Services etc. Content streaming via CDN, Live TV with CDN, IOT, Fog Computing, AR/VR based services etc. Localized AR, IOT, Interactive Gaming, Vehicle-to-Vehicle User/Edge to User/Edge etc. Services to Businesses B2B Online Gaming, Connected Autonomous Vehicle, AR/VR Services, IOT Services, Public Cloud Hosting, 3rd Party App hosting, Edge Analytics Business Managed Services Bi-Directional (User – Network)

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 Customer Survey “Where is that Edge” Deep analysis conducted across Multiple Theatres on Top 5 Key SP use-cases

Virtual Gateways/App Cloud RAN Public Cloud Hosting Immersive Augmented Interactive Gaming Connected Autonomous (vBNG, vCMTS, CUPS, CDN) (RU, DU, CU) (AWS GG) Reality Vehicle

“Sweet Spot ??”

Public Cloud Provider C-RAN Hub/ Pre-Agg. CO/Agg/MSO/HE Regional DC Central DC

Latency Increases in this Direction

Cost of Backhaul Increases in this Direction

Operational Complexity Increases in this Direction

Cost of Overall Telco Cloud Infrastructure Likely to Increase in this Direction

• Decompose the Mobile Packet Core into control plane and a multiplicity of user planes. • Control and User Plane Separation (CUPS) in 3GPP

• Enable User Planes (UP)s to be placed anywhere in the IP network, e.g. closer to the consumer on the Edge

• Augment UPs with with in-line services

• These types of functional disaggregation are enabled with virtualized workloads

• Challenge is in optimal placement of the Edge functionality - use case dependent CUPS: Control/User Plane Separation

4G Architecture vEPCEPC eNodeB Diameter

CPRI backhaul RU BBU Backhaul aggregation IP Edge IP Core Central Datacenter

Core: decompose, virtualise & distribute

5G Architecture Edge Datacenters vCore vCore vCore API Service User plane User plane Control plane Architecture vRAN (CU) MANO SDN SR SR SR SR Analytics/ Telemetry fronthaul midhaul backhaul Automate DU IP Edge IP Core Central Datacenter Managed MEC Video • Virtual Packet Core is the #1 workload for NFVi today • Distribution of the User Plane the only way to meet 5G low latency requirements

Policy / Slice Cross Domain Automation selection

Ent 1 User Plane 1 Control IoT User Plane 2 Plane EPC User Plane 3 Consumer CUPS

Tenant 1 Tenant 2 Tenant 3 Micro DC ACI / VxLAN

VPN 1 VPN 2 IP Transport PE PE VPN 3 Segment Routing

4G Network Architecture 5G Service Based Architecture (SBA) Fixed connections with stateful signaling NW functions decoupled from Service functions via APIs

HSS NSSF AUSF UDM NRF NEF

Diameter HTTP / JSON

MME PCRF PCF AMF SMF AF

UE RAN GW DN UE RAN UPFs DN

Centralised IP ‘bearers” Control and User Plane Separation (CUPS) Policy based flow control Service centric “slices” Network exposure for NaaS via APIs

• Multiple locations with lesser footprint (1 / 2 pairs in edge DC) Multi-Site • Multiple Central/Regional DC with ACI fabrics.

• 5G Slice calls for end to end automation, need SDN to integrated with Automation MANO as well as virtualization domains like K8, Openstack.

• As U-Plane goes out, services will move along and we would need to Service Chaining chain different APN’s to different services.

• 5G calls for end to end slicing with slice management, we need DC to Analytics stream real time telemetry data outside.

Distributed Edge DC Fabric Analytics and Assurance

• Multi site SDN Controller

• Network as a Fabric NSO • Streaming Telemetry and Analytics: • Scale to MEC sites with Remote leaf Visibility and Performance Consistent Policy Model across • Network Service Assurance: fabric MSO Simulation and validation Controller SDN Fabric Management Network and Security Policy enforcement ACI Controller • Driven by virtualization ACI Controller • Multi POD / Multi POD / Fabric Automation ~ ACI RemoteACI Leaf Controller Single Fabric Single Fabric • Ops tools for lifecycle management ~ 10s VMs+ • Policy driven with Service Chaining Mini POD • SDN-ACI & SR-MPLS Integration for NaaS

Cloud Management

CRAN/Pre-Agg CO/Edge DC Regional DC Central DC sGi LAN Internet Peering • Workload Management across clouds 1000s Few 100s >100s Few 10s - Openstack Transport with SR-MPLS/SRv6 - Kubernetes • Virtual Network Connectivity across clouds

Fabric Bring up Fabric Provisioning Policy Automation Step1 Step2 Step3 • Access interfaces • VXLAN underlay policies configuration • vPC, PC, L2, L3, • Interfaces VLAN, LACP, • Tenant, EPG, BD, configuration CDP, LLDP etc. Contract, VMM between Leaf & • Fabric policies domain etc. Spine • SPAN, NTP, PTP, • IGP configuration Out-of-band • VXLAN configuration • AAA, security, RBAC

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 Fabric Automation Step 1 75% reduction in time spent bringing up network • Fabric Provisioning • Validation • Inventory

Input simple details like Fabric ✔ Connect APICs Subnet , APIC Connect switches Login to APIC to Leaf Pair Out of Band Fabric is up and in Spine Leaf and register Management IP running topology Power on APIC switches and switches & Login credential on APIC CIMC

Self developed Provisioning tool for pushing one time fabric policies as an alternative to UI based configuration

Self Provisioning Tool

API Calls to provisioning interface policies, L3out, NTP and other one time polices

Customer can use Network Service Orchestrator to provision both ACI and transport network. This is an alternative option apart from GUI and API based configuration through self developed tool

Network Service Orchestrator (NSO) API Calls to provisioning Provisioning of transport interface policies, L3out, NTP routers including DC PE routers and other one time polices Transport Network

Gateway- 100.1.1.1

Automatic deployment of Tenant, VRF, EPG, BD (Gateway IP) when End Point is detected

100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1 100.1.1.1

100.1.1.100 100.1.1.101 100.1.1.102 Vlan 100 Vlan 100 Vlan 100 Host A Host B Host C

Distributed Edge DC Fabric Analytics and Assurance

• Multi site SDN Controller

• Network as a Fabric NSO • Streaming Telemetry and Analytics: • Scale to MEC sites with Remote leaf Visibility and Performance Consistent Policy Model across • Network Service Assurance: fabric MSO Simulation and validation Controller SDN Fabric Management Network and Security Policy enforcement ACI Controller • Driven by virtualization ACI Controller Multi POD / • Fabric Automation ACI Multi POD / ~ ACI RemoteController Leaf Single Fabric Single Fabric • Ops tools for lifecycle management ~ 10s VMs+ • Policy driven with Service Chaining Mini POD • SDN-ACI & SR-MPLS Integration for NaaS

Cloud Management

Single Telco DC Campus with multiple server halls

Pod ‘A’ Pod ‘B’ Server Hall ‘A’ IP Network Server Hall ‘B’ Location ‘A’ Location ‘A’

MP-BGP - EVPN

Same Central/Regional DC

 Managed by a single APIC Cluster  End-to-end policy enforcement  Single Management and Policy Domain  Control plane fault isolation

 Multi-Pod with each Pod in different location, and with a requirement of network slicing in the transport

 ACI Multi-Pod solution is only used for management of multiple Pods in this scenario

 Traffic between Pod is forwarded through WAN and not through IPN

IP Network

MP-BGP - EVPN

Pod ‘A’ Pod ‘B’ Server Hall ‘A’ WAN Server Hall ‘B’ Location ‘A’ (IP MPLS/SR/SRv6) Location ‘B’

 VRFs are not stretched across Pods  Different VRFs in each Pod is mapped to Common VRF in the WAN  Since each Pod has different VRF, communication between Pod will happen through WAN.

IP Network Pod ‘A’ Pod ‘B’

BL1 BL2 BL3 BL4

10.1.1.1/24 L3out L3out-1 L3out-2 L3out BD1 10.1.2.1/24 VRF1 VRF2 BD2 10.1.2.0/24-> BL1, BL2 (VRF1) 10.1.1.0/24-> BL3, BL4 (VRF2) VRF2 is not present in Pod ‘A’ VRF VRF VRF1 is not present in Pod ‘B’

WAN (IP MPLS/SR/SRv6)

 50msec latency between Pods  Increase MTU support to handle VXLAN encapsulated traffic  OSPF to peer with the spine nodes and learn VTEP reachability  Multicast BiDir PIM  needed to handle  DHCP-Relay Layer 2 BUM* traffic

Pod ‘A’ IP Network Pod ‘B’ Server Hall ‘A’ Server Hall ‘B’

MP-BGP - EVPN

Same Central/Regional DC

Inter-Site VXLAN Network (ISN)

MP-BGP - EVPN

Multi-Site Orchestrator

REST ACI Site1 GUI ACI Site2 Central/Regional DC1 API Central/Regional DC2

 Separate ACI Fabrics with independent APIC clusters  MP-BGP EVPN control plane between sites  End-to-end policy definition and enforcement  Data Plane VXLAN encapsulation across sites

 No EPG, BD or VRF is stretched across ACI sites.  Communication between the sites will happen via L3out towards WAN  Customers can configure network slicing in WAN Inter-Site Network

Central Provisioning Policy Depository

L3out L3out ACI Site1 ACI Site2 Central/Regional DC1 Central/Regional DC2 WAN (IP MPLS/SR/SRv6)

 Traffic between ACI site is forwarded through WAN

 No ISN and No need for VXLAN stretch between ACI Sites

 Multi-site Orchestrator is pushing policy configuration across sites along with day-2 operations

Multi-Site Orchestrator

REST GUI ACI Site1 API ACI Site2 Central/Regional DC1 Central/Regional DC2

WAN (IP MPLS/SR/SRv6)

 OSPF on the first hop routers to peer with the spine nodes and exchange TEP address reachability  Increased end-to-end MTU support (at least 1550 B)

Inter-Site Network (ISN)

MP-BGP - EVPN

Multi-Site Orchestrator

REST GUI ACI Site1 API ACI Site2 Central/Regional DC1 Central/Regional DC2

REST GUI API • Micro-services architecture . Three MSO nodes are clustered and run concurrently (active/active) ACI Multi-Site Orchestrator 150 msec RTT (max) VM VM VM • OOB Mgmt connectivity to the APIC clusters deployed in separate sites Hypervisor • Main functions offered by MSO: 1 sec RTT (max) . Monitoring the health-state of the different ACI Sites . Provisioning of day-0 configuration to establish inter-site ….. EVPN control plane . Defining and provisioning policies across sites Site 1 Site 2 Site n . Day-2 operation functionalities

Mix of Central/Regional DC deployment with ACI Multi-Pod and Multi-Site

Inter-Site Network

IPN

Pod ‘A’ Pod ‘B’

ACI Site1 ACI Site2 Regional DC1 with 2 Pods Regional DC2 with 1 Pods

Remote Location contains Nexus 9300 connected to IP Network and fully managed by APIC cluster of Main DC

APIC and Spine Nodes IP Network L2 / L3 remain at Main DC

vSwitch ` Hypervisor

Bare ACI Main DC Metal

Remote Location Local Traffic forwarding between endpoints

Traffic between Remote Leaf vPC pairs gets forwarded to Spines in ACI main DC Pod.

IP Network

vSwitch Hypervisor ` Bare Metal

ACI Main DC Remote Location

Traffic between Remote leaf switches is directly forwarded

IP Network

vSwitch Hypervisor ` Bare Metal

ACI Main DC Remote Location

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43 Remote Leaf with transport network slicing  No EPG, BD or VRF is stretched across ACI sites.  Communication between the sites will happen via L3out towards WAN  Customers can configure network slicing in WAN

IP Network

L3out L3out

Remote Location-2 Remote Location-1 Central/Regional DC ACI Pod ‘A’ WAN IP MPLS/SR-MPLS/SRv6

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44 Remote Leaf with Multi-site ACI 4.1.2 Remote Location Site1 ACI Main DC vSwitch Site1 Traffic is forwarded Hypervisor through the Spines of the local site ` Bare Metal

IP Network

vSwitch Hypervisor ` Bare Metal

ACI Main DC Remote Location Site2 Site2 #CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 45 Failure handling with Remote Leaf ACI Main DC Pod failure with Multi-Pod Pod ACI 4.2 Remote Location-2 Remote Location-2 Pod2 Pod1  All traffic from RL to other Pods and RL within and across Pod will continue to work. Traffic will flowing through alternate available Pod.

 RL can use alternate Pod’s spine for BGP and COPP connection.

IP Network IP Network IP Network X

ACI Main DC ACI Main DC Remote Location Pod1 Pod2 Pod1 Remote Location-1 All Pods are part of the same ACI Fabric (APIC Domain) Pod2 #CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47 ACI 4.1.2 ACI Main DC Pod failure with Single Pod

Remote Location Pod1  Local traffic within Remote location should continue to work for existing and new EPs

 Existing learnt EP to EP communication should work fine between RL pairs IP Network  Existing learnt remote EP may timeout if there is no communication and this may break communications to those remote EPs

X  New EP information across two RL pairs can’t be synced since Spine is down.

 BGP RR session from RL to Spine will be down, hence RL to RL X communication for external prefixes will be down  Inter-VRF communication will fail since EP of different VRF are ACI Main DC learnt through COOP and COOP connection to Spine is down Remote Location Pod1  Customers who needs full HA for spine failures should use ACI Pod1 Multi-Pod architecture

Since communication was locally happening through local L3out, failure of ACI main DC pod doesn’t impact the traffic between Remote Locations

IP Network

L3out-1 L3out-2 L3out X VRF1 L3out VRF2 VRF Remote Location-2 VRF Remote Location-1 Central/Regional DC ACI Pod ‘A’ WAN IP MPLS/SR-MPLS/SRv6

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51 Components of 5G EPC deployment Control functions are deployed in Central/Regional DC, while Data plane functions are being deployed in Edge DCs

IP Network

UPF Services VNFs- AMF Caching server UPF Services VNFs- NSSF TCP optimizers, CG- SMF TCP optimizers, CG- PCF NAT etc. NAT etc. Edge DC Central/Regional DC WAN IP MPLS/SR-MPLS/SRv6

Interface type Communication Traffic flow N1 UE to AMF Central DC to WAN

N2 RAN to AMF Central DC to WAN

N3 UPF to UE Edge DC to WAN Central DC to WAN(*some UPF might be in Central DC) N4 UPF to SMF Edge DC to WAN Central DC to WAN(*some UPF might be in Central DC) N6 UPF to external Network Edge DC to External Network Central DC to External Network(*some UPF might be in Central DC) N9 UPF to UPF Within Central DC/Edge DC

IP Network

EPG –AMF BD-AMF 10.1.1.1/16

L3out UPF Services VNFs- Caching server UPF TCP optimizers, CG- NAT etc. N2 AMF Edge DC Central/Regional DC WAN

IP MPLS/SR-MPLS/SRv6 N1

UPF is deployed in both Edge and Central DC

IP Network

L3out N3 L3out L3out L3out UE Pool Services VNFs- UE Pool UPF 172.17.1.0/16 Services VNFs- TCP optimizers, CG- 172.16.1.0/16 2001::/64 TCP optimizers, CG- NAT etc. UPF 2000::/64 NAT etc. Edge DC Central/Regional DC WAN IP MPLS/SR-MPLS/SRv6

IP Network

EPG –AMF BD-SMF 10.2.1.1/16

L3out N4 N4 L3out L3out L3out UE Pool Services VNFs- UPF 172.17.1.0/16 UE Pool Services VNFs- TCP optimizers, CG- 2001::/64 172.16.1.0/16 TCP optimizers, CG- NAT etc. UPF 2000::/64 SMF NAT etc. Edge DC Central/Regional DC WAN IP MPLS/SR-MPLS/SRv6

Remote Leaf to Central/Regional DC using IPN N4

IP Network

EPG –AMF BD-SMF 10.2.1.1/16

L3out N4 L3out L3out L3out UE Pool Services VNFs- UPF 172.17.1.0/16 UE Pool Services VNFs- TCP optimizers, CG- 2001::/64 172.16.1.0/16 TCP optimizers, CG- NAT etc. UPF 2000::/64 SMF NAT etc. Edge DC Central/Regional DC WAN IP MPLS/SR-MPLS/SRv6

IP Network

L3out L3out L3out L3out UE Pool UPF Services VNFs- UE Pool 172.17.1.0/16 Services VNFs- TCP optimizers, CG- 172.16.1.0/16 2001::/64 TCP optimizers, CG- NAT etc. UPF 2000::/64 NAT etc. Edge DC Central/Regional DC WAN IP MPLS/SR-MPLS/SRv6 Internet Internet RAN UE #CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58 N9 Interface deployment

IP Network

EPG –UPF EPG –UPF BD-UPF BD-UPF 10.3.1.1/16 10.4.1.1/16

L3out

UPF UPF Group of TCP Deep Packet CG-NAT Subscriber Pool Optimizers Inspection (DPI)

Gi-LAN Internet

Contract UPF EPG Internet EPG

Classified based Service Graph Template Classified based on subscriber on Internet Pool prefixes

10.0.0.0/8 Group of TCP DPI CG-NAT 2000::/8 Optimizers

UPF UPF Group of TCP CG-NAT Deep Packet Subscriber Pool Optimizers Inspection (DPI)

Gi-LAN Internet

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64 Integration with TCP Optimizer Flow 1 TCP Optimizer Integration with ACI Flow 2 Automatic Load-balancing and Symmetry of traffic flow

UPF UPF TCP TCP TCP TCP OPT 1 OPT 2 OPT 3 OPT N Subscriber Pool N-TCP Optimizers in a group Internet  Symmetric PBR ensure return traffic choses same TCP optimize  Automatic load-balancing of traffic across different TCP optimizers based on forwarding table hash (Source IP, Destination IP, Source Port, Destination Port)

Automatic Load-Balancing to remaining ICMP & TCP Tracking of TCP Optimizers after failure Inside & Outside Interface

Inside Outside 1.1.1.1 2.2.2.1 UPF UPF TCP TCP TCP TCP OPT 1 OPT 2 OPT 3 OPT N Subscriber Pool

N-TCP Optimizers in a group Removes whole TCP Optimizer if either Internet Inside or Outside interface goes down

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67 Flow 1 TCP Optimizer Integration with ACI Flow 2 Non-Resilient Hashing All flows get re-hashed on a PBR node failure, this can cause traffic drop for flows that lands on a PBR node that does not have a session information.

TCP OPT2 flows shifted to TCP OPT3

UPF UPF TCP TCP TCP TCP OPT 1 OPT 2 OPT 3 OPT N Subscriber Pool N-TCP Optimizers in a group Internet

All flows going through failed TCP optimizer gets rehashed to one of the TCP optimizer

UPF UPF TCP TCP TCP TCP OPT 1 OPT 2 OPT 3 OPT N Subscriber Pool N-TCP Optimizers in a group Internet

#CLUS BRKSPM-2009 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69 TCP Optimizer Integration with ACI ACI 4.2 Resilient Hashing with standby node Flow 1 Flow 2 Only the flows that were going through failed node gets re-hashed to standby node

UPF UPF TCP TCP TCP TCP OPT OPT 1 OPT 2 OPT 3 Standby Subscriber Pool N-TCP Optimizers in a group Internet

UPF UPF TCP TCP TCP TCP TCP OPT 1 OPT 2 OPT 3 OPT 4 OPT 31 Subscriber Pool N-TCP Optimizers in a group Internet

• Multiple BorderLeaf for high speed internet connectivity • Horizontal scaling of BorderLeaf for expansion • IP handoff to WAN router • No dependency on DC Edge Router

eBGP connection from ACI BorderLeaf to DC Edge Router UPF UPF

IP/MPLS Internet

 Traffic forwarding from Spine

IP/MPLS Internet  Supported with ASR9000, Nexus7000 and ASR1000  Optional automatic VRF creation Service-chaining VRF ASR9K Subscriber VRF on DC Edge Router IMS VRF Internet VRF  Dependency on DC Edge Router

VXLAN EVPN

Subscriber VRF Subscriber VRF Internet VRF Internet VRF IMS VRF Service-chaining VRF

TCP TCP Optimizer-1 Optimizer-N

Network service Orchestrator (NSO)

• ACI to SR Interworking at ACI border leaf

• Remote Leaf can be border leaf and support same design

• ACI to SR MPLS Interworking with N9300-FX2

• APIC/MSC to WAN controller integration L3out SR EVPN

PE PE

WAN (SR/SRv6)

No communication is allowed without contract Default action is deny, only with explicit permit condition packets are forwarded

Source EPG Destination EPG Permit Contract Classification based on Classification based on VLAN or prefix VLAN or prefix

Deny

Packet drop

Filter traffic based on 5-tuples (source IP, destination IP, protocol, source port, destination port) filter in ACI

SPAN/ERSPAN from ACI towards Nexus Data Broker (NDB) switches • Advance ingress packet filtering and modification Analytics and • Load-balancing in egress Monitoring Tools

Nexus Data

Broker

Topology Dashboard Troubleshooting Wizard End Point Tracker

Faults Capacity DashBoard Link Statistics

Health Score Card Traffic Map Upgrade/Downgrade

 Easy to SPAN packets to APIC without dedicated server  Requires in band connectivity  Packet policer to limit the number of packets sent to APIC  10 PCAP files of 1MB each to limit the usage of APIC

Decommission the old switch by removing the controller Register the new switch by provide same Node ID and Node Name

Old Leaf New Leaf

Resource Analytics

ACI Network Event Analytics Insights Resources

Flow Analytics

Deep Insights Into Network Health (Control Plane, Data Plane, Capacity, Utilization and Environmental Health)

Flow Anomalies Flow Analytics Dashboard

Packet Drops

Latency

End Point Move

Flow Analytics Dashboard Displays Key Indicators Of Infrastructure Data Plane Health.

Software/Hardware Recommendations Avoid multiple TAC calls Workarounds

EOL/EOS Keep Network up to date Field Notices Adhere to Cisco policies SMUs Recommendations

Network Known Issues/PSIRTs Remove Complexity Anomalies Unknown runtime Avoid Outages Insights Config anomalies Faster Deployment times Advisor Version Scale Limits/Hardening Significant CAPEX Check And OPEX Savings Configuration

Forwarding State Check Prevent traffic black holing Loops Detection Cable Checkers Avoid downtimes

Connectivity Service: • vEPC (SAEGW, MME) Connectivity Service: • HSS/HLR, PCRF, OCS • User Plane • IMS (incl. SMS, MMS, Voicemail) • Caching, CDN • Gi-LANv • Gi-LAN • Services

Edge Cloud: Central Telco Cloud 8 Locations with Remote 15+ fabric ACI Fabric Pre-Agg Leaf

Pre-Agg Access Ring

Pre-Agg DC Fabric IP Core IP Core Pre-Agg. Agg. Ring Ring Pre-Agg IP Core IP Core

Access Pre-Agg Ring

Pre-Agg

6.5 M sub, vEPC, OSS/BSS, IMS, NDB* 218 Leaf Switches

6 ACI fabric’s, 13 CVIM Pod’s per fabric. Small Cell Pod x 2 TBD

EPC Pod 20 Racks for Border IGW Router (Cisco) x 2 each Leaf NCS5508 Spine Switch IMS Pod 23 Racks for Internet (Nokia) x 2 each

BareMetal Pod (Nokia) x 1 4 Racks

MBH Router BSS Pod NCS5508 NetCracker (NEC) x 1 TBD

MBH BareMetal Pod InnoEye (NEC) x 1 TBD Border Mgmt Pod Leaf (Shared) x 2 TBD 800G (100G x 8) 100G DMZ Pod VIM (Shared) x 1 TBD Group of Racks ACI Fabric

Agility Policy Intent at Scale High performance resilient 20+ ACI fabrics deployed for vEPC SDN fabric for NFVi project 2000+ ACI nodes ACI Service Chain for their Gi-LAN services 400Tbps data capacity trending to 1.2Pbps

Visibility & Control Time to value Operations benefit from hardware Earlier 1 DC Fabric in 3 telemetry and visibility Weeks; Now 3 DC Fabric in 1 Software upgrade now completed week with ACI in couple of days without impact to 40+ New Telco DC deployed in services less than 3 months

• 5G deployments will create multiple highly distributed DC’s.

• Automation, Lifecycle Management, and Analytics will be key aspects.

• With ACI we do have an option to meet some of these challenges.

• Simplifying Service Enablement in Telco Data Centers using Cisco ACI https://youtu.be/KxebKVlKe60

• Demo: Multi-node Service Chaining in Telecom Data Centers Made Simple https://youtu.be/VeWbOl31UIM

• Demo: How to Simplify the Expansion of Services in Telco Data Centers https://youtu.be/9fhmEC9at9g

• ACI in Telco DC https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric- infrastructure/white-paper-c11-740717.html

Questions? Use Cisco Webex Teams to chat with the speaker after the session How 1 Find this session in the Cisco Live Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space

Webex Teams will be moderated cs.co/ciscolivebot#BRKSPM-2009 by the speaker until June 16, 2019.

#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97 Complete your online session • Please complete your session survey after each session. Your feedback evaluation is very important.

• Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live water bottle.

• All surveys can be taken in the Cisco Live Mobile App or by logging in to the Session Catalog on ciscolive.cisco.com/us.

Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com.

Demos in the Walk-in labs Cisco campus

Meet the engineer Related sessions 1:1 meetings

#CLUS