#CLUS VXLAN BGP EVPN based Multi-Site Extended
Max Ardica – Principal Engineer Lukas Krattiger – Principal Engineer BRKDCN-2035
#CLUS Who Are the Presenters?
Max Ardica Lukas Krattiger Principal Engineer – INSBU Principal Engineer – INSBU @maxardica @ccie21921
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 Session Objectives
At the end of the session, the participants should be able to:
. Articulate the value proposition of the new VXLAN Multi-Site architecture and list several use cases for which it should be positioned
. Understand the functionalities and specific design considerations associated to VXLAN Multi-Site Initial assumption:
. The audience already has a good knowledge of the VXLAN EVPN technology and its use to deploy modern Data Center Fabrics
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4 Agenda
. Data Center Interconnect (DCI) Evolution . VXLAN Multi-Site Introduction • Functional Components and Use Cases • HW/SW Support and Scalability Values • Supported Topologies . VXLAN Multi-Site Deep Dive • Border Gateway Deployment Considerations • Inter-Site BUM Traffic Handling • Control and Data Planes • Connectivity to the External Layer 3 Domain • Legacy Site Integration • Configuration Specifics (for your reference) . Conclusions
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Cisco Webex Teams
Questions? Use Cisco Webex Teams to chat with the speaker after the session How 1 Find this session in the Cisco Live Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space
Webex Teams will be moderated cs.co/ciscolivebot#BRKDCN-2035 by the speaker until June 16, 2019.
#CLUS © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 Data Center Interconnect (DCI) Evolution Single Fabric CLOS Topology
S S S S
L L L L . . . . L
Single Logical Data Center
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 Multiple Logical Data Centers
S S S S S S S S
L L L L . . . . L L L L L . . . . L
Multiple Logical Data Centers
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 Overlays Evolve/Spread
DC Local Overlay
End-to-End Overlay SS SS SS SS
S S S S S S S S
L L L L . . . . L L L L L . . . . L
Single Logical Data Center
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 The Ugly Truth Overlays Evolve/Spread
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 Tunnel adjacencies The Ugly Truth N * (N-1) What about the Required VXLAN Tunnel Adjacencies? 2
6 VTEP per DC 2 x 6 VTEP across Logical DCs 4 Leaf (VTEP) per Logical DC 2 Border Leaf (BL) per DC 2 Logical DCs 12 VTEP across DCs 12x(12-1)/2 66 adjacencies
204 VTEP per DC 10 x 204 VTEP across Logical DCs 200 Leaf (VTEP) per Logical DC 4 Border Leaf (BL) per DC 10 Logical DCs 2040 VTEP across DCs 2040x(2040-1)/2 ~2 millions adjacencies
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 Changing the Paradigm with Hierarchical Overlays
DC Local Overlay
Multi-Site Overlay SS SS SS SS
S S S S S S S S
L L L L . . . . L L L L L . . . . L
Multiple Logical Data Center
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 VXLAN Multi-Site Introduction Functional Components and Use Cases VXLAN Multi-Site https://tools.ietf.org/html/draft-sharma-multi-site-evpn Functional Components
Site-External DCI (IP Routing and Increased Border Gateways MTU Support) (Key Functional Components of VXLAN Multi-Site Architecture)
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Spine Spine Spine Spine Spine Spine Spine
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site-Internal Fabric Site 1 (Common VXLAN and Site n BGP-EVPN Functions)
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 VXLAN Multi-Site Characteristics
. Multiple Overlay Domains – Interconnected & Controlled
. Multiple Overlay Control-Plane Domains – Interconnected & Controlled
. Multiple Underlay Domains - Isolated
. Multiple Replication Domains for BUM – Interconnected & Controlled
. Multiple VNI Administrative Domains – Phase 2
Underlay Isolation – Overlay Hierarchies
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 VXLAN Multi-Site Main Use Cases
Scale-Up Model to Build a Large Intra-DC Network
Data Center Interconnect (DCI)
Integration with Legacy Networks (Coexistence and/or Migration)
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 VXLAN Multi-Site Introducing the Border Gateway
Overlay Multi-Site
Multi-Site VIP Multi-Site VIP 10.1.1.111 10.2.2.222 VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine OverlaySpine SiteSpine 1 Spine Spine OverlaySpine SiteSpine n Spine
Any VTEP
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site 1 Site n
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 VXLAN Multi-Site Underlay Isolation
Multi-Site VIP Multi-Site VIP 10.1.1.111 No Underlay Extension 10.2.2.222 VTEP VTEP VTEP VTEP
BGW BGW BGW BGW Border (PIP) Border (PIP) Border (PIP) Border (PIP) 10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine Site 1 Underlay Site n Underlay Routing Table Routing Table Border: Leaf: Border: Leaf: VTEP 10.1.1.101 10.1.1.1 10.2.2.101 10.2.2.1VTEP 10.1.1.1 10.2.2.7 VTEP VTEP VTEP VTEP VTEP VTEP10.1.1.102VTEP 10.1.1.2 VTEP VTEP VTEP VTEP VTEP10.2.2.102VTEP 10.2.2.2VTEP 10.1.1.111 10.1.1.3 10.2.2.222 10.2.2.3 10.1.1.4 10.2.2.4 10.1.1.5 10.2.2.5 Site 1 10.1.1.6 Site n 10.2.2.6 10.1.1.7 10.2.2.7
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 Inter-Site Network VXLAN Multi-Site Routing Table Site-External DCI Border Site1: Border Site2: 10.1.1.101 10.2.2.101 10.1.1.102 10.2.2.102 10.1.1.111 10.2.2.222
Multi-Site VIP Multi-Site VIP 10.1.1.111 Site-External DCI 10.2.2.222 VTEP VTEP VTEP VTEP
BGW BGW BGW BGW Border (PIP) Border (PIP) Border (PIP) Border (PIP) 10.1.1.101 10.1.1.102 10.2.2.101 10.2.2.102
Spine Spine Spine Spine Spine Spine Spine Spine
VTEP VTEP 10.1.1.1 10.2.2.7 VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site 1 Site n
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 Multi-Site – VXLAN Tunnel Adjacencies BG102# show nve peers Interface Peer-IP VNI Up Time ------nve1 Overlay10.1.1.1 Multi 30000-Site 00:12:16 nve1 10.1.1.4 30000 03:18:06 nve1 10.2.2.222 30000 00:12:23
Multi-Site VIP Multi-Site VIP 10.1.1.111 10.2.2.222 VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine OverlaySpine SiteSpine 1 Spine Spine OverlaySpine SiteSpine n Spine
VTEP VTEP VTEP 10.1.1.1 10.1.1.4 10.2.2.7 VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Leaf1-1# show nve peers Leaf2-7# show nve peers Interface Peer-IP VNI Up Time Interface Peer-IP VNI Up Time ------Site 1 ------Site n------nve1 10.1.1.4 30000 03:18:06 nve1 10.2.2.222 30000 00:12:25 nve1 10.1.1.111 30000 00:12:23 #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 Tunnel adjacencies The Multi-Site Truth N * (N-1) What about the Required VXLAN Tunnel Adjacencies? 2
6 VTEP per Site 2 x 6 VTEP across Sites 4 Leaf (VTEP) per Site 2 Border Leaf (BL) per Site 2 Logical Sites 12 VTEP across Sites 6x(6-1)/2 15 adjacencies per Site
204 VTEP per Site 10 x 204 VTEP across Sites 200 Leaf (VTEP) per Site 4 Border Leaf (BL) per Site 10 Logical Sites 2040 VTEP across Sites 204x(204-1)/2 ~20K adjacencies per site
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 Tunnel adjacencies The Ugly Truth N * (N-1) What about the Required VXLAN Tunnel Adjacencies? 2
6 VTEP per DC 2 x 6 VTEP across Logical DCs 4 Leaf (VTEP) per Logical DC 2 Border Leaf (BL) per DC 2 Logical DCs 12 VTEP across DCs 12x(12-1)/2 66 adjacencies
204 VTEP per DC 10 x 204 VTEP across Logical DCs 200 Leaf (VTEP) per Logical DC 4 Border Leaf (BL) per DC 10 Logical DCs 2040 VTEP across DCs 2040x(2040-1)/2 ~2 millions adjacencies
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 HW/SW Support and Scalability Values VXLAN Multi-Site HW/SW Support
. Minimum hardware and software requirements for Border Gateways Item Requirement
• Cisco Nexus 9300 EX platform • Cisco Nexus 9300 FX platform • Cisco Nexus 9300 FX2 platform Cisco Nexus Hardware • Cisco Nexus 9364C platform • Cisco Nexus 9332C platform • Cisco Nexus 9500 platform with X9700-EX line card • Cisco Nexus 9500 platform with X9700-FX line card
Cisco Nexus Software Cisco NX-OS Software Release 7.0(3)I7(1) or later
. The hardware and software requirements for the Site-Internal nodes of a VXLAN BGP EVPN site remain the same as those without the EVPN Multi-Site BGW
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 VXLAN Multi-Site Scalability Values as of 9.2(3) Release Multi-Site Scale Number of Sites 10 Number of BGWs per Site 4 (Anycast) or 2 (vPC) VTEP per Site 256
Border Gateway (BGW) Scale EX/FX/FX2 N9364C/N9332C Number of Layer-2 VNI (VLAN) 2,000 Number of Layer-3 VNI (VRF) 1,000 MAC per BGW 90,000 64,000 IPv4 Host Routes per BGW* ~530,000 ~60,000 IPv4 Network Routes per BGW* ~530,000 ~8,000 IPv6 Host Routes per BGW* ~24,000 ~7,000 IPv6 Network Routes per BGW* ~260,000 ~2,000
*The values provided in these tables focus on the scalability of one particular Route scale at a time
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 Supported Topologies BGW-to-Cloud
Layer-3 Network
BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW Site 1 Site 2 Site n
Spine Spine Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 29 BGW-to-Cloud
• Traditional DC Model • Access = Leaf • Distribution = Spine • Aggregation = BGW • Core = Cloud • Fits well for DataCenter Interconnect (DCI) use-cases • Cloud is undiscovered Territory • Requirements • MTU >1500 (>1550) • IP Reachability between local and remote BGW
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30 BGWs between Spine and Super-Spine
Super-Spine Super-Spine
BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW Site 1 Site 2 Site n
Spine Spine Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31 BGWs between Spine and Super-Spine
• Specifying the ”Cloud” from BGW-to-Cloud • Discovered Territory
• Multi-Tier Clos (3-4 Tier) • Leaf - Spine – BGW – Super-Spine
• Fits well for Scale and Compartmentalization use-cases • Uniform Connectivity between BGW via Super-Spine (2-way, 4-way, 8-way etc.)
• Requirements • MTU >1500 (>1550) • IP Reachability between local and remote BGW
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 32 BGWs on Spine
Super-Spine Super-Spine
BGW BGW BGW BGW BGW BGW BGW BGW BGW BGW Spine Spine Spine Spine Spine Spine Spine Spine Spine Spine Site 1 Site 2 Site n
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33 BGWs on Spine
• Integrating the BGW into the Spine
• Multi-Tier Clos (3 Tier) • Leaf – (Spine/BGW) – Super-Spine • Remember • Overloads function (complexity) • Impact on failure events (Spine failure = Multi-Site failue) • Scale (VTEP has a different profile vs. a Spine) • Requirements • MTU >1500 (>1550) • IP Reachability between local and remote BGW
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 34 BGWs Back-to-Back
. Recommended to limit the back-to-back deployment to two sites 2 Site topology can be fully automated using DCNM Recommended to insert Layer-3 Core network with 3+ sites
BGW BGW BGW BGW Site 1 Site 2
Spine Spine Spine Spine
Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf Leaf
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 35 BGWs Back-to-Back
• ”I don’t wanna spend money”
• Minimum Topology is a ”square” • Increase convergence/resiliency with square + cross (aka full-mesh BGW). • Fine for two Sites – Complicated for more than two Sites
• Requirements • Must have link between local BGWs • MTU >1500 (>1550) • IP Reachability between local and remote BGW
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 36 VXLAN Multi-Site Deep Dive Border Gateway Deployment Considerations VXLAN Multi-Site Border Gateways Deployment Considerations
. Border Gateways used for two main functions: Anycast Border Gateways
BGW BGW BGW BGW • Interconnecting each site to the Inter-Site network (for VTEP VTEP VTEP VTEP East-West traffic flows) • Connecting each site to the external Layer 3 domain (for North-South traffic flows) • May also be used to connect endpoints and/or network service nodes (FWs, ADCs) Site 1
. Possible deployment models: vPC Border Gateways • Anycast Border Gateways vPC BGW BGW
VTEP VTEP • vPC Border Gateways . BGW function enablement in the VXLAN EVPN fabric: • BGWs as leaf nodes • BGWs as spine nodes (Border-Spines) Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39 Anycast Border Gateways VXLAN Multi-Site Anycast Border Gateway (1)
Anycast Border Gateway . Up to 4 Border Gateways . Border Gateway
BGW BGW BGW BGW • Deploying at Leaf – 7.0(3)I7(1) VTEP VTEP VTEP VTEP • Deploying at Spine – 7.0(3)I7(2)
Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41 VXLAN Multi-Site Anycast Border Gateway (2)
Anycast Border Gateway . Common Multi-Site Virtual IP (Multi-Site VIP) Multi-Site VIP 10.1.1.111 across BGWs • Multi-Site VIP for communication between BGW BGW BGW BGW VTEP VTEP VTEP VTEP the Border Gateways in different Sites PIP-BGW1 PIP-BGW2 PIP-BGW3 PIP-BGW4 10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104 • Multi-Site VIP for communication between Border Gateways and Leaf nodes within a Multi-Site VIP 10.1.1.111 Site . Individual Primary IP (PIP) per BGW • Used for Broadcast, Unknown Unicast and Multicast (BUM) replication • PIP for communication with Single-Homed endpoints (routed only), intra- and inter- Site Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42 VXLAN Multi-Site Anycast Border Gateway (3)
Anycast Border Gateway Type: 00 System MAC: 00:00:00:00:00:01 IP: 10.1.1.101 4 Ethernet Segment: 00:00:07 VNI: 30010 . Per-VNI Designated Forwarder (DF) election • Each BGW can serve as DF for a single or a set of Layer-2 VNIs BGW BGW BGW BGW VTEP VTEP VTEP VTEP • DF election and assignment is automatic
DF DF DF DF 30010 30011 30012 30099 . Using BGP EVPN Route Type 4 for DF election • Operator Managed Assignment (Type: 00) BGP EVPN • Six Octet Site Identifier (System MAC: 00:00:00:00:00:01) RR RR Spine Spine • Multi-Site Discriminator (Ethernet- Segment: 00:00:07) • Originators IP Address (PIP): 10.1.1.101 • Layer-2 VNI: 30010 Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43 VXLAN Multi-Site Anycast Border Gateway (4)
External Connectivity Anycast Border Gateway Point-to-Point L3 Links (Physical/Sub-Interfaces) . Single-Homed End-Points only connected with L3 links • Services Appliance (i.e. Firewall, ADC etc.) BGW BGW BGW BGW VTEP VTEP VTEP VTEP • External routers PIP-BGW1 PIP-BGW2 PIP-BGW3 PIP-BGW4 10.1.1.101 10.1.1.102 10.1.1.103 10.1.1.104 • No SVI support on BGW nodes .1 .1 . Advertised and Reachable through Individual
Point-to-Point L3 Links Primary IP Address (PIP) Point-to-Point L3 Links (Physical/Sub-Interfaces) ADC ADC • Intra-Site: Leaf nodes use PIP to reach the device
ADC ADC connected to Border Gateways 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102 • Inter-Site: Remote Border Gateways use PIP to reach the device connected to Border Gateways
VTEP
Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44 vPC Border Gateways NXOS Release 9.2(1) NXOS Release 9.2(1) Anycast BGW vs. vPC Border Gateway
Anycast Border Gateway vPC Border Gateway
• Up to 4 BGW • 2 BGW with physical vPC Peer-Link
• Shared Nothing • Small Deployments • Simple Failure Scenarios • End-Point or Network Services • Any Deployments Connectivity on BGW • No End-Point or Network Services • Migration Use-Cases (Brownfield) Connectivity on BGW • Classic Ethernet/FabricPath to • Greenfield Deployments VXLAN EVPN
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46 NXOS Release 9.2(1) Multi-Site Border Gateway – Anycast vs. vPC
• Both Anycast and vPC Border Gateway needs to be configured with a common Multi-Site VIP address and an individual Primary IP (PIP) address • vPC Border Gateways share a secondary IP address to be used as vPC virtual IP (vPC VIP)
Anycast BGW vPC BGW vPC VIP 11.11.11.11 Multi-Site VIP Multi-Site VIP 100.100.100.100 100.100.100.100
PIP1 PIP1 PIP1 PIP1 10.1.1.1 10.1.2.1 10.1.1.1 10.1.2.1 vPC VTEP VTEP VTEP VTEP Fabric BGW1 …. BGW4 Fabric BGW1 BGW2
Spine Spine Spine Spine
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 47 NXOS Release VXLAN Multi-Site 9.2(1) vPC Border Gateway and Transit Traffic
vPC Border Gateway . Common Multi-Site Virtual IP (Multi-Site VIP) Multi-Site VIP 10.1.1.111 vPC across BGWs • Multi-Site VIP for Inter-Site transit communication BGW BGW (transit) VTEP VTEP PIP-BGW1 PIP-BGW2 10.1.1.101 vPC VIP 10.1.1.102 • Common vPC Virtual IP (vPC VIP) across BGWs 10.1.1.121 • Used by default for communication with external Multi-Site VIP networks 10.1.1.111 • Used for Broadcast, Unknown Unicast and Multicast (BUM) replication . Individual Primary IP (PIP) per BGW • Used for communication with external networks when “advertised-pip” is configured
Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 48 NXOS Release VXLAN Multi-Site 9.2(1) vPC Border Gateway and Locally Attached End-Points
vPC Border Gateway . Single- or Dual-Homed End-Points Multi-Site VIP 10.1.1.111 vPC • Services Appliance (i.e. Firewall, ADC etc.)
BGW BGW • Physical or Virtual Servers VTEP VTEP • Anycast Gateway function offered to the Anycast Anycast vPC VIP Gateway Gateway endpoints 10.1.1.121 . Advertised and Reachable through vPC Virtual Multi-Site VIP 10.1.1.111 IP Address (vPC VIP)
ADC Baremetal • Intra-Site: Leaf nodes use vPC VIP to reach End- Points connected to Border Gateways ADC EP 0000.3010.1102 0000.3010.1101 • Inter-Site: Remote Border Gateways use vPC VIP 192.168.10.102 192.168.10.101 to reach End-Points connected to Border Gateways • Traffic potentially traverses vPC Peer-Link Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 49 NXOS Release VXLAN Multi-Site 9.2(1) vPC Border Gateway and Designated BUM Forwarder
vPC Border Gateway . vPC-based Designated Forwarder Election
vPC . Per-Site Designated Forwarder (DF) election
BGW BGW • Using same approach as in vPC VTEP VTEP • Best Path to Rendezvous-Point or vPC Primary DF vPC VIP 10.1.1.121 Node • Same vPC node is elected DF for all the Layer-2 VNIs
Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50 NXOS Release vPC Border Gateways 9.2(1) Traffic between Locally Connected Endpoints across Sites
Src Dst VXLAN vPC VIP1 vPC VIP2 Original Packet Header
Inter-Site Network
vPC VIP1 vPC VIP2 vPC 11.11.11.11 vPC 22.22.22.22
VTEP VTEP VTEP VTEP
BGW1 BGW2 BGW1 BGW2
Site 1 Site 2
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 51 NXOS Release vPC Border Gateways 9.2(1) Traffic between Locally Connected Endpoints and Remote L3Out
Src Dst VXLAN vPC VIP1 vPC VIP2 Original Packet Header
Inter-Site Network
vPC VIP1 vPC VIP2 vPC 11.11.11.11 vPC 22.22.22.22
VTEP VTEP VTEP VTEP
BGW1 BGW2 BGW1 BGW2
L3
Site 1 Site 2
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 52 NXOS Release vPC Border Gateways 9.2(1) Traffic between Site Connected Endpoints across Sites
Src Dst Multi-Site Multi-Site VXLAN Original Packet VIP1 VIP2 Header
Inter-Site Network
Multi-Site VIP1 Multi-Site VIP2 vPC 100.100.100.100 vPC 200.200.200.200
VTEP VTEP VTEP VTEP
BGW1 BGW2 BGW1 BGW2
VTEP VTEP
Site 1 Site 2
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53 NXOS Release vPC Border Gateways 9.2(1) BUM Traffic across Sites
Src Dst VXLAN vPC VIP1 vPC VIP2 Original Packet Header
Inter-Site Network
vPC VIP1 vPC VIP2 11.11.11.11 vPC 22.22.22.22 vPC DF DF VTEP VTEP VTEP VTEP
BGW1 BGW2 BGW1 BGW2
BUM Traffic redirected via vPC peer-link toward the DF
VTEP VTEP
Site 1 Site 2
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 54 Inter-Site BUM Traffic Handling VXLAN Multi-Site BUM Traffic Forwarding
Overlay Multi-Site
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine OverlaySpine SiteSpine 1 Spine Spine OverlaySpine SiteSpine n Spine
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP BUM
Site 1 Site n Baremetal
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 56 VXLAN Multi-Site BUM Replication Modes (Multicast Intra-Site)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine OverlaySpine SiteSpine 1 Spine Spine OverlaySpine SiteSpine n Spine
Multicast Multicast
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site 1 Site n
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 57 VXLAN Multi-Site BUM Replication Modes (Ingress Replication Only)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine OverlaySpine SiteSpine 1 Spine Spine OverlaySpine SiteSpine n Spine
Ingress Replication Ingress Replication
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site 1 Site n
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58 VXLAN Multi-Site BUM Replication Modes (Mixed Mode Intra-Site)
Overlay Multi-Site
Ingress Replication
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine OverlaySpine SiteSpine 1 Spine Spine OverlaySpine SiteSpine n Spine
Ingress Replication Multicast
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Site 1 Site n
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 59 VXLAN Multi-Site BUM Traffic Policing
Overlay Multi-Site
Storm Control VTEP VTEP Broadcast 0-100% VTEP VTEP BGW BGW Unknown Unicast 0-100% BGW BGW Multicast 0-100%
Spine OverlaySpine SiteSpine 1 Spine Spine OverlaySpine SiteSpine n Spine
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP BUM
Site 1 Site n Baremetal
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 60 Control and Data Planes Multi-Site Control Plane VXLAN Multi-Site Control Plane Deployment Considerations . MP-eBGP EVPN only inter-Sites • Next-hop behavior (VXLAN tunnel termination and reorigination) and loop protection (as-path attribute)
. Two main options for underlay and overlay control plane deployment 1. I-E-I (Recommended) • Intra-Site: IGP (OSPF, IS-IS) as underlay CP, iBGP as overlay CP • Inter-Sites: eBGP for both underlay and overlay CPs 2. E-E-E* • Intra-Site and Inter-Sites: eBGP for both underlay and overlay CPs
. Full mesh of MP-eBGP EVPN adjacencies across sites • Recommended to deploy a couple of Route-Servers with 3 or more sites • RS in a separate AS only perform control plane functions (“eBGP Route-Reflectors”, IETF RFC 7947) • RS functions: EVPN routes reflection, next-hop-unchanged, route-target rewrite
*For more information on why eBGP for both underlay and overlay CP is not a good idea: https://learningnetwork.cisco.com/blogs/community_cafe/2017/10/17/the-magic-of-super-spines-and-rfc7938-with-overlays-guest-post
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 63 VXLAN Multi-Site Underlay Control Plane
DC Core (Layer-3 Unicast)
DCI
VTEP VTEP VTEP VTEP Fabric BGW …. BGW BGW …. BGW VXLAN EVPN VXLAN EVPN Spine Spine Spine Spine
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64 VXLAN Multi-Site Overlay Control Plane (L3 Core)
Route Server (eBGP ”Route Reflector”) RS
DC Core (Layer-3 Unicast)
DCI
VTEP VTEP VTEP VTEP Fabric BGW …. BGW BGW …. BGW
VXLANiBGP-EVPN EVPN VXLANiBGP-EVPN EVPN Spine Spine Spine Spine RR RR Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 65 VXLAN Multi-Site Overlay Control Plane (L3 Core, no RS)
eBGP-EVPN DC Core (Layer-3 Unicast)
DCI
VTEP VTEP VTEP VTEP Fabric BGW …. BGW BGW …. BGW
VXLANiBGP-EVPN EVPN VXLANiBGP-EVPN EVPN Spine Spine Spine Spine RR RR Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 66 VXLAN Multi-Site Overlay Control Plane
RS
L3VNI: 50001 L3VNI: 50001 Route-Target: DC Core Route-Target: 65502:50001 65501:50001 (Layer-3 Unicast)
VRF VRF Tenant1 Tenant1 DCI
VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 Fabric BGW …. BGW BGW …. BGW
VXLANiBGP-EVPN EVPN VXLANiBGP-EVPN EVPN Spine Spine Spine Spine RR RR Site1 Site2 L2VNI: 30010(VLAN 10) L2VNI: 30020 (VLAN 20) L2VNI: 30010 (VLAN 10) L3VNI: 50001(Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Host1 Host2 Host3 0000.3010.1101 0000.3020.2101 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 67 VXLAN Multi-Site Overlay Control Plane (Site 1)
RS
L3VNI: 50001 L3VNI: 50001 Route-Target: DC Core Route-Target: 65502:50001 65501:50001 (Layer-3 Unicast) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. VRF 2 0000.3010.1101/48 30010, 65501:30010 192.168.10.101/32 50001, 65501:50001VRF 10.1.1.1 Tenant1 2 0000.3020.2101/48 30020, 65501:30020 192.168.20.101/32 50001, 65501:50001Tenant1 10.1.1.111 DCI 2 0000.3010.1102/48 30010, 65501:30010 192.168.10.102/32 50001, 65501:50001 10.1.1.111 VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 Fabric BGW …. BGW BGW …. BGW VXLAN EVPN VXLAN EVPN Spine Spine Spine Spine RR RR Site1 Site2 L2VNI: 30010(VLAN 10) L2VNI: 30020 (VLAN 20) L2VNI: 30010 (VLAN 10) L3VNI: 50001(Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Host1 Host2 Host3 0000.3010.1101 0000.3020.2101 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 68 VXLAN Multi-Site Overlay Control Plane (Site 2)
RS
L3VNI: 50001 L3VNI: 50001 Route-Target: DC Core Route-Target: 65502:50001 65502:50001 (Layer-3 Unicast) Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq. 2 0000.3010.1101/48VRF 30010, 65502:30010 192.168.10.101/32 50001, 65502:50001 10.2.2.222 VRF 2 0000.3020.2101/48Tenant1 30020, 65502:30020 192.168.20.101/32 50001, 65502:50001 10.2.2.1 Tenant1 DCI 2 0000.3010.1102/48 30010, 65502:30010 192.168.10.102/32 50001, 65502:50001 10.2.2.3 VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 Fabric BGW …. BGW BGW …. BGW VXLAN EVPN VXLAN EVPN Spine Spine Spine Spine RR RR Site1 Site2 L2VNI: 30010(VLAN 10) L2VNI: 30020 (VLAN 20) L2VNI: 30010 (VLAN 10) L3VNI: 50001(Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Host1 Host2 Host3 0000.3010.1101 0000.3020.2101 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 69 VXLAN Multi-Site Overlay Control Plane (DCI)
RS
L3VNI: 50001 L3VNI: 50001 Route-Target: DC Core Route-Target: 65502:50001 65501:50001 (Layer-3 Unicast)
VRF VRF Tenant1 Tenant1 DCI
VTEP VIP1 VTEP VTEP VIP2 VTEP Type 10.1.1.111MAC / Length L2VNI / RT IP / Length L3VNI / RT Next10.2.2.222-Hop Seq. BGW BGW BGW BGW Fabric 2 …0000.3010.1101/48. 30010, 65599:30010 192.168.10.101/32 50001, 65599:50001 10.1.1.111…. 2 0000.3020.2101/48 30020, 65599:30020 192.168.20.101/32 50001, 65599:50001 10.2.2.222
VXLAN2 0000.3010.1102/48 EVPN 30010, 65599:30010 192.168.10.102/32 50001, 65599:50001VXLAN10.2.2.222 EVPN Spine Spine Spine Spine RR RR Site1 Site2 L2VNI: 30010(VLAN 10) L2VNI: 30020 (VLAN 20) L2VNI: 30010 (VLAN 10) L3VNI: 50001(Tenant1) L3VNI: 50001 (Tenant1) L3VNI: 50001 (Tenant1) VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Host1 Host2 Host3 0000.3010.1101 0000.3020.2101 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 70 Multi-Site – Selective Advertisements
. The Multi-Site architecture provides granular control on how Layer-2 and Layer-3 communication is extended across sites
. Layer-2 and/or Layer-3 VNIs configured on the Border Gateways (BGW) control the Control-Plane advertisement towards DCI
. Enhances the overall scalability of the solution • Scale up the total number of End-Points supported across sites
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 71 Multi-Site Data Plane VXLAN Multi-Site Overlay Data Plane
Inter-site VXLAN Data Plane
De-capsulation, policy application (i.e. BUM DC Core rate limiting) and re-encapsulation on (Layer-3 Unicast) De-capsulation and BGW (L2 or L3 lookup) Re-encapsulation on BGW (L2 or L3 lookup)
DCI
VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 Fabric BGW …. BGW BGW …. BGW VXLAN EVPN VXLAN EVPN Spine Spine Spine Spine Intra-site VXLAN Site1Data Plane Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Host1 Host2 Host3 0000.3010.1101 0000.3020.2101 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 73 Multi-Site Packet Walk (BUM)
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74 VXLAN Multi-Site Packet Walk Layer 2 (BUM) – Site 1 BUM Forwarding
SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF Traffic is replicated VTEP 30010 VTEP
intra-Site BGW11 BGW21 2 VTEP VTEP VXLAN EVPN VXLAN EVPN VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
DF VTEP 30010 VTEP 1 Host 1 sends a BGW12 BGW22 L2 BUM frame
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75 VXLAN Multi-Site Packet Walk Layer 2 (DF and Split Horizon) – Site 1 BUM Forwarding
SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF VTEP 30010 VTEP
BGW11 BGW21
VTEP BUM Forward VTEP VXLAN EVPN VXLAN EVPN VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
DF VTEP 30010 VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102 192.168.10.101 192.168.10.102
Drop due to Split-Horizon rule #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 76 VXLAN Multi-Site Packet Walk Layer 2 (BUM) – DCI BUM Forwarding
SIP DIP VXLAN SMAC DMAC SIP DIP
BGW11- BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 PIP Payload
BGW11- BGW22 30010 H1-MAC ALL-F H1-IP ALL-255 PIP
DF VTEP 30010 BGW11 replicates traffic VTEP inter-Sites toward BGW BGW11 BGW21 nodes 3 VTEP BUM Forward VTEP VXLAN EVPN VXLAN EVPN VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
DF VTEP 30010 VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 77 VXLAN Multi-Site Packet Walk Layer 2 (DF and Split Horizon) – DCI BUM Forwarding
SIP DIP VXLAN SMAC DMAC SIP DIP
BGW11- BGW21 30010 H1-MAC ALL-F H1-IP ALL-255 PIP Payload
BGW11- BGW22 30010 H1-MAC ALL-F H1-IP ALL-255 PIP
DF VTEP 30010 VTEP
BGW11 BGW21
VTEP BUM Forward VTEP VXLAN EVPN VXLAN EVPN VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
DF VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102 192.168.10.101 192.168.10.102
Drop due to Split-Horizon rule #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78 VXLAN Multi-Site Packet Walk Layer 2 (BUM) – Site 2 BUM Forwarding
SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW22-PIP DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF VTEP 30010 VTEP Traffic is replicated BGW11 BGW21 intra-Site
VTEP 4 VTEP VXLAN EVPN VXLAN EVPN VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
DF VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79 VXLAN Multi-Site Packet Walk Layer 2 (DF and Split Horizon) – Site 2 BUM Forwarding
SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW22-PIP DGROUP 30010 H1-MAC ALL-F H1-IP ALL-255
DF VTEP 30010 VTEP
BGW11 BGW21
VTEP VTEP VXLAN EVPN VXLAN EVPN VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
DF VTEP 30010 VTEP
BGW12 BGW22
BUM Forward
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 Drop due to Designated Forwarder (DF) rule 0000.3010.1102 192.168.10.101 192.168.10.102
Drop due to Split-Horizon rule #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 80 VXLAN Multi-Site Packet Walk Layer 2 (BUM) – Site 2 BUM Forwarding
DF VTEP 30010 VTEP
BGW11 BGW21
VTEP VTEP VXLAN EVPN VXLAN EVPN VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
DF VTEP 30010 VTEP BGW12 BGW22 5 Leaf20 sends traffic to local Host 2 Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81 Multi-Site Packet Walk (Bridging)
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 82 VXLAN Multi-Site Packet Walk Layer 2 (Host 1 to Host 2) – Site 1
Bridging
SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 30010 H1-MAC H2-MAC H1-IP H2-IP
Leaf10 performs L2 lookup and encapsulates toward VTEP VTEP
local BGW VIP1 address BGW11 BGW21 2 VTEP VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
1 VTEP VTEP Host 1 sends traffic BGW12 BGW22 destined to remote Host 2
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 83 VXLAN Multi-Site Packet Walk Layer 2 (Host 1 to Host 2) – DCI
Bridging
SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW-VIP2 30010 H1-MAC H2-MAC H1-IP H2-IP
BGW11 performs L2 lookup VTEP and encapsulates toward VTEP BGW11 remote BGW VIP2 address BGW21 3 VTEP VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
VTEP VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 84 VXLAN Multi-Site Packet Walk Layer 2 (Host 1 to Host 2) – Site 2
Bridging
SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 L20 30010 H1-MAC H2-MAC H1-IP H2-IP
VTEP VTEP BGW22 performs L2 lookup and encapsulates toward BGW11 BGW21 destination L20 node
VTEP 4 VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
VTEP VTEP BGW12 BGW22 5 Leaf20 bridges traffic to local Host 2 Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 85 VXLAN Multi-Site Packet Walk Layer 2 (Host 2 to Host 1) – Site 2
Bridging
SIP DIP VXLAN SMAC DMAC SIP DIP Payload L20 BGW-VIP2 30010 H2-MAC H1-MAC H2-IP H1-IP
Leaf20 performs L2 lookup VTEP VTEP and encapsulates toward BGW11 BGW21 local BGW VIP2 address
VTEP 7 VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
VTEP VTEP BGW12 BGW22 6 Host 2 replies to remote Host 1 Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 86 VXLAN Multi-Site Packet Walk Layer 2 (Host 2 to Host 1) – DCI
Bridging
SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 BGW-VIP1 30010 H2-MAC H1-MAC H2-IP H1-IP
BGW21 performs L2 VTEP lookup and encapsulates VTEP
BGW11 toward remote BGW VIP1 BGW21 address 8 VTEP VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
VTEP VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 87 VXLAN Multi-Site Packet Walk Layer 2 (Host 2 to Host 1) – Site 1
Bridging
SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 L10 30010 H2-MAC H1-MAC H2-IP H1-IP
BGW12 performs L2 lookup and encapsulates toward VTEP VTEP
destination L10 node BGW11 BGW21 9 VTEP VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN Leaf10 Site1 DCI Site2 Leaf20
VTEP VTEP 10 Leaf10 bridges traffic toward Host 1 BGW12 BGW22
Baremetal Baremetal
Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 88 Multi-Site Packet Walk (Routing)
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 89 VXLAN Multi-Site Packet Walk Layer 3 (Host 1 to Host 3) – Site 1
Routing
SIP DIP VXLAN SMAC DMAC SIP DIP Payload L10 BGW-VIP1 50001 L10-MAC BGW-VMAC1 H1-IP H3-IP
Leaf10 performs a L3 lookup and encapsulates toward local VTEP VTEP
BGW VIP1 address BGW11 BGW21 2 VTEP VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN VMAC1 VMAC2 Leaf10 Site1 DCI Site2 Leaf20
1 Host 1 sends a VTEP VTEP data packet to the BGW12 BGW22 remote Host 3
Baremetal Baremetal
Host 1 Host 3 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.20.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 90 VXLAN Multi-Site Packet Walk Layer 3 (Host 1 to Host 3) – DCI
Routing
SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP1 BGW-VIP2 50001 BGW-VMAC1 BGW-VMAC2 H1-IP H3-IP
BGW11 performs a L3 lookup VTEP and encapsulates toward VTEP
BGW11 remote BGW VIP2 address BGW21 3 VTEP VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN VMAC1 VMAC2 Leaf10 Site1 DCI Site2 Leaf20
VTEP VTEP
BGW12 BGW22
Baremetal Baremetal
Host 1 Host 3 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.20.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 91 VXLAN Multi-Site Packet Walk Layer 3 (Host 1 to Host 3) – Site 2
Routing
SIP DIP VXLAN SMAC DMAC SIP DIP Payload BGW-VIP2 L20 50001 BGW-VMAC1 L20-MAC H1-IP H3-IP
VTEP VTEP BGW21 performs a L3 lookup BGW11 BGW21 and encapsulates toward destination L20 node VTEP 4 VTEP VXLAN EVPN VIP1 VXLAN EVPN VIP2 VXLAN EVPN VMAC1 VMAC2 Leaf10 Site1 DCI Site2 Leaf20
VTEP VTEP BGW12 BGW22 5 Leaf20 routes traffic to local Host 3 Baremetal Baremetal
Host 1 Host 3 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.20.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 92 Connectivity to the External Layer 3 Domain VXLAN Multi-Site Connectivity to the External Layer 3 Domain
. Different connectivity models are supported
• VRF-Lite peering with external WAN Edge routers
• MP-BGP EVPN peering with external WAN Edge routers (Shared Border deployment model)
. Dedicated or shared pair of WAN Edge routers across sites
. The BGW nodes can also be used to provide Layer-3 external connectivity to each site
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 94 VXLAN Multi-Site Border Gateways and VRF-Lite to External Routers
Separate IPv4/IPv6 routing peering for each VRF (IGP Dedicated physical VRF-ExternalA VRF-B VRF-C or eBGP) . Separate IPv4/IPv6 routing peering for interfaces / sub- Connectivity interfaces for each VRF each VRF established with the external routers on dedicated physical Multi-Site interfaces/sub-interfaces
External Overlay -
Site . Must use separate interfaces for BGW BGW BGW BGW inter-site communication VTEP VTEP VTEP VTEP . No support for VXLAN encapsulated
traffic on sub-interfaces Internal
- VRF-A VRF-B VRF-C
Site Site 1
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 95 VXLAN Multi-Site Border Gateway Connectivity to Shared Border Router
Shared border router operates like a traditional VXLAN EVPN VTEP (Layer 3 only) . Single MP-BGP EVPN peering Single MP-BGP EVPN routing VXLAN Encapsulated established with the external instance to exchange routes VRF-ExternalA VRF-B VRF-C traffic destined to BGWs in for all VRFs Connectivity remote sites routers to exchange routes for all the VRFs
. VXLAN Data-Plane between the Multi-Site BGWs and the external routers Overlay Routed interface extending ‘underlay’ connectivity to
the external routers . Same spine uplinks used for all External - VXLAN encapsulated traffic (North-
Site South and East-West) BGW BGW BGW BGW . Required because of the use of DCI VTEP VTEP VTEP VTEP link tracking VXLAN Data Plane between BGW and
Internal Shared Border Router . Various northbound hand-off -
Site Site 1 options depending on specific HW support: VRF-Lite, MPLS-VPN, LISP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 96 VXLAN Multi-Site Internet/WAN Gateways Shared between Sites
External Internet/WAN WAN EdgeConnectivityWAN Edge
Multi-Site Overlay DCI
VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 BGW BGW BGW BGW Fabric …. Inter-Site VXLAN …. Communication between VXLAN EVPN Border Gateways VXLAN EVPN Spine Spine Spine Spine
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Host1 Host2 Host3 0000.3010.1101 0000.3020.2101 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97 VXLAN Multi-Site Per Site Internet/WAN Gateways
External Internet/WAN WAN Edge WAN Edge Connectivity WAN Edge WAN Edge
Multi-Site Overlay DCI
VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 BGW BGW BGW BGW Fabric …. Inter-Site VXLAN …. Communication between VXLAN EVPN Border Gateways VXLAN EVPN Spine Spine Spine Spine
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Host1 Host2 Host3 0000.3010.1101 0000.3020.2101 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 98 VXLAN Multi-Site Per Site Internet/WAN Gateways WAN Isolation Scenario External Internet/WAN WAN Edge WAN Edge Connectivity WAN Edge WAN Edge
Multi-Site Overlay DCI
VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 BGW BGW BGW BGW Fabric …. Inter-Site VXLAN …. Communication between VXLAN EVPN Border Gateways VXLAN EVPN Spine Spine Spine Spine
Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Host1 Host2 Host3 0000.3010.1101 0000.3020.2101 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.10.102
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 99
Legacy Site Integration Main Use Cases VXLAN Multi-Site with vPC BGWs Migration/Coexistence Use Case
Pair of vPC BGWs Layer-2 and Layer-3 Pair of vPC BGWs (EX/FX/FX2 Switches) Extension via VXLAN vPC (EX/FX/FX2 Switches)
VTEP VTEP VTEP VTEP BGW BGW BGW BGW IRIR forfor BUMBUM ++ aggregated BUM Spine Spine Spine Spine containment
VTEP VTEP VTEP VTEP VTEP VTEP VTEP
Greenfield Site Legacy Site
. Coexistence and/or migration use cases • Need to extend Layer-2 and Layer-3 multi-tenant connectivity across sites . Deploy a pair of vPC BGWs in the legacy site • Seamless connectivity extension via VXLAN • Leveraging native Multi-Site functions (Ingress Replication for BUM, BUM containment, etc.)
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 102 VXLAN Multi-Site with vPC BGWs Next-Gen DCI Use Case
Pair of vPC BGWs Layer-2 and Layer-3 Pair of vPC BGWs (EX/FX/FX2 Switches) vPC Extension via VXLAN vPC (EX/FX/FX2 Switches)
VTEP VTEP VTEP VTEP BGW BGW BGW BGW IRIR forfor BUMBUM ++ aggregated BUM containment
Legacy Site 1 Legacy Site 2
. A pair of vPC BGWs inserted in each legacy site to extend Layer-2 and Layer-3 connectivity between sites • Replacement of traditional DCI technologies (EoMPLS, VPLS, OTV, …) . Provides the option of slowing phasing out the legacy networks and replace them with modern VXLAN EVPN fabrics
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 103 VXLAN Multi-Site with vPC BGWs Next-Gen DCI Use Case with Back-to-Back BGWs
vPC vPC
VTEP VTEP VTEP VTEP BGW BGW BGW BGW
Legacy Site 1 Legacy Site 2
. Typical topology leveraging dedicated dark fiber links or DWDM circuits . ‘Squared’ and ‘full mesh’ topologies are both fully supported . Recommended to limit the back-to-back deployment to two sites 2 sites topology can be fully automated using DCNM Recommended to insert Layer 3 core network with 3+ sites
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 104 Migrating Legacy DCs to VXLAN EVPN Fabrics Migrating Legacy DCs to VXLAN EVPN Fabrics Step 1 - Insert a Pair of vPC BGWs in Each Legacy Site
vPC vPC
VTEP VTEP Double-sided vPC VTEP VTEP BGW BGW connections BGW BGW
L3 L3 L2 L2
Legacy Site 1 Legacy Site 2
. Recommended to deploy double-sided vPC connections between legacy aggregation devices and vPC BGWs Allows to create a single L2 logical connection with all links actively forwarding traffic Can apply BPDU filtering between aggregation devices and vPC BGWs to mitigate impact of TCNs . Default gateway functions still offered on the legacy aggregation devices (Active/Standby across sites)
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 106 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 1 - Insert a Pair of vPC BGWs in Each Legacy Site
vPC vPC
VTEP VTEP VTEP VTEP BGW BGW BGW BGW
Non vPC/MLAG capable devices
Legacy Site 1 Legacy Site 2
. Local port-channels from the aggregation switches can be used if those devices do not support vPC/MLAG . Spanning-Tree is then used to break the local Layer-2 loops
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 107 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 2 - Configure vPC BGWs DCI Underlay Peerings
EBGP vPC EBGP vPC
VTEP VTEP VTEP VTEP BGW BGW BGW BGW
Legacy Site 1 Legacy Site 2
. Establish underlay routing adjacencies with the first-hop L3 devices in the core network EBGP is the recommended protocol of choice Establish EBGP point-to-point peerings using the physical interfaces IP addresses . Underlay connectivity across the core network required to exchange BGW loopback addresses with the remote vPC BGWs
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 108 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 3 - Configure vPC BGWs DCI Overlay Peerings
vPC vPC VTEP VTEP MP-EBGP - EVPN VTEP VTEP BGW BGW BGW BGW
Legacy Site 1 Legacy Site 2
. Establish overlay routing adjacencies between vPC BGWs deployed in separate sites Mandatory establishment of EBGP session across sites Full-mesh EBGP peering is required Alternatively, can use route-server services in the core network
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 109 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 4 - Configure vPC BGWs for DCI Layer 2 Extension across Sites
Layer-2 Extension via VXLAN Map VLAN 10 vPC vPC Map VLAN 10 to L2VNI 5010 VTEP VTEP VTEP VTEP to L2VNI 5010 BGW BGW BGW BGW
L3 L3 Add VLAN 10 Add VLAN 10 L2 to L2 trunk to L2 trunk L2
Baremetal Baremetal Legacy Site 1 Legacy Site 2 VLAN 10 VLAN 10
. Layer-2 extension can now start being performed between vPC BGWs pairs Add the VLANs that need to be extended on the L2 trunk between legacy network and vPC BGWs Map the VLANs to L2VNI segments on the vPC BGW devices MAC information would start being advertised across sites for endpoints connected to those VLANs
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 110 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 4 - Configure vPC BGWs for DCI Layer 2 Extension across Sites
MAC NH MAC NH Layer-2 Extension via VXLAN 0000.3010.1101 Po1 0000.3010.1101 vPC VIP1 vPC vPC
0000.3010.1102 vPC VIP2 VTEP VTEP VTEP VTEP 0000.3010.1102 Po2 BGW BGW All Allendpoints End-Pointsin the in legacythe BGW BGW legacynetwork network are learned are learned as Po1 Po2 L3 directlyas directly connected connected to the to L3 theBGWs BGWs L2 L2
Baremetal Baremetal Legacy Site 1 Legacy Site 2 Host 1 Host 2 0000.3010.1101 0000.3010.1102 192.168.10.101 192.168.10.102
. Endpoints connected to the legacy network are discovered as directly connected to the local vPC BGW pair . VXLAN tunnels for intersite Layer-2 connectivity are established between the vPC VIP addresses
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 111 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 4 – Potential Creation of End-to-End Layer 2 Loops
Layer-2 Extension via VXLAN vPC L2 Loop! vPC VTEP VTEP Legacy DCI VTEP VTEP BGW BGW extending BGW BGW Po1 VLAN 10 Po2 L3 L3 L2 L2
Baremetal Baremetal Legacy Site 1 Legacy Site 2 VLAN 10 VLAN 10
. If a traditional DCI solution (OTV, VPLS, etc.) is already deployed between the legacy sites, it is possible to end up with the creation of an end-to-end Layer-2 loop . STP BPDUs are not forwarded via Multi-Site, so extending the same VLAN via traditional and NextGen DCI solutions at the same time should be avoided
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 112 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 5 - Migrate Default Gateway to the vPC BGWs
Layer-3 Extension via VXLAN vPC vPC
VTEP VTEP L3 L3 VTEP VTEP L3 BGW BGW BGW BGW L2 L2 L2 L3 Peering Po1 Po2 L3 Peering L3 L3 L2 L2
Baremetal Baremetal Legacy Site 1 Baremetal Baremetal Legacy Site 2 VLAN 10 VLAN 20 VLAN 30 VLAN 10 . The migration of the default gateway on the vPC BGW can be performed on a subnet by subnet basis . Allows to provide an all-active default gateway in both sites . Until the gateway for all the IP subnets is migrated, it is required to create a L3 peering between the legacy network and the vPC BGW
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 113 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 5 - Migrate Default Gateway to the vPC BGWs
IP NH IP NH Layer-3 Extension via VXLAN 192.168.10.101 Po1 192.168.10.101 vPC VIP1 vPC vPC 192.168.10.102 vPC VIP2 192.168.10.102 Po2 VTEP VTEP L3 L3 VTEP VTEP BGW BGW BGW BGW 192.168.20.101 NH Agg-L3 L2 L2 192.168.20.101 vPC VIP1 192.168.30.101 L3 PeeringvPC-VIP2 Po1 Po2 L3 Peering192.168.30.101 NH Agg-L3 L3 L3 L2 L2
Baremetal Baremetal Legacy Site 1 Baremetal Baremetal Legacy Site 2 Host 1 Host 3 Host 4 Host 2 0000.3010.1101 0000.3010.1103 0000.3010.1104 0000.3010.1102 192.168.10.101 192.168.20.101 192.168.30.101 192.168.10.102
. Endpoints connected to the legacy network are discovered as directly connected to the local vPC BGW pair . VXLAN tunnels to allow intersite Layer-3 connectivity are established between the vPC VIP addresses
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 114 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 6 – Start Deploying a New Local VXLAN Fabric
vPC vPC
VTEP VTEP L3 VTEP VTEP L3 BGW BGW BGW BGW L2 L2
Spine Spine Spine Spine
VTEP VTEP VTEP L3 L3 VTEP VTEP VTEP L2 L2
Baremetal Baremetal Baremetal Baremetal ‘Mixed’ Site 1 ‘Mixed’ Site 2
. Introduce VXLAN EVPN spines and additional VTEPs in each site . Migrate endpoints between the legacy network and the new VXLAN EVPN fabric
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 115 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 7 – Decommission the Legacy Networks
vPC vPC
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Spine Spine Spine
L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP L3 L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP L3 L2 L2 L2 L2
Baremetal Baremetal Greenfield Site 1 Baremetal Greenfield Site 2 Baremetal
. Decommission the legacy networks and leave only the VXLAN EVPN fabrics in place
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 116 Migrating Legacy DCs to VXLAN EVPN Fabrics Step 8 – Convert the vPC BGWs to Anycast BGWs (Optional)
VTEP VTEP VTEP VTEP
BGW BGW BGW BGW
Spine Spine Spine Spine
L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP L3 L3 VTEP VTEP VTEP VTEP VTEP VTEP VTEP L3 L2 L2 L2 L2
Baremetal Baremetal Greenfield Site 1 Baremetal Greenfield Site 2 Baremetal
. Move endpoints directly connected to the vPC BGW nodes (if any) to regular leaf nodes and migrate to the Anycast BGW model . Anycast BGW is the recommended deployment options . The migration can be done in a non disruptive way, one node at the time
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 117
Provisioning via CLI VXLAN Multi-Site Site 1 Setup – Enable Border Gateway
. Multi-Site Commands are marked in red . Various options do exist but the recommended design choices are: • Site-Internal IGP Underlay, iBGP Overlay • Site-External (primary choice) VTEP VTEP eBGP Underlay, eBGP Overlay Fabric BGW1 BGW2 …. Route Server for DCI Overlay peerings DC Core for reachability across n Sites Spine Spine • Site-External (alternative option) Any Routing Protocol Underlay, eBGP Overlay Full-Mesh for DCI Overlay peerings VTEP VTEP VTEP VTEP Back-to-Back Site Reachability (physical, full- mesh)
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 120 Anycast BGW VXLAN Multi-Site Site 1 Setup – Enable Border Gateway vPC BGW
BGW1 BGW2 feature nv overlay nv overlay evpn feature bgp
evpn multisite border-gateway
VTEP VTEP Fabric BGW1 …. BGW2 All BGWs part of the same site must be configured with the same site-id Spine Spine value
VTEP VTEP VTEP VTEP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 121 Anycast BGW VXLAN Multi-Site Site 1 Setup – Anycast BGW1 RID & VTEPs
BGW1
interface loopback0 description RID ip address 10.10.10.101/32 tag 54321 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode
interface loopback1
VTEP VTEP description PIP VTEP ip address 10.1.1.101/32 tag 54321 Fabric BGW1 …. ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode
Spine Spine interface loopback100 description VIP Multi-Site 1 ip address 10.1.1.111/32 tag 54321 ip router ospf UNDERLAY area 0.0.0.0 VTEP VTEP VTEP VTEP ip pim sparse-mode
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 122 Anycast BGW VXLAN Multi-Site Site 1 Setup – Anycast BGW2 Loopback & VTEP
BGW2
interface loopback0 description RID ip address 10.10.10.102/32 tag 54321 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode
interface loopback1 VTEP VTEP description PIP VTEP ip address 10.1.1.102/32 tag 54321 Fabric …. BGW2 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode
Spine Spine interface loopback100 description VIP Multi-Site 1 ip address 10.1.1.111/32 tag 54321 ip router ospf UNDERLAY area 0.0.0.0 VTEP VTEP VTEP VTEP ip pim sparse-mode
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 123 vPC BGW VXLAN Multi-Site Site 1 Setup – vPC BGW1 Loopback & VTEP
BGW1
interface loopback0 description RID ip address 10.10.10.101/32 tag 54321 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode
interface loopback1 VTEP VTEP description vPC VTEP ip address 10.1.1.101/32 tag 54321 Fabric BGW1 ip address 10.1.1.110/32 secondary tag 54321 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode
Spine Spine interface loopback100 description VIP Multi-Site 1 ip address 10.1.1.111/32 tag 54321 ip router ospf UNDERLAY area 0.0.0.0 VTEP VTEP VTEP VTEP ip pim sparse-mode
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 124 vPC BGW VXLAN Multi-Site Site 1 Setup – vPC BGW2 Loopback & VTEP
BGW2
interface loopback0 description RID ip address 10.10.10.102/32 tag 54321 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode
interface loopback1 VTEP VTEP description vPC VTEP ip address 10.1.1.102/32 tag 54321 Fabric BGW2 ip address 10.1.1.110/32 secondary tag 54321 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode
Spine Spine interface loopback100 description VIP Multi-Site 1 ip address 10.1.1.111/32 tag 54321 ip router ospf UNDERLAY area 0.0.0.0 VTEP VTEP VTEP VTEP ip pim sparse-mode
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 125 vPC BGW VXLAN Multi-Site Site 1 Setup – vPC BGW1 vPC Domain Configuration
BGW1
feature vPC
vPC domain 1 peer-switch peer-keepalive destination 172.19.217.122 \
VTEP VTEP source 172.19.217.123 delay-restore 150 Fabric BGW1 peer-gateway auto-recovery reload-delay 360 ipv6 nd synchronize Spine Spine ip arp synchronize
interface port-channel10 vPC peer-link
VTEP VTEP VTEP VTEP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 126 vPC BGW VXLAN Multi-Site Site 1 Setup – vPC BGW2 vPC Domain Configuration
BGW2
feature vPC
vPC domain 1 peer-switch peer-keepalive destination 172.19.217.123 \
VTEP VTEP source 172.19.217.122 delay-restore 150 Fabric BGW2 peer-gateway auto-recovery reload-delay 360 ipv6 nd synchronize Spine Spine ip arp synchronize
interface port-channel10 vPC peer-link
VTEP VTEP VTEP VTEP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 127 Anycast BGW VXLAN Multi-Site Site 1 Setup – Fabric Link Tracking on BGW1 vPC BGW BGW1 interface Ethernet1/53 description TO-SPINE1 • For vPC BGWs, the vPC peer-link does not need ip address 10.0.1.1/30 to be tracked as a Fabric link ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking
interface Ethernet1/54 description TO-SPINE2 ip address 10.0.2.1/30 ip router ospf UNDERLAY area 0.0.0.0
VTEP VTEP ip pim sparse-mode evpn multisite fabric-tracking Fabric BGW1 BGW2 …. BGW2 e1/53 e1/54 e1/53 e1/54 interface Ethernet1/53 description TO-SPINE1 Spine Spine ip address 10.0.1.5/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking
VTEP VTEP VTEP VTEP interface Ethernet1/54 description TO-SPINE2 ip address 10.0.2.5/30 ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode evpn multisite fabric-tracking #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 128 Anycast BGW VXLAN Multi-Site Site 1 Setup – Multi-Site DCI Tracking vPC BGW
BGW1 interface Ethernet1/1 • For vPC BGWs, the vPC peer-link does not need description TO-DC-CORE1 to be tracked as a DCI link ip address 10.111.111.1/30 tag 12345 DC Coreevpn multisite dci-tracking (Layer-3 Unicast) interface Ethernet1/2 description TO-DC-CORE2 ip address 10.111.222.1/30 tag 12345 evpn multisite dci-tracking DCI e1/1 e1/1 e1/2 e1/2 VTEP VTEP BGW2 Fabric BGW1 BGW2 interface Ethernet1/1 …. description TO-DC-CORE1 ip address 10.222.111.1/30 tag 12345 evpn multisite dci-tracking Spine Spine interface Ethernet1/2 description TO-DC-CORE2 ip address 10.222.222.1/30 tag 12345 evpn multisite dci-tracking VTEP VTEP VTEP VTEP
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 129 Anycast BGW VXLAN Multi-Site Site 1 BGW1 Setup – Multi-Site Underlay Peering vPC BGW
RS
DC Core (Layer-3 Unicast)
BGW1 router bgp 65501 DCI e1/1 e1/2 router-id 10.10.10.101 VTEP VTEP address-family ipv4 unicast
BGW1 redistribute direct route-map REDIST-LOCAL Fabric …. neighbor 10.111.111.2 remote-as 65599 update-source ethernet1/1 Spine Spine address-family ipv4 unicast neighbor 10.111.222.2 remote-as 65599 update-source ethernet1/2 address-family ipv4 unicast VTEP VTEP VTEP VTEP
RS – Route Server (eBGP ”Route Reflector”) #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 130 Anycast BGW VXLAN Multi-Site Site 1 BGW2 Setup – Multi-Site Underlay Peering vPC BGW
RS
DC Core (Layer-3 Unicast)
BGW2 router bgp 65501 DCI e1/1 e1/2 router-id 10.10.10.102 VTEP VTEP address-family ipv4 unicast
BGW2 redistribute direct route-map REDIST-LOCAL Fabric …. neighbor 10.222.111.2 remote-as 65599 update-source ethernet1/1 Spine Spine address-family ipv4 unicast neighbor 10.222.222.2 remote-as 65599 update-source ethernet1/2 address-family ipv4 unicast VTEP VTEP VTEP VTEP
RS – Route Server (eBGP ”Route Reflector”) #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 131 vPC BGW VXLAN Multi-Site Site 1 vPC BGW1 Setup – Underlay Peerings over Peer-Link
• Recommended to establish both OSPF (or IS-IS) and iBGP peering over the vPC Peer-Link RS iBGP peering needed to handle traffic recovery DC Core BGW1 under specific failure scenarios vlan 3600 (Layer-3 Unicast) interface Vlan3600 description vPC-Peer-Link SVI no shutdown DCI mtu 9216 no ip redirects VTEP VTEP VLAN 3600 ip address 10.1.10.49/30 Fabric BGW1 no ipv6 redirects OSPF + iBGP IPv4 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode Spine Spine system nve infra-vlans 3600
router bgp 65501 neighbor 10.1.10.50 VTEP VTEP VTEP VTEP remote-as 65501 address-family ipv4 unicast
RS – Route Server (eBGP ”Route Reflector”) #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 132 vPC BGW VXLAN Multi-Site Site 1 vPC BGW2 Setup – Underlay Peerings over Peer-Link
• Recommended to establish both OSPF (or IS-IS) and iBGP peering over the vPC Peer-Link RS iBGP peering needed to handle traffic recovery DC Core BGW2 under specific failure scenarios vlan 3600 (Layer-3 Unicast) interface Vlan3600 description vPC-Peer-Link SVI no shutdown DCI mtu 9216 no ip redirects VTEP VTEP VLAN 3600 ip address 10.1.10.50/30 Fabric BGW2 no ipv6 redirects OSPF + iBGP IPv4 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 ip pim sparse-mode Spine Spine system nve infra-vlans 3600
router bgp 65501 neighbor 10.1.10.49 VTEP VTEP VTEP VTEP remote-as 65501 address-family ipv4 unicast
RS – Route Server (eBGP ”Route Reflector”) #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 133 Anycast BGW VXLAN Multi-Site Site 1 BGW1 Setup – Multi-Site Overlay Peering vPC BGW
RS
DC Core (Layer-3 Unicast)
BGW1 DCI router bgp 65501 VTEP VTEP router-id 10.10.10.101 neighbor 10.99.99.201 BGW1 Fabric …. remote-as 65599 update-source loopback0 ebgp-multihop 5 Spine Spine peer-type fabric-external address-family l2vpn evpn send-community send-community both rewrite-evpn-rt-asn
VTEP VTEP VTEP VTEP
RS – Route Server (eBGP ”Route Reflector”) #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 134 Anycast BGW VXLAN Multi-Site Site 1 BGW2 Setup – Multi-Site Overlay Peering vPC BGW
RS
DC Core (Layer-3 Unicast)
BGW2 DCI router bgp 65501 VTEP VTEP router-id 10.10.10.102 neighbor 10.99.99.201 BGW2 Fabric …. remote-as 65599 update-source loopback0 ebgp-multihop 5 Spine Spine peer-type fabric-external address-family l2vpn evpn send-community send-community both rewrite-evpn-rt-asn
VTEP VTEP VTEP VTEP
RS – Route Server (eBGP ”Route Reflector”) #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 135 Anycast BGW VXLAN Multi-Site Site 1 Setup – Multi-Site Overlay Peering vPC BGW
RS
DC Core (Layer-3 Unicast)
DCI . peer-type fabric-external VTEP VTEP • Enables Next-Hop Rewrite for Multi-Site Fabric BGW1 …. BGW2 • Defines Site External BGP neighbors for EVPN exchange
Spine Spine . rewrite-evpn-rt-asn • Rewrites Route-Target Auto information to simplify MAC-VRF and IP-VRF configuration • Normalizes outgoing Route-Targets AS number to VTEP VTEP VTEP VTEP match remote AS number • Uses BGP configured Neighbors Remote AS
RS – Route Server (eBGP ”Route Reflector”) #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 136 Anycast BGW VXLAN Multi-Site Site 1 Setup – Multi-Site Overlay Peering vPC BGW
BGP Update: MAC: 0000.3010.1101 (L2VNI 30001)DC Core IP: 192.168.10.101 (L3VNI(Layer 50001)-3 Unicast) NH: 10.1.1.111 RMAC: BGW-VMAC1 Rewrite Next-Hop IP and Next- Hop MAC (RMAC) based on DCI Neighbor Site BGW peer-type fabric-external peer-type fabric-external VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 Fabric BGW …. BGW BGW BGP… Update:. BGW BGP Update: MAC: 0000.3010.1101 (L2VNI 30001) MAC: 0000.3010.1101 (L2VNI 30001) IP: 192.168.10.101 (L3VNI 50001) IP: 192.168.10.101VXLAN (L3VNI EVPN 50001) Rewrite Next-Hop IP and Next- VXLANNH: 10.2.2.222 EVPN NH: Spine10.1.1.1 Spine Hop MAC (RMAC) based on Spine RMAC: BGW-VMAC2Spine RMAC: Leaf1 Neighbor Site BGW Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Host1 0000.3010.1101 192.168.10.101
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 137 Anycast BGW VXLAN Multi-Site Site 1 Setup – Multi-Site Overlay Peering vPC BGW
route-target both auto RT = ASN:VNI
DC Core BGP Update: Rewrite Route-Target based on (Layer-3 Unicast)Remote AS: 65501 BGP Neighbors Remote ASN VNI: 50001 Route-Target: 65501:50001
DCI rewrite-evpn-rt-asn rewrite-evpn-rt-asn VTEP VIP1 VTEP VTEP VIP2 VTEP 10.1.1.111 10.2.2.222 BGW BGW BGW BGW Fabric …. BGP… Update:. Remote AS: 65502 BGP Update:VXLAN EVPN VXLANVNI: EVPN 50001 RemoteSpine AS : 65501 Spine Spine Route-Target:Spine65502:50001 VNI: 50001 Route-Target: 65501:50001Site1 Site2
VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Host1 Host2 0000.3010.1101 0000.3020.2101 192.168.10.101 192.168.20.101
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 138 Anycast BGW VXLAN Multi-Site Site 1 Setup – Anycast BGW VTEP Configuration vPC BGW
DC Core (Layer-3 Unicast)
BGW1 BGW2 DCI interface nve1 VTEP VTEP no shutdown host-reachability protocol bgp Fabric BGW1 BGW2 …. source-interface loopback1 multisite border-gateway interface loopback100 member vni 30010 Spine Spine multisite ingress-replication mcast-group 239.1.1.1 member vni 30011-30020 mcast-group 239.1.1.2 VTEP VTEP VTEP VTEP member vni 50001 associate-vrf
RS – Route Server (eBGP ”Route Reflector”) #CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 139 Anycast BGW VXLAN Multi-Site Site 1 Setup – Anycast BGW VTEP Configuration vPC BGW
DC Core (Layer-3 Unicast)
DCI
VTEP VTEP Fabric BGW1 …. BGW2 . multisite border-gateway interface loopback# • Defines the Loopback Interface used for the Border Gateway Virtual IP Address (VIP) Spine Spine . multisite ingress-replication • Per-VNI knob for extending Layer-2 VNI
VTEP VTEP VTEP VTEP • Defines the Multi-Site BUM Replication method
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 140 Anycast BGW VXLAN Multi-Site Site 1 Setup – BUM Traffic Policing vPC BGW
DC CoreBGW1 BGW2 (Layer-3 Unicast) evpn storm-control broadcast level 10 evpn storm-control unicast level 10 evpn storm-control multicast level 10 DCI
VTEP VTEP Fabric BGW1 …. BGW2 . BUM Traffic Policing • Limits Broadcast, Unknown Unicast and Layer-2
Spine Spine Multicast Traffic across sites • Aggregate policing for each traffic type, enforced on encapsulation towards remote Sites Level 0 = No B/U/M Forwarding VTEP VTEP VTEP VTEP Level 100 = All B/U/M Forwarding Forwarding
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 141 Conclusions Multi-Site Advantages – ”The Multiple”
. Multiple Overlay Domains – Interconnected & Controlled • Scaling and Segregating VXLAN EVPN Networks . Multiple Overlay Control-Plane Domains – Interconnected & Controlled • Limited Overlay Control-Plane Update Propagation . Multiple Underlay Domains - Isolated • Isolated Underlay Domains – No need for Extension . Multiple Replication Domains for BUM – Interconnected & Controlled • Individual BUM flooding domain with Traffic control
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 143 Resources
• VXLAN EVPN Multi-Site Design and Deployment White Paper https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11- 739942.html
• NextGen DCI with VXLAN EVPN Multi-Site Using vPC Border Gateways White Paper https://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/whitepaper-c11- 742114.html
• Cisco Live Online - VXLAN BGP EVPN based Multi-POD, Multi-Fabric and Multi-Site - BRKDCN-2035 https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035&showMyInterest=false#/ • Cisco DCNM 11.1(2) - Multi-Site Domain for VXLAN BGP EVPN Fabrics https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/11_2_1/config_guide/lanfabric/b_dcnm_fabric_lan.html
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 144 In Summary…
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 145 Complete your online session • Please complete your session survey after each session. Your feedback evaluation is very important.
• Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live water bottle.
• All surveys can be taken in the Cisco Live Mobile App or by logging in to the Session Catalog on ciscolive.cisco.com/us.
Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com.
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 146 Continue your education
Demos in the Walk-in labs Cisco campus
Meet the engineer Related sessions 1:1 meetings
#CLUS BRKDCN-2035 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 147 Thank you
#CLUS #CLUS