VXLAN Design and Deployment
Marian Klas, Systems Engineer [email protected] Agenda
• Why VXLAN?
• VXLAN Fundamentals
• Underlay Deployment Considerations
• Overlay Deployment Considerations
• Summary and Conclusion
2 Trend: Flexible Data Center Fabrics Create Virtual Networks on top of an efficient IP network
Mobility Segmentation + Policy Scale Automated & Programmable
V V Physical M M Full Cross Sectional BW Hosts O O S S Virtual L2 + L3 Connectivity Use VXLAN to Create DC Fabrics Physical + Virtual
3 VXLAN Fundamentals
4 Why Overlays?
Seek well integrated best in class Overlays and Underlays
Robust Underlay/Fabric Flexible Overlay Virtual Network
• High Capacity Resilient Fabric • Mobility – Track end-point attach at edges
• Intelligent Packet Handling • Segmentation
• Programmable & Manageable • Scale – Reduce core state • Distribute and partition state to network edge
• Flexibility/Programmability • Reduced number of touch points
5 Overlay Taxonomy
Overlay Control Plane Service = Virtual Network Instance (VNI) VTEPs Identifier = VN Identifier (VNID) NVE = Network Virtualization Edge Encapsulation VTEP = VXLAN Tunnel End-Point Edge Devices (NVE) Edge Device (NVE) Hosts Underlay Network (end-points)
Underlay Control Plane
6 VXLAN is an Overlay Encapsulation Data Plane Learning Protocol Learning Flood and Learn over a multidestination Advertise hosts in a protocol distribution tree joined by all edge devices amongst edge devices
Overlay Control Plane
Encapsulation
VXLAN t
7 VXLAN Packet Structure Ethernet in IP with a shim for scalable segmentation
Outer MAC Header Outer IP Header Outer UDP Header VXLAN Header Original Layer 2 Frame FCS
14 Bytes (4 Bytes Optional) 20 Bytes 8 Bytes 8 Bytes Ethernet Payload VNI Tag Port Source 0x8100 Header 0x0000 0x0800 Address Dest. IP Address Protocol VLAN ID Reserved Reserved RRRRIRRR Source IP Src. MAC Checksum Checksum IP Header 0x11 (UDP) VLAN Type Dest. MAC Misc. Data Ether Type VXLAN Port UDP Length VXLAN Flags
48 48 16 16 16 72 8 16 32 32 16 16 16 16 8 24 24 8
Src VTEP MAC Address Src and Dst addresses of Large scale the VTEPs Allows for 16M segmentation UDP 4789 possible segments Next-Hop MAC Address Hash of the inner L2/L3/L4 headers of the original frame. Enables entropy for ECMP Load Tunnel Entropy 50 (54) Bytes of overhead balancing in the Network.
8 Data Plane Learning Dedicated Multicast Distribution Tree per VNI
PIM Join for Multicast PIM Join for Multicast Group 239.1.1.1 Group 239.2.2.2
V V V V V Web DB DB Web VM VM VM VM
9 Data Plane Learning Dedicated Multicast Distribution Tree per VNI
V V V V V
VM1 VM2 VM3
10 Data Plane Learning Learning on Broadcast Source - ARP Request Example
ARP Req IP A è G
ARP Req IP A è G V V V V V
ARP Req VM1 VM2 VM3 MAC IP Addr MAC IP Addr VM 1 VTEP 1 VM 1 VTEP 1
ARP Req ARP Req 11 Data Plane Learning Learning on Unicast Source - ARP Response Example
ARP Resp V V V V V
ARP Resp VTEP 2 è VTEP 1 VM1 VM2 VM3 MAC IP Addr MAC IP Addr VM 2 VTEP 2 VM 1 VTEP 1
12 ARP Resp VXLAN Evolution
• Head-end replication enables unicast-only mode Multicast Independent • Control Plane provides dynamic VTEP discovery
Protocol Learning • Workload MAC addresses learnt by VXLAN NVEs • Advertise L2/L3 address-to-VTEP association prevents floods information in a protocol
• VXLAN HW Gateways to other encaps/networks External Connectivity • VXLAN HW Gateway redundancy • Enable hybrid overlays
• VXLAN Routing IP Services • Distributed IP Gateways
14 VXLAN Evolution: Using a Control Protocol VTEP Discovery BGP consolidates and 2 propagates VTEP list for VNI
BGP Route Reflector VTEPs advertise their VNI membership in BGP 1
1 1
IP A IP C IP B TOR 1 V V V V V Overlay Neighbors TOR 3 , IP C TOR 3 TOR 2 TOR 2 , IP B VTEP can perform 3 4 VTEP obtains list of Head-End Replication VTEP neighbors for each VNI 15 VXLAN Unicast Mode Head-end replication *Broadcast, Unknown Unicast or Multicast
5 Frames are unicast to the neighbors
4 VXLAN Encap
BUM Frame IP A è IP B IP A IP C IP B Overlay Neighbors BUM Frame IP A è IP C POD3 , IP C TOR 1 V V V V V POD2 , IP B 3 VTEP performs Head- 2 End Replication TOR 3 TOR 2 VTEP retrieves the list of Overlay Neighbors** 1 BUM Frame **Information statically configured or dynamically retrieved via control plane A host sends a L2 BUM* frame (VTEP discovery) BGP EVPN Control Plane for VXLAN Host and Subnet Route Distribution
Route-Reflectors deployed for scaling purposes
RR RR
iBGP Adjacencies V V V V V
§ Host Route Distribution decoupled from the Underlay protocol § Use MP-BGP on the leaf nodes to distribute internal host/subnet routes and external reachability information 17 BGP EVPN Control Plane MAC IP VNI Next- Encap Seq Hop Host Advertisement 1 1 5000 IP L1 VXLAN 0 MAC L1
NLRI: Host MAC1, IP1 NVE IP L1/MAC L1 RR RR VNI 5000 Ext.Community: Encapsulation: VXLAN, NVGRE Sequence 0
V V V V V
VNI 5000
Host 1 VLAN 10 1. Host Attaches 2. Attachment NVE advertises host’s MAC (+IP) through BGP RR 3. Choice of encapsulation is also advertised 18 BGP EVPN Control Plane MACMAC IPIP VNIVNI NextNext-- EncapEncap SeqSeq HopHop Host Moves 11 11 50005000 IPIP L1L3 VXLANVXLAN 01 MACMAC L1L3
NLRI: Host MAC1, IP1
NVE IP L3/MAC L3 RR RR VNI 5000 Ext.Community: Encapsulation: VXLAN, NVGRE Sequence 1
V V V V V
VNI 5000
Host 1 VLAN 10 1. Host Moves to NVE3 2. NVE3 detects Host1 and advertises H1 with seq#1 19 3. NVE1 sees more recent route and withdraws its advertisement VXLAN L2 and L3 Gateways Connecting VXLAN to the broader network
Egress interface chosen (bridge L2 Gateway: VXLAN to VLAN may .1Q tag the packet) VXLANORANGE VXLAN L2 Bridging Ingress VXLAN packet on Gateway Orange segment
Destination is in another segment. Ingress VXLAN packet on Packet is routed to the new segment L3 Gateway: VXLAN to X Routing Orange segment SVI • VXLAN VXLANORANGE VXLANBLUE VLAN100 VXLAN VLAN200 • VLAN Router Egress interface chosen (bridge may .1Q tag the packet)
N7K w/F3 ASR1K/ASR Nexus 3K N5600 N9K LC 9K L2 Gateway Yes Yes Yes Yes Yes L3 Gateway Yes No Yes Yes Yes VXLAN BGP EVPN Nexus Programmable Fabric Support
Platform Spine Leaf Function Border Leaf DCNM Fabric Function Function Management
Nexus 5600 Yes Yes Yes Yes 7.3(0)N1(1) 7.3(0)N1(1) 7.3(0)N1(1) 7.2(3)
Nexus Yes Yes Yes Yes 7000/7700 7.3(0)D1(1) 7.3(0)D1(1) 7.3(0)D1(1) 7.2(3) (F3 Only) Nexus 9300 Yes Yes Yes Yes 7.0(3)I1(3) 7.0(3)I1(3) 7.0(3)I1(3) 7.2(3)
Nexus 9500 Yes Yes Yes Yes 7.0(3)I1(3) 7.0(3)I1(3) 7.0(3)I1(3) 7.2(3)
21 Underlay Deployment Considerations
22 Leaf/Spine Topology – Underlay Considerations
• High Bi-Sectional Bandwidth • Wide ECMP: Unicast or Multicast • Uniform Reachability, ECMP Links Deterministic Latency (Layer-3) • High Redundancy: Node/Link Failure • Line rate, low latency, for all traffic Deployment Considerations Underlay
• MTU and Overlays
• Unicast Routing Protocol and IP Addressing
• Multicast for BUM* Traffic Replication
24 MTU and VXLAN Underlay
• VXLAN adds 50 Bytes to the Outer MAC Header Original Ethernet Frame
• Avoid Fragmentation by adjusting Outer IP Header Underlay the IP Networks MTU
UDP Header • Data Centers often require Jumbo MTU; most Server NIC do support VXLAN Header up to 9000 Bytes
• Using a MTU of 9216* Bytes 50 (54) Bytes of Overhead accommodates VXLAN Overhead Original Layer-2 Frame Overlay plus Server max. MTU
*Cisco Nexus 5600/6000 switches only support 9192 Byte for Layer-3 Traffic
25 Building your IP Network – Interface Principles Underlay
• Know your IP addressing and IP scale requirements • Best to use single Aggregate for all Underlay Links and Loopbacks • IPv4 only • For each Point-2-Point (P2P) 2 connection, minimum /31 required V1 V • Loopback requires /32
• Routed Ports/Interfaces • Layer-3 Interfaces between Spine and Leaf (no switchport) V3 • VTEP uses Loopback as Source-
Interface 26 Building your IP Network – Routing Protocols; OSPF Underlay
• OSPF – watch your Network type • Network Type Point-2-Point (P2P) • Preferred (only LSA type-1) • No DR/BDR election • Suits well for routed interfaces/ports (optimal from a LSA Database perspective) 2 • Full SPF calculation on Link Change V1 V • Network Type Broadcast • Suboptimal from a LSA Database perspective (LSA type-1 & 2) • DR/BDR election • Additional election and Database 3 Overhead V
29 Building your IP Network – Routing Protocols; IS-IS Underlay
• IS-IS – what was this CLNS? - Independent of IP (CLNS) - Well suited for routed interfaces/ports - No SPF calculation on Link change; only if Topology changes - Fast Re-convergence 2 1 V - Not everyone is familiar with it V
V3
31 Building your IP Network – Routing Protocols; iBGP Underlay
• iBGP + IGP = The Routing Protocol Combo • IGP for underlay topology & reachability (e.g. IS-IS, OSPF) • iBGP for VTEP (loopback) reachability • iBGP route-reflector for simplification and scale RR RR RR RR • Requires two routing protocols
iBGP • Separates Links (IGP) from VTEPs 2 1 V (iBGP) V • End-Host information are still in iBGP but different address-family
V3
33 Multicast Enabled Underlay
• May use PIM-ASM or PIM-BiDir (Different hardware has different capabilities) Nexus 7K Nexus 9K CSR 1000V N1KV Nexus 3K Nexus 5K/6K ASR9K with F3 LC Standalone ASR1K PIM-ASM PIM-ASM PIM-ASM & Bidir-PIM PIM-ASM & Mcast mode IGMP v2/v3 (Bidir – Bidir-PIM (Bidir – Bidir-PIM (ASM –Future) Bidir-PIM Future) Future) • Cisco Nexus 9000 Series Switches (supports ASM/SSM/Ingress-Replication for VXLAN VTEP support) and Cisco Nexus 5600 Series Switches (supports only PIM BIDIR for VXLAN VTEP support) can be part of the same VXLAN EVPN fabric but not share the same Layer-2 VNI.
• Spine and Aggregation switches make good RP locations in clos and traditional topologies respectively
• Reserve a range of multicast channels to service the overlay and optimize for diverse VNIs
• In clos topologies with lean spine, using multiple RPs across the multiple spines and mapping different VNIs to different RPs will provide a simple load balancing measure
• Design a multicast underlay for a network overlay, host VTEPs will simply leverage this network. 35 Multicast Enabled Underlay Host Overlay to Hybrid Overlay
• Host Overlay VTEPs join multicast groups as hosts using IGMP reports
• Host overlays will work over an L2 underlay, ensure IGMP snooping is in PIM place to scope the reach of multicast
• A multicast enabled L3 underlay is the better option as it enables a hybrid overlay (host and network VTEPs) IGMP HW VTEP • Ensure that the first hop router for the host in the underlay is configured to service the SW VTEP IGMP reports from the host VTEP
VNI 6000
36 Multi-Pathing and Entropy
NV-edge NV-edge
• Symmetric Underlay Network topologies facilitate ECMP routing: • Multi-path load balancing • Fast Re-convergence on link Failures
• Polarization: Encapsulated flows appear as a single flow which hashes to a single path
• Entropy in the encapsulation header to depolarize tunnels • Variable UDP source port in VXLAN outer header • Underlay must support ECMP hashing on L4 port numbers
49 Overlay Deployment Considerations
55 Type of Overlay Service
Layer 2 Overlays Layer 3 Overlays
• Emulate a LAN segment • Abstract IP based connectivity
• Transport Ethernet Frames (IP and non-IP) • Transport IP Packets • Single subnet mobility (L2 domain) • Full mobility regardless of subnets
• Exposure to open L2 flooding • Contain network related failures (floods)
• Useful in emulating physical topologies • Useful in abstracting connectivity and policy
Hybrid L2/L3 Overlays offer the best of both domains
Building your VTEP (VXLAN Tunnel End-Point) Overlay
Configuration Example # Features & Globals Enables VTEP (only required on Leaf or Border) feature bgp feature nv overlay nv overlay evpn Enables EVPN Control-Plane in BGP # Spine (S1)
# Leaf (V1) Configure the VTEP interface interface nve1 source-interface loopback0 RR RR RR RR host-reachability protocol bgp Use a Loopback for Source Interface
iBGP 2 1 V Enable BGP for Host reachabilityV
V3
*Simplified BGP configuration; would have 4 BGP peers (RR) IGP not shown 58 Building your Overlay Control-Plane Overlay
Configuration Example # Features & Globals feature bgp feature nv overlay nv overlay evpn Enables EVPN Control-Plane in BGP # Spine (S1) router bgp 65500 router-id 10.10.10.S1 address-family ipv4 unicast address-family l2vpn evpn RR RR RR RR neighbor 10.10.10.V1 remote-as 65500 Activate L2VPN EVPN under each BGP neighbor update-source loopback0
address-family l2vpn evpn iBGP 2 send-community both V1 V route-reflector-client
# Leaf (V1) router bgp 65500 router-id 10.10.10.V1 Send Extended BGP Community address-family ipv4 unicast to distribute EVPN route attributes V3 neighbor 10.10.10.S1 remote-as 65500 *Simplified BGP configuration; would have 4 BGP peers (RR) update-source loopback0 59 address-family l2vpn evpn IGP not shown send-community both * Extend your VLAN to VXLAN Overlay
• Mapping a IEEE 802.1Q VLAN ID to a VXLAN Configuration Example VNI # Features • VLAN to VNI configuration on a per-Switch based feature vn-segment-vlan-based
• VLAN becomes “Switch Local Identifier” # VLAN to VNI mapping (MT-Lite) • VNI becomes “Network Global Identifier” vlan 43 VLAN to Layer-2 VNI mapping vn-segment 30000 • 4k VLAN limitation per-Switch does still apply • 4k Network limitation has been removed # Activate Layer-2 VNI for EVPN • Dependent on VLAN Space! evpn vni 30000 l2 Enables EVPN Control- rd auto Plane for Layer-2 route-target import auto Services route-target export auto Alternative is to use VLAN # Activate Layer-2 VNI on VTEP “ingress-replication interface nve1 protocol bgp” source-interface loopback0 ethernet VLAN VNI vxlan host-reachability protocol bgp member vni 30000 Multi-Tenancy Lite (MT-Lite) mcast-group 239.239.239.100 suppress-arp Enables Layer-2 VNI on VTEP and suppress60 ARP Distributed Gateway Function in L3 Overlays
L3 Boundary
L2/L3 Fabric L3 Boundary
App App App App OS OS OS OS Virtual Physical Virtual Physical Traditional L2 - centralised L2/L3 boundary L2/L3 fabric (or overlay)
• Always bridge, route only at an aggregation point • Always route (at the leaves), bridge when necessary
• Large amounts of state converge • Distribute and disaggregate necessary state
• Scale problem for large# of L2 segments • Optimal scalability
• Traditional L2 and L2 overlays • Enhanced forwarding and L3 overlays 61 Distributed IP Anycast Gateway
The same “Anycast” SVI IP/MAC is used at all VTEPs/ToRs A host will always find its SVI anywhere it moves
L3 Fabric
VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 Gateway Gateway Gateway Gateway Gateway
SVI IP Address SVI IP Address MAC: 0000.dead.beef MAC: 0000.dead.beef IP: 10.1.1.1 IP: 10.1.1.1 VM VM
OS OS 62 Distributed IP Anycast Gateway Detailed View Underlay Underlay / IP Core / IP Core
L3 GWY L3 GWY SVI A SVI B SVI B SVI A
VLAN A' VNI A VLAN A VLAN B' VNI B VLAN B VTEP L2 GWY L2 GWY VTEP
Consistent Anycast SVI IP / MAC L3 Fabric address at all leaves
VXLAN L3 VXLAN L3 VXLAN L3 VLAN-IDs are locally significant Gateway Gateway Gateway
63 Distributed IP Anycast Gateway* Configuration example
V2
V3
V1
Host Y VLAN 55 Host A VLAN 43 64
*Requires EVPN Control-Plane. Routing in VXLAN – define the resources Overlay
Configuration Example for VRF-A
vxlan # Define VLAN for VRF routing instance vlan 2500 vn-segment 50000 VLAN to Layer-3 VNI mapping
# Define SVI for VRF routing instance interface Vlan2500 VNI50000 no shutdown VLAN to Layer-3 VNI mapping mtu 9216 - ip forward required for prefix- vrf member VRF-A based routing ip forward
VRF-A # VRF configuration for “customer” VRF vrf context VRF-A vni 50000 VRF context definition rd auto - VNI ethernet ethernet address-family ipv4 unicast route-target both auto - Route-Distinguisher route-target both auto evpn - Route-Targets - IPv4 and/or IPv6
65 Distributed IP Anycast Gateway* Overlay
Configuration Example for “BLUE” (V1 & V3) Configuration Example for “RED” (V1-3) # Features & Globals # Features & Globals feature interface-vlan feature interface-vlan fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF fabric forwarding anycast-gateway-mac 2020.DEAD.BEEF
# VLAN to VNI mapping (MT-Lite) # VLAN to VNI mapping (MT-Lite) vlan 43 vlan 55 vn-segment 30000 vn-segment 30001
# Anycast Gateway MAC, inherited by any interface # Anycast Gateway MAC, inherited by any interface (SVI) using “fabric forwarding” (SVI) using “fabric forwarding” fabric forwarding anycast-gateway-mac 0002.0002.0002 fabric forwarding anycast-gateway-mac 0002.0002.0002
# Distributed IP Anycast Gateway (SVI) # Distributed IP Anycast Gateway (SVI) interface vlan 43 interface vlan 55 no shutdown no shutdown vrf member VRF-A vrf member VRF-A ip address 11.11.11.1/24 tag 12345 ip address 98.98.98.1/24 tag 12345 fabric forwarding mode anycast-gateway fabric forwarding mode anycast-gateway
*Requires EVPN Control-Plane. VRF and BGP configuration not shown 66 Routing in VXLAN – configure the routing Overlay
Enables Layer-3 VNI on VTEP Configuration Example for VRF-A
and associatevxlan it to VRF # Activate Layer-3 VNI on VTEP interface nve1 source-interface loopback0 host-reachability protocol bgp member vni 30000 mcast-group 239.239.239.100 VNI50000 suppress-arp member vni 50000 associate-vrf
# Route-Map for Redistribute Subnet route-map REDIST-SUBNET permit 10 VRF-A match tag 12345
# Control-Plane configuration for VRF (Tenant) router bgp 65500 ethernet ethernet … vrf VRF-A address-family ipv4 unicast VRF/Tenant definition advertise l2vpn evpn within Overlay Control-Plane redistribute direct route-map REDIST-SUBNET maximum-paths ibgp 2
67
Host Subnet Redistribution Overlay
• Host “A” is a silent Host • Not known via ARP/IP
• How can Host “Y” reach Host “A” I know Subnet “A” • Host “A” and “Y” are in different VLAN/Subnet V2 • Route for Host “A”-Subnet will be advertised by V1 and V2 V3
• Host “Y” will reach either V1 or V2 based on ECMP V1 • From V1 or V2, Host “A” can be Host Y reached via Layer-2 Segment. VLAN 55 Host A VLAN 43 69 VXLAN Hardware VTEP Redundancy (vPC) Southbound Connectivity
• VXLAN vPC Domain Configuration Classic Ethernet
• Configure VXLAN specific vPC Peer-Link Configuration
• Extend the IP Interface (Loopback) 5 configuration for the VTEP V • Secondary IP address (anycast) is V4 used as the anycast VTEP address • Both vPC VTEP switches need to have the identical secondary IP address configured under the loopback interface Host D VNI 30000
70 VXLAN Hardware Gateway Redundancy (vPC) Southbound Connectivity vPC VTEP Configuration Example for (V4-5) # VLAN to VNI mapping (MT-Lite) vlan 55 interface loopback0 vn-segment 30000 ip address 10.10.10.5/32 ip address 10.10.10.99/32 # VTEP IP Interface; Source/Destination for all secondary VXLAN Encapsulated Traffic. § Primary IP address is used for Orphan Hosts § Secondary IP is for vPC Hosts (same IP on both vPC Peers) V5 interface loopback0 Add Secondary IP to VTEP Loopback ip address 10.10.10.V/32 4 ip address 10.10.10.VAnycast/32 secondary V
# VTEP configuration using Loopback as source. Destination Group for VNI 30001 is “239.1.1.2” interface nve1 source-interface loopback0 interface loopback0 host-reachability protocol bgp ip address 10.10.10.4/32 ip address 10.10.10.99/32 member vni 30000 secondary mcast-group 239.239.239.100 Host D VNI 30000 suppress-arp
member vni 50000 associate-vrf 71 VXLAN Hardware Gateway Redundancy (vPC) Not to Forget! vPC VTEP Configuration Example for (V4-5) # VPC Domain Configuration vpc domain 99 peer-gateway needs to be interface loopback0 peer-switch ip address 10.10.10.5/32 peer-keepalive destination V4-mgmt sourceenabled v5-mgmt so that vPC VTEP ip address 10.10.10.99/32 peer-gateway switches can forward traffic secondary ip arp synchronize for each other’s router MAC address # VPC Peer-Link interface port-channelXX V5 switchport mode trunk vpc peer-link V4 # VPC Domain Routing Adjacency Routed Interface (SVI) for routing interface Vlan3999 adjacency across VPC Peer-Link no shutdown ip address 10.254.254.1/30 ip router ospf 1 area 0.0.0.0 interface loopback0 ip ospf network point-to-point ip address 10.10.10.4/32 ip address 10.10.10.99/32 ip pim sparse-mode secondary Host D VNI 30000
72 Folded Clos Topology Providing Topology Symmetry
Spine
L3 Fabric
VXLAN L2/3 VXLAN L2/3 VXLAN L2/3 VXLAN L2/3 VXLAN L2/3 VXLAN L2/3 Gateway Gateway Gateway Gateway Gateway Gateway Leaf Border Leaf
WAN/DCI
• Fully Symmetric, BW rich topology, Optimized for East-West traffic
• Lean Spine does not do any VXLAN termination/gateway
• Access to other networks through border leaf block 73 VXLAN/EVPN Fabric External Routing Overlay
• The Border Leaf/Spine provides Layer-2 and Layer-3 connectivity to external Network
• Flexible routing protocol options for for external routing V2 • Today, VRF-lite allows to extend VRF outside of the fabric V3 BL • With Nexus 7000/7700 and F3, V other options are coming V1
WAN VXLAN/EVPN Fabric External Routing Overlay
VRF for External Routing BL 2 need to exist on Border Leaf VRFVVRF VRF V A B C V3
Interface-Type Options: • Physical Routed Ports V1 • Sub-Interfaces • VLAN SVIs over Trunk Ports Peering Interface can be in Global or Tenant VRF
WAN VXLAN/EVPN Fabric External Routing (eBGP) Overlay
Border Leaf Configuration Example # Sub-Interface Configuration BL interface Ethernet1/1 V 2 no switchport VRF VRF VRF V interface Ethernet1/1.10 A B C mtu 9216 encapsulation dot1q 10 3 vrf member VRF-A V ip address 10.254.254.1/30
# eBGP Configuration router bgp 65500 V1 … vrf VRF-A Advertise external learned routes address-family ipv4 unicast into EVPN (Route-Type 5) advertise l2vpn evpn aggregate-address 10.0.0.0/8 summary-only neighbor 10.254.254.2 remote-as 65599 update-source Ethernet1/1.10 WAN address-family ipv4 unicast * AS# 65599 *Ensure that non-necessary routes are not advertised towards the External Network VXLAN/EVPN Fabric External Routing (eBGP) Overlay
BL 2 VRFVVRF VRF V Edge Router Configuration Example A B C # Interface Configuration V3 interface Ethernet1/1 vrf member VRF-A ip address 10.254.254.2/30 1 # eBGP Configuration V router bgp 65599 … vrf VRF-A address-family ipv4 unicast neighbor 10.254.254.2 remote-as 65500 update-source Ethernet1/1 address-family ipv4 unicast WAN AS# 65599 MPLS IP-VPN & VXLAN – in NX-OS 7.3 L3 Handoff – Border PE P OTVMPLS/IP Transport Core P P
Border Border PE PE VXLAN L3 VXLAN L3 N7K Gateway Gateway
L3 Fabric
VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 VXLAN L3 Gateway Gateway Gateway Gateway Gateway Gateway N5600, N7K, N9K
79 Summary and Conclusion
80 Summary recommendations & takeaways
• Optimize the location of L2 and L3 GWYs to optimize routing and minimize failure exposure
• Leverage L3 VXLAN services enabled by control protocols as the main service and L2 extensions as the exception
• Design the underlay with the VXLAN overlay in mind
• Design the network hierarchically: both the underlay as well as the overlay
• L3 Gateways are key to a sound overlay design
• A combination of pull protocols and push protocols may render optimal scale and resiliency
• Link the provisioning of the overlay and scoping of VNIs to the host orchestration system for optimal scale
81 Thank you
82 We’re ready. Are you?