Implementing VXLAN In a Data Center Lilian Quan, Principle Engineer, INSBU Erum Frahim, Technical Leader, Services Kevin Cook, Solution Architecture

LTRDCT-2223 Agenda

. VxLAN Overview . Flood-&-Learn VXLAN . VXLAN with MP-BGP EVPN Control Plane . VXLAN Design Options . MP-BGP EVPN VXLAN Configuration . Lab Introduction Agenda

. VxLAN Overview . Flood-&-Learn VXLAN . VXLAN with MP-BGP EVPN Control Plane . VXLAN Design Options . MP-BGP EVPN VXLAN Configuration . Lab Introduction Recap – What is VXLAN ?

Tunnel

Ethernet Frames Endpoints

NETWORK OVERLAY Host Host 1 4 IP Addr IP Network IP Addr Host 1.1.1.1 2.2.2.2 Host 2 Switch Switch 5

Host Host

3 IP/UDP Packets 6 PLANE

DATA Outer Outer Outer Outer Outer Outer VXLAN Inner Inner Optiona Original CRC MAC MAC MAC MAC l Inner 802.1Q IP DA IP SA UDP ID CRC DA SA (24 bits) DA SA 802.1Q Payload

VXLAN Encapsulation Original Ethernet Frame

• VXLAN uses MAC in UDP encapsulation (UDP destination port 4789)

5 Why VXLAN? VXLAN provides a Network with Segmentation, IP Mobility, and Scale

• “Standards” based Overlay

• Leverages Layer-3 ECMP – all links forwarding

• Increased Name-Space to 16M identifier

• Segmentation and Multi-Tenancy

• Integration of Physical and Virtual

• It’s SDN 

6 VXLAN VTEP

VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point). Each VTEP has two interfaces, one is to provide bridging function for local hosts, the other has an IP identification in the core network for VXLAN encapsulation/decapsulation.

Underlay Network (IP )

VTEP VTEP IP Interface IP Interface

Local LAN Segment Local LAN Segment

Host Host Host Host

7 VXLAN Underlay Network – IP Routing

• IP routed Network

• Flexible topologies IP Transport Network

• Recommend a network with redundant paths using ECMP for load sharing

• Support any routing protocols --- OSFP, EIGRP, IS-IS, BGP, etc. • All proven best practices for IP routing network apply

8 VXLAN Underlay Network – Typical DC Topologies

Fabric Design 3-Tier Design DC Spine DC Core

DC Aggregation

DC Access DC Leaf DC Interconnect Collapsed Core/Aggregation 2-Tier Design DC-1 DC-2

DC Core/ Aggregation

DC Access WAN

9 VXLAN BUM Traffic Handling

• BUM Traffic --- Multi-destination traffic • Broadcast • Unknown Layer-2 Unicast

• Multicast VTEP-3

• BUM Traffic transport mechanisms Multicast or / Unicast Replication

• Multicast replication in the underlay network IP Network • Requests the underlay network to run IP multicast • Ingress unicast replication VTEP-1 VTEP-2 • One unicast replica per remote VTEP • Increase traffic load throughout the network

10 Agenda

. VxLAN Overview . Flood-&-Learn VXLAN . VXLAN with MP-BGP EVPN Control Plane . VXLAN Design Options . MP-BGP EVPN VXLAN Configuration . VxLAN Capability on Nexus 9000 Series Switches . Lab Introduction

Flood-&-Learn VxLAN End System End . No VXLAN control plane System End

. Data driven flood-&-learn

VTEP

IP 3 VTEP

-

3

- 3

Multicast/Unicast Replication VTEP-1 VTEP-2 IP Network VTEP 1 VTEP 2 End System A End System B IP-1 IP-2 MAC-A MAC-B IP-A IP-B

12 Challenges with Flood-&-Learn VXLAN Deployments Scale, Mobility and Security Limitations

VXLAN Overlay

VTEP VTEP VTEP VTEP VTEP VTEP VTEP

LIMITED SCALE LIMITED WORKLOAD MOBILITY Security Risk • Flood and learn (BUM)- • Centralized Gateways – Traffic • No authentication for VXLAN Inefficient Bandwidth Utilization Hair-pining devices (VTEPs) • Resource Intensive – Large • Sub-Optimal Traffic Flow MAC Tables

Barrier for Scaling out Large Data Centers and Cloud Deployments13 Agenda

. VxLAN Overview . Flood-&-Learn VXLAN . VXLAN with MP-BGP EVPN Control Plane . VXLAN Design Options . MP-BGP EVPN VXLAN Configuration . VxLAN Capability on Nexus 9000 Series Switches . Lab Introduction What is VXLAN/EVPN? • Standards based Overlay (VXLAN) with Standards based Control-Plane (BGP)

• Layer-2 MAC and Layer-3 IP information distribution by Control-Plane (BGP)

• Forwarding decision based on Control-Plane (minimizes flooding)

• Integrated Routing/Bridging (IRB) for Optimized Forwarding in the Overlay

Control- EVPN MP-BGP - RFC 7432 Plane (draft-ietf-l2vpn-evpn)

Provider Backbone Bridges Data- Multi-Protocol Label Switching (MPLS) Network Virtualization Overlay (NVO) (PBB) Plane draft-ietf-l2vpn-evpn draft-sd-l2vpn-evpn-overlay draft-ietf-l2vpn-pbb-evpn

 EVPN over NVO Tunnels (VXLAN, NVGRE, MPLSoE) for Data Center Fabric encapsulations  Provides Layer-2 and Layer-3 Overlays over simple IP Networks

15 Cisco Supported VXLAN IETF Drafts

ID Title Ver Category Comments

RFC 7348 Virtual eXtensible Local Area Data Plane Flood and Learn based implemented Network RFC 7432 BGP MPLS based Ethernet VPNs EVPN Control Plane Control Plane is based on this RFC draft-ietf-bess-evpn-overlay A Network Virtualization Overlay 0 EVPN Control Plane Implemented Solution using EVPN draft-ietf-bess-evpn-inter- Integrated Routing and Bridging in 0 EVPN Control Plane Implemented subnet-forwarding EVPN draft-rabadan-l2vpn-evpn- IP Prefix Advertisement in EVPN 2 EVPN Control Plane Implemented evpn-prefix-advertisement

Draft-tissa-nvo3-oam-fm NVO3 Fault Management 1 Mgmt Plane Working in progress

16 Functions of VXLAN/EVPN

Host/Network Advertise host/network reachability Reachability information through control protocol (MP- Advertisement BGP) VTEP Security & Authenticate VTEPs through BGP peer Authentication authentication Distributed Anycast Gateway Seamless and Optimal vm-mobility

Early ARP termination ARP Suppression Localize ARP learning process Minimize network flooding Dynamic Ingress Replication Unicast Alternative to Multicast underlay

17 EVPN Primer --- MP-BGP Review

Virtual Routing and Forwarding (VRF) Layer-3 segmentation for tenants’ routing space

BGP advertisement: VPN-IPv4 Addr = RD:16.1/16 Route Distinguisher (RD): BGP Next-Hop = PE1 8-byte field, VRF parameters; unique value to make Route Target = 100:1 eBGP: Label=42 eBGP: VPN IP routes unique: RD + VPN IP prefix 16.1/16 16.1/16 IP Subnet IP Subnet

PE1 P P PE2 Selective distribute VPN routes: CE1 Blue VPN CE2 Route Target (RT): 8-byte field, VRF parameter, unique value to define the import/export rules for ipVRFvrfparameters:blue-vpn VPNv4 routes NameRD 1:100 = blue-vpn RDroute = 1:100-target export 1:100 Importroute- targetRoute import-Target 1:100 = 100:1 Export Route-Target = 100:1 VPN Address-Family: Distribute the MP-BGP VPN routes

18 MP-BGP for VXLAN EVPN Control Plane EVPN Control Plane – Reachability Distribution • EVPN Control Plane -- Host and Subnet Route Distribution

BGP Update Spine • Host-MAC • Host-IP • Internal IP Subnet • External Prefixes

VTEP VTEP VTEP VTEP Leaf

. Use MP-BGP with EVPN Address Family on leaf nodes to distribute internal host MAC/IP addresses, subnet routes and external reachability information . MP-BGP enhancements to carry up to 100s of thousands of routes with reduced convergence time

19 EVPN Control Plane -- Host Advertisement

Install host info to RIB/FIB: Install host info to RIB/FIB: H-MAC-1  MAC table BGP Update: H-MAC-1  MAC table BGP Update: H-IP-1  VRF IP host table 4 H-MAC-1 H-MAC-1 4 H-IP-1  VRF IP host table H-IP-1 Host VNI VTEP H-IP-1 3 Route IP VTEP-1 MAC Host VNI VTEP VTEP-1 3 Reflector VNI-1 IP MAC-1 H-IP-1 VNII-1 VTEP-1 VNI-1 H-MAC-1 H-IP-1 VNII-1 VTEP-1 2 VTEP-2 VTEP-3 BGP Update: H-MAC-1 H-IP-1 VTEP-1 VNI-1

1 VTEP-1

MAC Host VNI VTEP IP

H-MAC-1 H-IP-1 VNII-1 VTEP-1 Local learning of host info: H-MAC-1 H-IP-1 H-MAC-1 (MAC table) VLAN-1 /VNI-1 H-IP-1 (VRF IP host table ) 20 VXLAN BGP Control Plane EVPN Control Plane --- Host Movement

NLRI: Spine • Host H-MAC-1, H-IP-1 • NVE VTEP-1 • VNI 5000

Ext. Community:

• Encapsulation: VXLAN VTEP-1 VTEP-2 VTEP-3 VTEP-4 Leaf • Cost • Sequence number :0 Host 1 H-MAC-1 H-IP-1 VLAN 10 VXLAN 5000 1. Host 1 attaches to VTEP-1

2. VTEP-1 detects Host1 and advertises H1 with seq #0

3. Other VTEPs learn about the host route of Host 1 MAC IP VNI Next-Hop Encap Seq# H-MAC-1 H-IP-1 5000 VTEP-1 VXLAN 0

21 VXLAN BGP Control Plane EVPN Control Plane --- Host Movement

NLRI: Spine • Host H-MAC-1, H-IP-1 • NVE VTEP-3 • VNI 5000

Ext. Community:

• Encapsulation: VXLAN VTEP-1 VTEP-2 VTEP-3 VTEP-4 Leaf • Cost • Sequence number: 1 Host 1 H-MAC-1 H-IP-1 VLAN 10 VXLAN 5000 1. Host 1 moves to VTEP-3 from VTEP-1

2. VTEP-3 detects Host 1, sends MP-BGP update for Host 1 with its own VTEP address and a new seq #1

3. Other VTEPs learn about the new route of Host 1 MAC IP VNI Next-Hop Encap Seq# H-MAC-1 H-IP-1 5000 VTEP-3 VXLAN 1

22 VXLAN Routing VXLAN EVPN has two slightly different integrated Route/Bridge (IRB) semantics

Routing ? IP Transport Network SVI A SVI B VTEP-1 VTEP-2 VTEP-3 VTEP-4

Host 1 Host 2 H-MAC-1 H-MAC-2 H-IP-1 H-IP-2 VNI-A VNI-B

Asymmetric Symmetric • Routing on the ingress VTEP and bridging • Routing on both the ingress and the egress on the egress VTEP VTEPs • Requires each VTEP to have all VNIs – • A VTEP only needs to have VNIs in which can result in forwarding table resource they have local hosts. Optimal utilization of wastage. forwarding table resources

• Cisco follows Symmetric IRB 23 Symmetric Integrated Routing & Bridging (IRB) (1)

• Routing on both ingress and egress VTEPs • Layer-3 VNI Layer-3 VNI (VRF VNI) • Tenant VPN indicator • One per tenant VRF • VTEP MAC Layer-2 VNI Layer-2 VNI (Network VNI) (Network VNI) • Ingress VTEP routes packets onto the Layer-3 VNI VTEP VTEP VTEP VTEP • Egress VTEP routes packets to the destination Layer-2 VNI

24 Symmetric Integrated Routing & Bridging (IRB) (2)

Egress VTEP routes packets Ingress VTEP S-IP: VTEP-1 from L3 VNI to the routes packets D-IP: VTEP-4 destination VNI- VNI: L3 VNI from source VNI-A 1 2 B/VLAN. to L3 VNI. D-MAC S-MAC: Router-MAC-1 in the inner header D-MAC: Router-MAC-4 is the egress S-IP: H-IP-1 D-IP: H-IP-2 VTEP router MAC S-MAC: MAC-VTEP4 local port D-MAC: H-MAC-2 VNI L3 L3 VNI S-IP: H-IP-1 A VNI VNI B D-IP: H-IP-2

VTEP-1 VTEP-4 VTEP-2 VTEP-3 Router MAC-1 Router MAC-4

S-MAC: H-MAC-1 D-MAC: GW-MAC Host 1 Host 2 S-IP: H-IP-1 H-MAC-1 H-MAC-2 D-IP: H-IP-2 H-IP-1 H-IP-2 VNI-A VNI-B

25 EVPN Control Plane --- ARP Suppression

Minimize flood-&-learn behavior for host learning

MAC IP VNI Next-Hop Encap Seq Spine

H-MAC-2 H-IP-2 5000 VTEP-3 VXLAN 0

2 VTEP VTEP VTEP VTEP VTEP-1 receives and intercepts the ARP 1 2 3 4 Leaf Request. Checks in its own host table.

• If it has an match for H-IP-2, it’ll send ARP Host 1 Host 2 response on behave of Host-2 H-MAC1 H-MAC-2 H-IP 1 H-IP-2 • If it doesn’t have a match for H-IP-2, it’ll VLAN 10 VLAN 10 forward the ARP request to remote VTEPs VXLAN 5000 VXLAN 5000 via multicast encap or head-end replication

1 Host-1 sends ARP Request for H-IP-2

26 EVPN Control Plane -- Head-end Replication Head-end Replication (aka. Ingress replication): Eliminate the need for underlay multicast to transport overlay BUM traffic

Multicast-Free Spine Underlay

VTEP VTEP VTEP VTEP 1 2 2 3 4 Leaf VTEP-1 receives the overlay BUM traffic, encapsulates the packets into unicast VXLAN packets, sends one copy to each remote VTEP peer in the same VXLAN VNI

1 Host-1 sends BUM traffic into the VXLAN VNI

27 Distributed Anycast Gateway in MP-BGP EVPN

# VLAN to VNI mapping vlan 200 vn-segment 5200

# Anycast Gateway MAC, identically configured on all VTEPs fabric forwarding anycast-gateway-mac 0002.0002.0002

# Distributed IP Anycast Gateway (SVI) The same anycast gateway virtual IP # Gateway IP address needs to be identically configured on all address and MAC address are VTEPs configured on all VTEPs in the VNI. interface vlan 200 no shutdown vrf member Tenant-A ip address 20.0.0.1/24 fabric forwarding mode anycast-gateway

SVI SVI SVI SVI GW IP GW IP GW IP GW IP GW MAC GW MAC GW MAC GW MAC VTEP VTEP VTEP VTEP

Host 1 Host 2 Host 3 Host 4 MAC1 MAC2 MAC3 MAC4 IP 1 IP 2 IP 3 IP 4 VLAN A VLAN A VLAN A VLAN A VXLAN A VXLAN A VXLAN A VXLAN A 28 Next-Gen VXLAN Fabric with BGP-EVPN Control Plane Multi-Tenancy, Seamless Host Mobility and Security at Cloud Scale

Route Route Reflector Reflector

BGP-EVPN VXLAN Overlay

BGP Peers VTEP VTEP VTEP VTEP VTEP VTEP VTEP

INTEROPERABLE INCREASED SCALE OPTIMIZED OPERATIONAL SECURITY • Standards Based • Eliminates MOBILITY FLEXIBILITY • VTEP peer • BGP-EVPN Flooding • Distributed • Layer 2 or Layer 3 authentication via Anycast Gwy BGP • VXLAN • Conversational • Controller Choice Learning • Integrated • White-list for • Policy-Based Routing /Bridging unauthenticated Updates • vPC & ECMP VTEPs

Breaking the Traditional VXLAN Scale Barriers 29 Agenda

. VxLAN Overview . Flood-&-Learn VXLAN . VXLAN with MP-BGP EVPN Control Plane . VXLAN Design Options . MP-BGP EVPN VXLAN Configuration . VxLAN Capability on Nexus 9000 Series Switches . Lab Introduction VXLAN Inter-PoD Extension L3 Core L2 Link L3 Link

VTEP VTEP

VXLAN Overlay (VLAN Extension)

Pod 1 Pod 2 IP GW IP GW

Layer-2 VLAN Domain Layer-2 VLAN Domain

31 VXLAN Fabric Design with MP-iBGP EVPN

RR RR Spine MP-iBGP Sessions

VXLAN Overlay MP-iBGP EVPN

VTEP VTEP VTEP VTEP VTEP VTEP Leaf

• VTEP Functions are on leaf layer • Spine nodes are iBGP route reflector • Spine nodes don’t need to be VTEP

32 VXLAN Fabric Design with MP-eBGP EVPN

AS 65000 Spine

MP-eBGP Sessions

VTEP VTEP VTEP VTEP VTEP VTEP Leaf

AS 65001 AS 65002 AS 65003 AS 65004 AS 65005 AS 65006

• VTEP Functions are on leaf layer • Spine nodes are MP-eBGP Peers to VTEP leafs • Spine nodes don’t need to be VTEP • VTEP leafs can be in the same or different BGP AS’s 33 EVPN VXLAN Fabric External Routing

VXLAN Overlay Spine EVPN VRF/VRFs Space

VXLAN Overlay EVPN MP-BGP Border Leaf

VTEP VTEP VTEP VTEP VTEP VTEP Leaf

Routing Protocol of Choice Global Default VRF Or User Space VRFs

IP Routing

34 EVPN VXLAN External Routing with BGP Sample Configuration Router bgp 100 vrf evpn-tenant-1 address-family ipv4 unicast network 20.0.0.0/24 neighbor 30.10.1.2 remote-as 200 RR RR address-family ipv4 unicast Spine prefix-list outbound-no-hosts out VXLAN Overlay Border Leaf interface Ethernet2/9.10 EVPN VRF Instance Space mtu 9216 VTE VTE VTE VTE VTEP P P P P encapsulation dot1q 10 vrf member evpn-tenant-1 ip address 30.10.1.1/30

interface Ethernet1/50.10 mtu 9216 encapsulation dot1q 10 ip address 30.10.1.2/30

IP Routing in the Default router bgp 200 VRF Instance address-family ipv4 unicast network 100.0.0.0/24 network 100.0.1.0/24 neighbor 30.10.1.1 remote-as 100 address-family ipv4 unicast 35 DCI with VXLAN EVPN (Option A)

VXLAN Overlay

RR RR RR EVPNRR VRF/VRFs Space

DC #1 DC #2

EVPN iBGP Border Leaf Border Leaf EVPN iBGP

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

EVPN Domain #2 EVPN Domain #1

VTEP VTEP VLAN hand-off VLAN hand-off Flood-&-Learn Flood-&-Learn Inter-DC EVPN OVT/VPLS

Inter-DC EVPN Domain

36 DCI with VXLAN EVPN (Option B)

VXLAN Overlay

RR RR RR EVPNRR VRF/VRFs Space

DC #1 DC #2 DCI Border Leaf EVPN iBGP DCI Border Leaf EVPN iBGP

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Inter-DC EVPN eBGP (multi-hop)

Global Default VRF Or User Space VRFs

One EVPN Administrative Domain Stretched Across Two Data Centers

37 Agenda

. VxLAN Overview . Flood-&-Learn VXLAN . VXLAN with MP-BGP EVPN Control Plane . VXLAN Design Options . MP-BGP EVPN VXLAN Configuration . Lab Introduction

38 Building your VTEP (VXLAN Tunnel End-Point)

# Features & Globals Enables VTEP (only required on Leaf or Border) feature bgp feature nv overlay nv overlay evpn Enables EVPN Control-Plane in BGP # Spine (S1)

# Leaf (V1) Configure the VTEP interface interface nve1 source-interface loopback0 RR RR RR RR host-reachability protocol bgp Use a Loopback for Source Interface

iBGP 2 1 V Enable BGP for Host reachabilityV

V3 *Simplified BGP configuration; would have 4 BGP peers (RR) IGP not shown 39 Building your EVPN MP-BGP Control-Plane # Features & Globals feature bgp feature nv overlay nv overlay evpn Enables EVPN Control-Plane in BGP # Spine (S1) router bgp 65500 router-id 10.10.10.1 address-family ipv4 unicast address-family l2vpn evpn Activate L2VPN EVPN under each BGP neighborRR RR RR RR neighbor 10.10.10.0/24 remote-as 65500 update-source loopback0 address-family l2vpn evpn iBGP send-community both 2 V1 V route-reflector-client

# Leaf (V1) router bgp 65500 Send Extended BGP Community router-id 10.10.10.10 to distribute EVPN route attributes address-family ipv4 unicast V3 neighbor 10.10.10.1 remote-as 65500 update-source loopback0 address-family l2vpn evpn send-community both * 40 Extend your VLAN to VXLAN

• VLAN to VNI configuration on a per-Switch based • VLAN becomes “Switch Local Identifier” # Features feature vn-segment-vlan-based • VNI becomes “Network Global Identifier” # VLAN to VNI mapping (MT-Lite) • 4k VLAN limitation per-Switch does still apply Vlan 10 VLAN to Layer-2 VNI mapping • 4k Network limitation has been removed vn-segment 5010 • VLAN can be port-significant. The same vlan on # Activate Layer-2 VNI for EVPN evpn different ports can be mapping to different VNIs. vni 5010 l2 Enables EVPN Control- rd auto Plane for Layer-2 route-target import auto Services route-target export auto Alternative is to use VLAN # Activate Layer-2 VNI on VTEP “ingress-replication interface nve1 protocol bgp” ethernet VLAN VNI vxlan source-interface loopback0 host-reachability protocol bgp

Multi-Tenancy Lite (MT-Lite) member vni 5010 mcast-group 239.239.239.100 suppress-arp Enables Layer-2 VNI on VTEP and suppress41 ARP Distributed Anycast Gateway for Extended VLANs

• All VTEPs in a VXLAN are the distributed anycast gateway for its IP subnet. • All VTEPs in a VXLAN need to be configured with an identical anycast gateway virtual MAC address # VLAN to VNI mapping vlan 200 • All VTEPs in a VXLAN need to be configured with vn-segment 5200

an identical anycast gateway virtual IP address # Anycast Gateway MAC, identically configured on all VTEPs fabric forwarding anycast-gateway-mac 0002.0002.0002

# Distributed IP Anycast Gateway (SVI) One gateway virtual MAC per VTEP # Gateway IP address needs to be identically configured on all VTEPs interface vlan 200 no shutdown vrf member Tenant-A ip address 20.0.0.1/24 One gateway virtual IP per VLAN/VXLAN fabric forwarding mode anycast-gateway

42 Routing in VXLAN – Define the Resources

Configuration Example for VRF-A

vxlan # Define VLAN for VRF routing instance Vlan 999 vn-segment 9999 VLAN to Layer-3 VNI mapping

# Define SVI for VRF routing instance interface Vlan999 VNI50000 VLAN to Layer-3 VNI mapping no shutdown mtu 9216 - ip forward required for prefix- vrf member VRF-A based routing ip forward VRF-A # VRF configuration for “customer” VRF vrf context VRF-A vni 9999 VRF context definition rd auto ethernet ethernet - VNI address-family ipv4 unicast route-target both auto - Route-Distinguisher route-target both auto evpn - Route-Targets - IPv4 and/or IPv6

43 Routing in VXLAN – Configure the Routing

Enables Layer-3 VNI on VTEP Configuration Example for VRF-A

and associatevxlan it to VRF # Activate Layer-3 VNI on VTEP interface nve1 source-interface loopback0 host-reachability protocol bgp member vni 5010 mcast-group 239.239.239.100 VNI50000 suppress-arp member vni 9999 associate-vrf

# Route-Map for Redistribute Subnet VRF-A route-map REDIST-SUBNET permit 10 match tag 12345

# Control-Plane configuration for VRF (Tenant) router bgp 65500 ethernet ethernet … vrf VRF-A address-family ipv4 unicast VRF/Tenant definition advertise l2vpn evpn within Overlay Control-Plane redistribute direct route-map REDIST-SUBNET maximum-paths ibgp 2 44 VXLAN Hardware Gateway Redundancy (vPC)

• Redundant connectivity for classic Ethernet hosts

• Extend the IP Interface (Loopback) configuration for the vPC VTEP V5 • Secondary IP address (anycast) is used as the anycast VTEP address V4 • Both vPC VTEP switches need to have the identical secondary IP address configured under the loopback interface Host D VNI 30000

45 VXLAN Hardware Gateway Redundancy (vPC) vPC VTEP Configuration Example

# VLAN to VNI mapping (MT-Lite) vlan 55 interface loopback0 vn-segment 30000 ip address 10.10.10.5/32 # VTEP IP Interface; Source/Destination for all ip address 10.10.10.99/32 secondary VXLAN Encapsulated Traffic. . Primary IP address is used for Orphan Hosts . Secondary IP is for vPC Hosts (same IP on both vPC Peers) interface loopback0 V5 ip address 10.10.10.5/32 Add Secondary IP to VTEP Loopback. ip address 10.10.10.99/32 secondary V4 VXLAN automatically picks up # VTEP configuration using Loopback as source. the secondary IP address as interface nve1 the VTEP address source-interface loopback0 host-reachability protocol bgp interface loopback0 member vni 5010 ip address 10.10.10.4/32 mcast-group 239.239.239.100 ip address 10.10.10.99/32 secondary suppress-arp Host D member vni 9999 associate-vrf VNI 30000

46 VXLAN Hardware Gateway Redundancy (vPC) vPC VTEP Configuration Example # VPC Domain Configuration vpc domain 99 peer-gateway needs to be interface loopback0 peer-switch enabled so that vPC VTEP ip address 10.10.10.5/32 peer-keepalive destination V4-mgmt source v5-mgmt ip address 10.10.10.99/32 secondary peer-gateway switches can forward traffic ip arp synchronize for each other’s router MAC address # VPC Peer-Link interface port-channelXX 5 switchport mode trunk V vpc peer-link V4 # VPC Domain Routing Adjacency Routed Interface (SVI) for routing interface Vlan3999 no shutdown adjacency across VPC Peer-Link ip address 10.254.254.1/30 ip router ospf 1 area 0.0.0.0 interface loopback0 ip ospf network point-to-point ip address 10.10.10.4/32 ip address 10.10.10.99/32 secondary ip pim sparse-mode Host D VNI 30000

47 EVPN Configuration for Stretched Fabric

VXLAN Overlay

RR RR RR EVPNRR VRF/VRFs Space

DC #1 DC #2 DCI Border Leaf EVPN iBGP DCI Border Leaf EVPN iBGP

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP

Inter-DC EVPN eBGP (multi-hop)

Global Default VRF Or User Space VRFs

One EVPN Administrative Domain Stretched Across Two Data Centers

48 EVPN Configuration for Stretched Fabric

VXLAN Overlay

RR RR RR EVPNRR VRF/VRFs Space

DC #1 DC #2 EVPNiBGP iBGP EVPNiBGP iBGP DCI Border Leaf DCI Border Leaf

VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP VTEP Multi-hop Inter-DC EVPN eBGPeBGP(multi-hop)

Global Default VRF Or User Space VRFs

One EVPN Administrative Domain Stretched Across Two Data Centers

49 eBGP EVPN Configuration (1) Next-hop Unchange

• BGP next-hop is used as the tunnel tail end address. It shall be the advertising VTEP’s address. eBGP configuration on a spine switch • Ensure the next-hop in the BGP route route-map permit-all permit 10 isn’t changed during the route distribution route-map nh-unchange permit 10 set ip next-hop unchanged • eBGP changes next-hop to by default. router bgp 65000 Need to change the policy to next-hop router-id 10.1.1.1 unchanged address-family ipv4 unicast address-family l2vpn evpn nexthop route-map nh-unchange retain route-target all neighbor 192.167.11.2 remote-as 65001 Set next-hop policy not to change address-family ipv4 unicast the next-hop attribute address-family l2vpn evpn send-community extended route-map permit-all out

50 eBGP EVPN Configuration(2) Manually configure import/export route-target

• With eBPG, VTEPs will have different route-targets if using auto RT generation vrf context evpn-tenant-1 • Need to manually configure RTs on vni 9999 rd auto eBGP peers so that they have the same address-family ipv4 unicast RTs route-target import 100:9999 route-target import 100:9999 evpn route-target export 100:9999 route-target export 100:9999evpn Manually configure route-target for evpn vni 5010 l2 VRF rd auto route-target import 100:5010 route-target export 100:5010

Manually configure route-target for L2 VNI under EVPN

51 Agenda

. VxLAN Overview . Flood-&-Learn VXLAN . VXLAN with MP-BGP EVPN Control Plane . VXLAN Design Options . MP-BGP EVPN VXLAN Configuration . Lab Introduction Lab Overview Module Details

• Module 1 – Network Based Overlay DC1: In this module, students will configure a network based overlay with Nexus 9000 switches and use them as a VTEP. Students will also learn how to extend the Anycast gateway using BGP EVPN.

• Module 2 – FW-Security Zone : In this module, students will create the secure zone via placing the FW ( in transparent mode) between the Fabric and External Connectivity.

• Module 3- Stretch Fabric: In this module, students will be able to extend the VLANs from the first DC to the second DC and able to stretch the fabric.

53 Lab Topology

54 Network Overlay DC1 Using BGP EVPN for Control Plane

• Using OSPF for underlay

• IBGP-for overlay-VXLAN EVPN control plane- Route Reflectors are on Spines S11 and S12

• Ingress Replication for the BUM Traffic

• Each student has their own VRF that will be representative of multi- tenancy. Security Module Creating the Security Zone via VXLAN

• The Hosts in the Secured Zone are in VXLAN X53.

• Transparent FW is attached to Edge E11-( No redundancy in this lab).

56 Firewall Usecase Flow

57 DC2- Stretch Fabric Using EBGP EVPN for Underlay/Overlay

• Using EBGP for Underlay/Overlay

• Ingress Replication for the BUM Traffic.

• One of the VLAN X51 will be stretch from DC1 to DC2.

• Each student has their own VRF that will be representative of multi-tenancy.

58 Manual Overview • Manual available at https://cisco.box.com/LTRDCT-2223-Manual

• Password will be provided by proctor

• Any time you see text highlighted in cyan, change the numbers to your pod number – Pod2 is always used as a sample

• Change XX to your two digit pod number (Pod1 = 01, Pod10 = 10)

• Change X to your single/two digit pod number (Pod1 = 1, Pod10 = 10)

• Only type commands that are shown in a box. Commands shown under “Configuration Sample” are not meant to be typed into the devices. Complete Your Online Session Evaluation

• Please complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt.

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Thank you