IP Unnumbered

Building Simplified, Automated and Scalable Data Center Networks with Overlays (VXLAN/FabricPath)

BRKDCT-3378

Lukas Krattiger, Technical Marketing Engineer [email protected] @CCIE21921 Session Objectives

• Focus on Data Center Networks and Fabrics with Overlays • Closer Look on Packet Encapsulation (VXLAN) – Underlay – the Transport for the Overlay – Control-Plane – Exchanging Information – Optimizing the Forwarding • Overview on Frame Encapsulation (FabricPath) – Similarities to Packet Encapsulation • Fabric Management & Automation approaches

• Deep-Dive into FabricPath – There are many Sessions and Recordings • Comparison between different Orchestration and Management Tools • Automation Workflows or Services Catalogs

• Born behind the Mountains • Other People call it Switzerland

• Living in the land of E-Cars and Startups • Aka San Francisco Bay Area

• Not sure if I’m 100% Human • US Immigration calls me “Legal Alien”

• Dad, Husband, Geek

6 Agenda

• Data Center Fabric Properties • Optimized Networking with VXLAN – Overview – Underlay – Control & Data Plane – Multi-Tenancy • Optimized Networking with FabricPath • Fabric Management & Automation

MAN/WA N

FabricPath VXLAN /BGP /EVPN

MAN/WA MAN/WA N N

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public Data Center Fabric Properties Data Center Fabric Properties Scale, Resiliency, Efficiency • Any subnet, anywhere, rapidly • Reduced Failure Domains • Extensible Scale & Resiliency • Profile Controlled Configuration

 Full Bi-Sectional Bandwidth (N Spines)

 Any/All Leaf Distributed Default Gateways

 Any/All Subnets on Any Leaf

• Wide ECMP: Unicast or Multicast RR RR • Uniform Reachability, Deterministic Latency • High Redundancy: Node/Link Failure • Line rate, low latency, for all traffic

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 Variety of Fabric Sizes Data Center Fabric Properties • Fabric size: Hundreds to 10s of Thousands of 10G ports RR RR • Variety of Building Blocks: – Varying Size – Varying Capacity – Desired oversubscription – Modular and Fixed • Scale Out Architecture – Add compute, service, external connectivity as the demand grows

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 Variety of South-bound Topological Connectivity Data Center Fabric Properties • Flexible connectivity options to the Leaf nodes – FEX in straight-through or dual-active RR RR mode (eVPC) – Blade Switches – Hypervisors or bare-metal servers attached in vPC mode

Virtual Switch

Extended Namespace Scalable Layer-2 Domains Integrated Route and Bridge Hybrid Overlays Inter-Pod connectivity Multi-Tenancy

Classic Ethernet • Traditionally VLAN is expressed DMAC SMAC 802.1Q Etype Payload CRC over 12 bits (802.1Q tag) Frame – Limits the maximum number of

segments in a Data Center to 4096 Destination MAC (DMAC)

VLANs Source MAC (SMAC)

TPID TCI 4 bytes 802.1Q 0x8100 PCP CFI VID (16 bits) (3 bits) (1 bits) (12 bits)

Ether Type (Etype)

Data (Payload)

CRC/FCS

VLAN ID 12 bits

TPID = Tag Protocol Identifier, TCI = Tag Control Information, PCP = Priority Code Point,

Classic Ethernet • Cisco FabricPath introduces Classic DMAC SMAC 802.1Q Etype Payload CRC Ethernet Frame Encapsulation by Frame adding 16 bytes

• Traditionally VLAN, expressed over 16 bytes IEEE 802.1Q tag, stays the same Cisco Outer Outer FP CRC DA SA Tag DMAC SMAC 802.1Q Etype Payload FabricPath (new) Frame (48) (48) (32)

Classic Ethernet • Cisco FabricPath introduces Classic DMAC SMAC 802.1Q Etype Payload CRC Ethernet Frame Encapsulation by Frame adding 16 bytes

• Traditionally VLAN, expressed over 16 bytes IEEE 802.1Q tag, stays the same Cisco Outer Outer FP FabricPath+Segm Segment CRC DMAC SMAC Etype Payload DA SA Tag ID ent-ID (48) (48) (32) (new) Frame • FabricPath w/Segment-ID leverages a two 802.1Q tags for a total address space of 24 bits 802.1Q VLAN ID (12 bits) Segment-ID 24 bits – Support of ~16M segments 802.1Q VLAN ID (12 bits)

TPIDBRKDCT 0x8100-3378 = VLAN© 2015 Tagged Cisco and/or Frame its affiliates. with All IEEErights reserved. 802.1Q /Cisco TPID Public 0x893B19 = TRILL Fine Grained Labeling Overview Introducing FabricFath w/Segment-ID (2)

Classic Ethernet • Cisco FabricPath introduces Classic DMAC SMAC 802.1Q Etype Payload CRC Ethernet Frame Encapsulation by Frame adding 16 bytes

• Traditionally VLAN, expressed over 16 bytes IEEE 802.1Q tag, stays the same Cisco Outer Outer FP FabricPath+Segm Segment CRC DMAC SMAC Etype Payload DA SA Tag ID ent-ID (48) (48) (32) (new) Frame

• FabricPath w/Segment-ID leverages TCI outer TPID a double 802.1Q tag for a total 4 bytes 0x8100 PCP CFI VID 802.1Q (16 bits) (3 bits) (1 bit) (12 bits) address space of 24 bits Inner TPID TCI – Support of ~16M segments 4 bytes 0x893B PCP CFI VID 802.1Q (16 bits) (3 bits) (1 bit) (12 bits)

TPIDBRKDCT 0x8100-3378 = VLAN© 2015 Tagged Cisco and/or Frame its affiliates. with All IEEErights reserved. 802.1Q /Cisco TPID Public 0x893B20 = TRILL Fine Grained Labeling Overview Introducing VXLAN • Traditionally VLAN is expressed over 12 bits (802.1Q tag) – Limits the maximum number of segments in a Data Center to 4096 VLANs

Classical Ethernet Frame DMAC SMAC 802.1Q Etype Payload CRC

• VXLAN leverages the VNI field with 50 bytes Original CE Frame

Outer a total address space of 24 bits VXLAN IP UDP VxLAN 802.1Q CRC MAC DMAC SMAC Etype Payload (20) (8) (8) optional (new) Frame (14) – Support of ~16M segments Cisco DFA Frame 8 bits 24 bits 24 bits 8 bits • The VXLAN Network Identifier ags Reserved VNIVNI Reserved (VNI/VNID) is part of the VXLAN Header VNI

Src VTEP MAC Address VXLAN Frame Format Dest. MAC Address 48 MAC-in-IP Encapsulation Src. MAC Address 48 VLAN Type 14 Bytes 16 IP Header 0x8100 (4 Bytes Optional) 72 Misc. Data VLAN ID 16 Tag Outer MAC Header Protocol 0x11 (UDP) 8 Ether Type 16 Header 0x0800 16 20 Bytes Checksum

Source IP 32 Outer IP Header Src and Dst Underlay addresses of the Source 16 Dest. IP 32 VTEPs Port UDP Header VXLAN Port 16 Hash of the inner L2/L3/L4 headers of 8 Bytes the original frame. UDP Length 16 Enables entropy for ECMP Load balancing in the Network. VXLAN Header UDP 4789

Checksum 0x0000 16 50 (54) Bytes of Overhead of Bytes (54) 50 VXLAN Flags 8 RRRRIRRR Allows for 16M Original Layer-2 Frame Overlay possible Reserved 24 Segments 8 Bytes VNI 24

Reserved 8

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 Optimized Networks with VXLAN Data Center Fabric Properties Extended Namespace Scalable Layer-2 Domains Integrated Route and Bridge Multi-Tenancy

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 Why VXLAN? VXLAN provides a Network with Segmentation, IP Mobility, and Scale • “Standards” based Overlay • Leverages Layer-3 ECMP – all links forwarding • Increased Name-Space to 16M identifier • Integration of Physical and Virtual • It’s SDN 

Edge Device Edge Device

Local LAN Local LAN IP Interface Segment Segment

Physical Physical Host Edge Device Local LAN Host Segment

Virtual Switch

VTEP VTEP

V V Local LAN Local LAN Segment Segment Encapsulation

Physical Physical Host VTEP V Local LAN Host Segment

Session ID Session Title Speaker Day / Time

LTRDCT-1224 Implementing VXLAN in Datacenter Lilian Quan Tuesday, 9:30 Technical Marketing Engineer Wednesday, 9:00 BRKAPP-9004 Data Center Mobility, VXLAN & ACI Fabric Architecture Bradley Wong Tuesday, 11:15 Principal Engineer PNLDCT-2001 Panel: Overlays in the Data Center - A Customer Perspective Panel Tuesday, 4:45p

BRKDCT-2328 Evolution of Network Overlays in Data Center Clouds Victor Moreno Wednesday, 11:30 Distinguished TME BRKDCT-2456 Building Multi-tenant DC using Cisco Nexus Switching Elyor Khakimov Wednesday, 2:30p Technical Marketing Engineer BRKDCT-2404 VXLAN deployment models - A practical perspective Victor Moreno Thursday, 9:00 Distinguished TME BRKACI-2001 Integration and Interoperation of existing Nexus networks into Mike Herbert Thursday, 11:30 an ACI architecture Principal Engineer BRKDCT-2049 Overlay Transport Virtualization Brian Farnham Thursday, 2:30p Technical Marketing Engineer

MAC VNI VTEP MAC VNI VTEP MAC_A 30000 E1/12 MAC_B 30000 E1/4

2 V1 V

MAC VNI VTEP MAC_C 30000 E1/8

Host A 3 MAC_A / IP_A V Host B MAC_B / IP_B

Virtual Switch

MAC VNI VTEP MAC VNI VTEP MAC_A 30000 E1/12 MAC_B 30000 E1/4 3 MAC_A 30000 IP_V1

ARP Request for IP_B Src MAC: MAC_A 2 ARP Request for IP_B Dst MAC: FF:FF:FF:FF:FF:FF 1 V Src MAC: MAC_A V Dst MAC: FF:FF:FF:FF:FF:FF 1 3 2 4 SMAC: MAC_V1 MAC VNI VTEP DMAC: 00:01:5E:01:01:01 MAC_C 30000 E1/8 SIP: IP_V1 DIP: 239.1.1.1 Underlay MAC_A 30000 IP_V1 Host A UDP 3 MAC_A / IP_A V Host B MAC_B / IP_B VXLAN VNID: 30000 ARP Request for IP_B Src MAC: MAC_A

ARP Request 4 Dst MAC: FF:FF:FF:FF:FF:FF Overlay SMAC: MAC_A Virtual Switch DMAC: FF:FF:FF:FF:FF:FF

MAC VNI VTEP MAC VNI VTEP MAC_A 30000 E1/12 MAC_B 30000 E1/4 MAC_B 30000 IP_V2 MAC_A 30000 IP_V1

ARP Response for IP_B Src MAC: MAC_B 2 ARP Response from IP_B Dst MAC: MAC_A 1 V Src MAC: MAC_B V Dst MAC: MAC_A 7 6 5 SMAC: MAC_V2 DMAC: hop-by-hop

SIP: IP_V2

DIP: IP_V1 Underlay Host A 3 UDP MAC_A / IP_A V Host B MAC_B / IP_B VXLAN VNID: 30000

ARP Response Overlay SMAC: MAC_B DMAC: MAC_A

VXLAN FabricPath

Encapsulation Packet Encapsulation (PE) Frame Encapsulation (FE) Transport Medium Requirement Layer-3 Layer-1 (mandatory) End-Host Reachability and Distribution Flood & Learn Flood & Learn (+Conversational Learning)

End-Host Detection Flood & Learn Flood & Learn Multi-Destination Traffic (BUM*) Multicast (PIM) FabricPath IS-IS forwarding Underlay Control-Plane Any Unicast Routing Protocol FabricPath IS-IS (static, OSPF, IS-IS, eBGP)

Unique Node Identifier VTEP IP SwitchID Standard Reference RFC 7348 TRILL based (Cisco Proprietary)

*BUM: Broadcast, Unknown Unicast, Multicast

• Data Center Fabric Properties • Optimized Networks with VXLAN – Overview – Underlay – Control & Data Plane – Multi-Tenancy • Optimized Networks with FabricPath • Fabric Management & Automation

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 39 Deployment Considerations Underlay • MTU and Overlays • Unicast Routing Protocol and IP Addressing • Multicast for BUM* Traffic Replication

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 40 MTU and VXLAN Underlay • VXLAN adds 50 Bytes to the Outer MAC Header Original Ethernet Frame • Avoid Fragmentation by adjusting Outer IP Header Underlay the IP Networks MTU

UDP Header • Data Centers often require Jumbo MTU; most Server NIC do support VXLAN Header up to 9000 Bytes • Using a MTU of 9216* Bytes 50 (54) Bytes of Overhead of Bytes (54) 50 accommodates VXLAN Overhead Original Layer-2 Frame Overlay plus Server max. MTU

*Cisco Nexus 5600/6000 switches only support 9192 Byte for Layer-3 Traffic

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 41 Building your IP Network – Interface Principles Underlay • Know your IP addressing and IP scale requirements – Best to use single Aggregate for all Underlay Links and Loopbacks – IPv4 only – For each Point-2-Point (P2P) 2 connection, minimum /31 required V1 V – Loopback requires /32 • Routed Ports/Interfaces – Layer-3 Interfaces between Spine and Leaf (no switchport) V3 • VTEP uses Loopback as Source- Interface

Interface Configuration Example for (V1) # Loopback Interface Configuration (VTEP) interface loopback 0 ip address 10.10.10.V1/32 mtu 9192

# Point-2-Point (P2P) Interface Configuration interface Ethernet 2/1 no switchport 2 ip address 192.168.1.1/31 1 V mtu 9192 V interface Ethernet 2/2 no switchport ip address 192.168.1.3/31 mtu 9192 . . V3

Example from depicted topology: 4 Spine * 3 Leaf = 12 Point-2-Point (P2P) Links 12 Links * 2 (/31) + 3 VTEP + 2 RR

= 24 IP Addresses for P2P Links = 5 IP Addresses for Loopback Interfaces

29 IP Addresses required == /27 Prefix

2 A More Realistic Scenario: V1 V 4 Spine * 40 Leaf = 160 Point-2-Point (P2P) Link 160 Links * 4 (/30) + 40 VTEP + 2 RR

= 640 IP Addresses for P2P Links = 42 IP Addresses for Loopback Interface

682 IP Addresses required == /22 Prefix V3

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 44 IP Unnumbered – Simplifying the Principals Underlay • IP Unnumbered – Single IP Address for multiple Interfaces • Well-Known from Serial Interfaces (back in time) • Used for Layer-3 Interfaces between 2 Spine and Leaf (no switchport) V1 V • For each Switch in the fabric, single IP address is sufficient – Loopback for VTEP – IP Unnumbered from Loopback for V3 routed Interfaces

Check Platform & Release Support for Ethernet IP Unnumbered

Interface Configuration Example for (V1) # Loopback Interface Configuration (VTEP & IP Unnumbered) interface loopback 0 ip address 10.10.10.V1/32 mtu 9192

# Point-2-Point (P2P) Interface Configration interface Ethernet 2/1 2 no switchport 1 V ip unnnumbered loopback 0 V mtu 9192 interface Ethernet 2/2 no switchport ip unnnumbered loopback 0 mtu 9192 . . V3

Check Platform & Rlease Support for Ethernet IP Unnumbered

Example from showed Topology: 4 Spine + 3 Leaf = 7 Individual Devices

= 7 IP Addresses for Loopback Interface (Used for VTEP & Routed Interfaces; IP Unnumbered)

7 IP Addresses required == /29 Prefix

2 A More Realistic Scenario: 1 V 4 Spine + 40 Leaf = 44 Individual Devices V

= 44 IP Addresses for Loopback Interface (Used for VTEP & Routed Interfaces; IP Unnumbered)

44 IP Addresses required == /26 Prefix V3

Check Platform & Release Support for Ethernet IP Unnumbered

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 47 Building your IP Network – Routing Protocols; OSPF Underlay • OSPF – watch your Network type – Network Type Point-2-Point (P2P) • Preferred (only LSA type-1) • No DR/BDR election • Suits well for routed interfaces/ports (optimal from a LSA Database perspective) 2 • Full SPF calculation on Link Change V1 V – Network Type Broadcast • Suboptimal from a LSA Database perspective (LSA type-1 & 2) • DR/BDR election

• Additional election and Database 3 Overhead V

Configuration Example for (V1) # Loopback Interface Configuration (VTEP) interface loopback 0 ip address 10.10.10.V1/32 mtu 9192 ip router ospf 1 area 0.0.0.0 ip ospf network point-to-point

# Point-2-Point (P2P) Interface Configuration 2 interface Ethernet 2/1 1 V no switchport V ip address 192.168.1.1/31 mtu 9192 ip router ospf 1 area 0.0.0.0 ip ospf network point-to-point *

*Additional OSPF Configuration like Routing Process, Authentication or BFD are not shown BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 49 Building your IP Network – Routing Protocols; IS-IS Underlay • IS-IS – what was this CLNS? - Independent of IP (CLNS) - Well suited for routed interfaces/ports - No SPF calculation on Link change; only if Topology changes - Fast Re-convergence 2 1 V - Not everyone is familiar with it V

Configuration Example for (V1) # Loopback Interface Configuration (VTEP) interface loopback 0 ip address 10.10.10.V1/32 mtu 9192 ip router isis 1 medium p2p

# Point-2-Point (P2P) Interface Configuration 2 interface Ethernet 2/1 1 V no switchport V ip address 192.168.1.1/31 mtu 9192 ip router isis 1 medium p2p *

*Additional ISIS Configuration like Routing Process, Authentication or BFD are not shown BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 51 Building your IP Network – Routing Protocols; iBGP Underlay • iBGP + IGP = The Routing Protocol Combo – IGP for underlay topology & reachability (e.g. IS-IS, OSPF) – iBGP for VTEP (loopback) reachability – iBGP route-reflector for simplification and scale RR RR RR RR – Requires two routing protocols

iBGP 2 V1 V

Configuration Example # Spine (S1) router bgp 65500 neighbor 10.10.10.V1 remote-as 65500 description S1-Loopback-to-V1-Loopback address-family ipv4 unicast route-reflector-client neighbor 10.10.10.V2 remote-as 65500 description S1-Loopback-to-V2-Loopback address-family ipv4 unicast RR RR RR RR route-reflector-client

# Leaf (V1) router bgp 65500 iBGP 2 neighbor 10.10.10.S1 remote-as 65500 V1 V description V1-Loopback-to-S1-Loopback neighbor 10.10.10.S2 remote-as 65500 description V1-Loopback-to-S2-Loopback * V3

*Simplified BGP configuration; IGP not shown

• eBGP

eBGP – eBGP Peer is IP interface 2 V 1 V

• Loopback would require additional IGP AS #65501 AS #65502 and eBGP multi-hop

– Multiple Autonomous-Systems (AS) V 3 • Minimum amount of AS is two AS #65500 AS #65503 – Many BGP Neighbors RR RR RR RR • For each neighboring p2p interface

– AS Path eBGP 2 • Src and Dst AS might be same V1 V – No Route-Reflector • But next-hop needs to be unchanged

V3 AS #65501

Configuration Example 1 # Spine (S ) eBGP 2 router bgp 65500 1 V V neighbor 192.168.1.1 remote-as 65501 AS #65501 AS #65502 description Spine1-Interface-to-V1 neighbor 192.168.1.5 remote-as 65501 description Spine1-Interface-to-V2 V 3 # Leaf (V1) AS #65500 router bgp 65501 AS #65503 RR RR RR RR neighbor 192.168.1.2 remote-as 65500

description V1-Interface-to-S1

address-family ipv4 unicast eBGP allowas-in 2 1 V neighbor 192.168.1.6 remote-as 65500 V description V1-Interface-to-S2 address-family ipv4 unicast allowas-in * V3 AS #65501 *Simplified BGP configuration

ASR 1000 Nexus 1000v Nexus 3000 Nexus 5600 Nexus 7000/F3 Nexus 9000 ASR 9000 CSR 1000 Multicast IGMP v2/v3 PIM ASM PIM BiDir PIM ASM / PIM BiDir PIM ASM PIM BiDir PIM ASM / PIM BiDir Mode • Spine and Aggregation Switches make good Rendezvous-Point (RP) Locations in Topologies • Reserve a range of Multicast Groups (Destination Groups/DGroups) to service the Overlay and optimize for diverse VNIs • In Spine/Leaf topologies with lean Spine – Use multiple Rendezvous-Point across the multiple Spines – Map different VNIs to different Rendezvous-Point for simple load balancing measure – Use Redundant Rendezvous-Pint • Design a Multicast Underlay for a Network Overlay, Host VTEPs will leverage this Network

• Redundant Rendezvous-Point using PIM RP RP Anycast-RP or MSDP • Source-Tree or Unidirectional Shared- Tree (Source-Tree shown) 2 – Shared-Tree will always use RP for 1 V forwarding V • 1 Source-Tree per Multicast-Group per VTEP (each VTEP is Source & Receiver) RP Rendezvous-Point • Example from depicted topology V3 – 3 VTEPs sharing same VNI and Multicast- Group mapping (single Multicast-Group) – 3x Source-Tree (1 per VTEP per Multicast- BRKDCT-Group3378 )© 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 57 Multicast Enabled Underlay – PIM ASM Underlay

Configuration Example for (V1) # Using Anycast Rendezvous-Point ip pim rp-address 10.10.10.SX RP RP # Loopback Interface Configuration (VTEP) interface loopback 0 ip address 10.10.10.V1/32 mtu 9192 ip pim sparse-mode # Anycast-RP Configuration 2 1 V # Point-2-Point (P2P) Interface Configrationip pim rp-address 10.10.10.SXV interface Ethernet 2/1 ip pim anycast-rp 10.10.10.SX 10.10.10.S1 no switchport ip pim anycast-rp 10.10.10.SX 10.10.10.S2 ip address 192.168.1.1/31 mtu 9192 # Loopback Interface Configuration (RP) ip pim sparse-mode interface loopback 0 ip address 10.10.10.S1/32 RP Rendezvous-Point ip pim sparse-mode V3 # Loopback Interface Configuration (Anycast RP) interface loopback 1 ip address 10.10.10.SX/32 ip pim sparse-mode

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 58 Multicast Enabled Underlay – PIM SSM Underlay • PIM Source Specific Multicast (SSM) • No Rendezvous-Point required • Source-Tree or Unidirectional Shared- Tree • 1 Source-Tree per Multicast-Group per 2 VTEP (each VTEP is Source & V1 V Receiver) • Example from showed Topology – 3 VTEPs sharing same VNI and Multicast- Group mapping (single Multicast-Group) – 3x Source-Tree (1 per VTEP per Multicast- V3 Group)

Configuration Example for (V1) # Specificing PIM Group-Range ip pim ssm range 239.128.1.0/24

# Loopback Interface Configuration (VTEP) interface loopback 0 ip address 10.10.10.V1/32 mtu 9192 ip pim sparse-mode # Specificing PIM Group-Range 2 1 V # Point-2-Point (P2P) Interface Configrationip pim ssm range 239.128.1.0/24V interface Ethernet 2/1 no switchport # Loopback Interface Configuration (RP) ip address 192.168.1.1/31 interface loopback 0 mtu 9192 ip address 10.10.10.S1/32 ip pim sparse-mode ip pim sparse-mode

# Point-2-Point (P2P) Interface Configuration interface Ethernet 2/1 V3 no switchport ip address 192.168.1.1/31 mtu 9192 ip pim sparse-mode

• Redundant Rendezvous-Point using RP RP Phantom-RP • Building Bi-Directional Shared-Tree – Uses shortest path between Source and Receiver with RP as routing-vector 2 V1 V • 1 Shared-Tree per Multicast-Group • Example from depicted topology – 3 VTEPs sharing same VNI and Multicast- RP Rendezvous-Point Group mapping (single Multicast-Group) – 1x Shared-Tree (1 per Multicast-Group) V3

Configuration Example for (V1) # Using Phantom Rendezvous-Point ip pim rp-address 10.10.10.SX bidir RP RP # Loopback Interface Configuration (VTEP) interface loopback 0 ip address 10.10.10.V1/32 # Using Phantom Rendezvous-Point mtu 9192 ip pim rp-address 10.10.10.SX bidir ip pim sparse-mode # Loopback Interface Configuration (RP) 2 interface loopback 0 1 V # Point-2-Point (P2P) Interface Configration V ip address 10.10.10.S1/32 interface Ethernet 2/1 ip pim sparse-mode # Using Phantom Rendezvous-Point no switchport ip pim rp-address 10.10.10.SX bidir ip address 192.168.1.1/31 mtu 9192 # Loopback Interface Configuration (Anycast RP) interface loopback 1 # Loopback Interface Configuration (RP) ip pim sparse-mode interface loopback 0 ip address 10.10.10.SX/32 RP Rendezvous-Point ip pim sparse-mode ip address 10.10.10.S2/32 ip pim sparseV-3mode # Loopback Interface Configuration (Anycast RP) interface loopback 1 ip address 10.10.10.SX/31 ip pim sparse-mode BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 62 Multicast Enabled Underlay – PIM ASM vs. BiDir Underlay - Flow Diagram Shared Tree

PIM ASM RP PIM BiDir RP with Shared Tree with Shared Tree

src rcvr src rcvr

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 63 To Remember Multicast Enabled Underlay • Multi-Destination Traffic (Broadcast, Unknown Unicast, etc.) needs to be replicated to ALL VTEPs serving a given VNI – Each VTEP is Multicast Source & Receiver • For a given VNI, all VTEPs act as a Sender and a Receiver • Head-End Replication will depend on hardware scale/capability • Resilient, efficient, and scalable Multicast Forwarding is highly desirable – Choose the right Multicast Routing Protocol for your need (type/mode) – Use redundant Multicast Rendezvous Points (Spine/Aggregation generally preferred) – 99% percent of Overlay problems are in the Underlay (OTV experience) Keep in Mind Overlay Convergence = Underlay Convergence!

• No IP addressing on Point-2-Point connections* • No ARP flooding on Underlay • Optimized Multi- Destination Topology for Scale and Convergence

*Related to VXLAN Layer-3 Underlay (IP Un-Numbered) BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 65 Optimized Networks with VXLAN Data Center Fabric Properties Extended Namespace  Scalable Layer-2 Domains Integrated Route and Bridge Multi-Tenancy

VXLAN FabricPath

Unique Node Identifier VTEP IP SwitchID Standard Reference RFC 7348 TRILL based (Cisco Proprietary)

*BUM: Broadcast, Unknown Unicast, Multicast

Control- EVPN MP-BGP Plane draft-ietf-l2vpn-evpn

Multi-Protocol Label Switching Provider Backbone Bridges Network Virtualization Overlay Data- (MPLS) (PBB) (NVO) Plane draft-ietf-l2vpn-evpn draft-ietf-l2vpn-pbb-evpn draft-sd-l2vpn-evpn-overlay

 EVPN over NVO Tunnels (VXLAN, NVGRE, MPLSoE) for Data Center Fabric encapsulations  Provides Layer-2 and Layer-3 Overlays over simple IP Networks

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 74 EVPN – Ethernet VPN Additional Information • IETF Layer-2 Virtual Private Network (l2vpn) Working Group – http://datatracker.ietf.org/wg/l2vpn/ • RFC 7209: Requirements for Ethernet VPN (EVPN) – http://tools.ietf.org/html/rfc7209 • Based Specifications: draft-ietf-l2vpn-evpn – http://tools.ietf.org/html/draft-ietf-l2vpn-evpn

• Multi-Protocol BGP (MP-BGP) based Control-Plane using Protocol Learning EVPN NLRI (Network Layer Reachability Information)

• Workload MAC and IP • Make Forwarding decisions at VTEPs for Layer-2 (MAC) Addresses learnt by VXLAN and Layer-3 (IP); Integrated Route/Bridge (IRB) Edge Devices (NVEs) • Advertises Layer-2 and • Reduce Flooding Layer-3 Address-to-VTEP Association (Overlay • Reduce impact of ARP on the Network Control-Plane) • Flood Prevention • Standards Based (IETF draft) • Optimized ARP forwarding

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 76 Host and Subnet Route Distribution VXLAN/EVPN • Host Route Distribution decoupled from the Underlay protocol RR RR • Use MultiProtocol-BGP (MP-BGP) on the Leaf nodes to distribute internal Host/Subnet Routes and external reachability information 2 V1 V • Route-Reflectors deployed for scaling purposes

RR BGP Route-Reflector

V3 iBGP Adjacency

RR RR

1 1 1 2 V1 V

Host A 3 MAC_A / IP_A V Host B MAC_B / IP_B VTEPs advertise Host Routes (IP+MAC) 1 for the Host within the Control-Plane Virtual Switch

Host C Host Y MAC_C / IP_C MAC_Y / IP_Y BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 78 Protocol Learning & Distribution (2) VXLAN/EVPN 3 3 RR RR MAC, IP VNI NH MAC, IP VNI NH MAC_B, IP_B 30000 IP_V2 MAC_A, IP_A 30000 IP_V1 2 MAC_C, IP_C 30000 IP_V3 MAC_C, IP_C 30000 IP_V3 MAC_Y, IP_Y 30001 IP_V32 MAC_Y, IP_Y 30001 IP_V3 2 2 V1 V 3

MAC, IP VNI NH MAC_A, IP_A 30000 IP_V1 MAC_B, IP_B 30000 IP_V2 Host A 3 MAC_A / IP_A V Host B MAC_B / IP_B BGP propagates routes for 2 The Host to all other VTEPs Virtual Switch 3 VTEPs obtain host routes for remote hosts and install in RIB/FIB Host C Host Y MAC_C / IP_C MAC_Y / IP_Y BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 79 V1# sh bgp l2vpn evpn IP_A BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 30000:V1 Host Advertisement BGP routing table entry for [2]:[0]:[0]:[48]:[MAC_A]:[32]:[IP_A]/272, version 28838 Paths: (1 available, best #1) VXLAN/EVPN Flags: (0x000202) on xmit-list, is not in l2rib/evpn 48, MAC, 32, IP Advertised path-id 1 Path type: internal, path is valid, is best path, no labeled nexthop 1. Host Attaches AS-Path: NONE, path sourced internal to AS IP_V1 (metric 3) from RR (RR) Origin IGP, MED not set, localpref 100, weight 0 2. VTEP V1 advertises Host A MAC Received label 30000 50000 RR RR Extcommunity: RT:1000:30000 RT:1000:50000 ENCAP:3 ENCAP:3 = VXLAN (+IP) through BGP RR Originator: IP_V1 Cluster list: RR Remote Next-hop Attribute: IP_V1 encapsulation VXLAN VNID 50000 MAC MAC_V1 3. Choice of Encapsulation is also advertised 2 V1 V

MAC, IP VNI VNI NH Encap Seq (L2) (L3) MAC_A, IP_A 30000 50000 IP_V1 3:VXLAN 0 RR BGP Route-Reflector

V3 iBGP Adjacency

Host A MAC_A / IP_A

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 80 V1# sh bgp l2vpn evpn IP_A BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 30000:V1 BGP routing table entry for [2]:[0]:[0]:[48]:[MAC_A]:[32]:[IP_A]/272, version 28838 Paths: (1 available, best #1) Flags: (0x000202) on xmit-list, is not in l2rib/evpn 48, MAC, 32, IP Advertised path-id 1 Path type: internal, path is valid, is best path, no labeled nexthop AS-Path: NONE, path sourced internal to AS IP_V1 (metric 3) from RR (RR) Origin IGP, MED not set, localpref 100, weight 0 Received label 30000 50000 ENCAP:3 = VXLAN Extcommunity: RT:1000:30000 RT:1000:50000 ENCAP:3 Originator: IP_V1 Cluster list: RR Remote Next-hop Attribute: IP_V1 encapsulation VXLAN VNID 50000 MAC MAC_V1

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 81 V1# sh bgp l2vpn evpn IP_A BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 30000:V3 Host Moves BGP routing table entry for [2]:[0]:[0]:[48]:[MAC_A]:[32]:[IP_A]/272, version 28839 Paths: (1 available, best #1) VXLAN/EVPN Flags: (0x000202) on xmit-list, is not in l2rib/evpn 48, MAC, 32, IP Advertised path-id 1 Path type: internal, path is valid, is best path, no labeled nexthop 1. Host Moves to V3 AS-Path: NONE, path sourced internal to AS IP_V3 (metric 3) from RR (RR) Origin IGP, MED not set, localpref 100, weight 0 2. V3 detects Host A and advertises Received label 30000 50000 RR RR Extcommunity: RT:1000:30000 RT:1000:50000 ENCAP:3 ENCAP:3 = VXLAN it with Seq #1 Originator: IP_V3 Cluster list: RR Remote Next-hop Attribute: IP_V3 encapsulation VXLAN VNID 50000 MAC MAC_V3 3. V1 sees more recent route and withdraws its advertisement 2 V1 V

MAC, IP VNI VNI NH Encap Seq (L2) (L3) MAC_A, IP_A 30000 50000 IP_V3 3:VXLAN 1 RR BGP Route-Reflector

V3 iBGP Adjacency

Host A MAC_A / IP_A

RR RR MAC, IP VNI NH MAC, IP VNI NH MAC_B, IP_B 30000 IP_V2 MAC_A, IP_A 30000 IP_V1 MAC_C, IP_C 30000 IP_V3 MAC_C, IP_C 30000 IP_V3 MAC_Y, IP_Y 30001 IP_V3 MAC_Y, IP_Y 30001 IP_V3 ARP Request for IP_B 2 Src MAC: MAC_A 1 V Dst MAC: FF:FF:FF:FF:FF:FF V 1 ARP Response for IP_B 2 Src MAC: MAC_B Dst MAC: MAC_A MAC, IP VNI NH MAC_A, IP_A 30000 IP_V1 MAC_B, IP_B 30000 IP_V2 Host A 3 MAC_A / IP_A V Host B MAC_B / IP_B

1 ARP Request sent for IP_B sent from Host A Virtual Switch 2 V1 knows about IP_B and can respond. No need for ARP forwarding across the Network Host C Host Y MAC_C / IP_C MAC_Y / IP_Y BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 83 ARP Handling on Lookup “Miss” (1) VXLAN/EVPN Missing RR RR MAC, IP “B”VNI NH MAC, IP VNI NH MAC_C, IP_C 30000 IP_V3 MAC_A, IP_A 30000 IP_V1 MAC_Y, IP_Y 30001 IP_V3 2 MAC_C, IP_C 30000 IP_V3 MAC_Y, IP_Y 30001 IP_V3 ARP Request for IP_B Src MAC: MAC_A 2 ARP Request for IP_B Dst MAC: FF:FF:FF:FF:FF:FF 1 V Src MAC: MAC_A V Dst MAC: FF:FF:FF:FF:FF:FF 1 2

MAC, IP VNI NH MAC_A, IP_A 30000 IP_V1

Host A 3 MAC_A / IP_A V Host B MAC_B / IP_B ARP Request for IP_B Src MAC: MAC_A Dst MAC: FF:FF:FF:FF:FF:FF 1 ARP Request sent for IP_B sent from Host A Virtual Switch 2 Miss of IP_B. Forward ARP Request to all Ports except source-port (ARP snooping) Host C Host Y MAC_C / IP_C MAC_Y / IP_Y BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 84 ARP Handling on Lookup “Miss” (2) VXLAN/EVPN

RR RR MAC, IP VNI NH

MAC, IP VNI NH MAC_A, IP_A 30000 IP_V1 MAC_C, IP_C 30000 V3IP_V3 4 MAC_C, IP_C 30000 IP_V3 MAC_Y, IP_Y 30001 V3IP_V3 MAC_Y, IP_Y 30001 IP_V3 ARP Response for IP_B MAC_B, IP_B 30000 IP_V2 2 ARP Response from IP_B Src MAC: MAC_B 1 V Src MAC: MAC_B Dst MAC: MAC_A V Dst MAC: MAC_A 4 3 MAC, IP VNI NH MAC_A, IP_A 30000 V1IP_V1 MAC_B, IP_B 30000 IP_V2 Host A 3 MAC_A / IP_A V Host B MAC_B / IP_B

3 ARP Response is sent to V2 Virtual Switch 4 V2 will populate this information in the control-plane (learn) and forward it subsequently Host C Host Y MAC_C / IP_C MAC_Y / IP_Y BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 85 Packet Forwarding (Bridge) VXLAN/EVPN

RR RR MAC, IP VNI NH MAC, IP VNI NH MAC_A, IP_A 30000 Local MAC_B, IP_B 30000 Local MAC_B, IP_B 30000 IP_V2 MAC_A, IP_A 30000 IP_V1 SMAC: MAC_A SMAC: MAC_A DMAC: MAC_B DMAC: MAC_B 2 SIP: IP_A 1 V V SIP: IP_A DIP: IP_B DIP: IP_B 1 2 3 4 SMAC: MAC_V1 SMAC: hop-by-hop DMAC: hop-by-hop DMAC: MAC_V2

SIP: IP_V1 SIP: IP_V1

DIP: IP_V2 DIP: IP_V2 Underlay Underlay Host A UDP 3 MAC_A / IP_A V UDP Host B MAC_B / IP_B VXLAN VNID: 30000 VXLAN VNID: 30000

SMAC: MAC_A SMAC: MAC_A

DMAC: MAC_B DMAC: MAC_B

Overlay Overlay SIP: IP_A SIP: IP_A DIP: IP_B DIP: IP_B

RR RR MAC, IP VNI NH VRF MAC, IP VNI NH VRF MAC_A, IP_A 30000 Local 50000 MAC_A, IP_A 30000 Local 50000 MAC_F, IP_F 30005 IP_V2 50000 MAC_F, IP_F 30005 E1/4 50000 SMAC: MAC_A SMAC: MAC_GW DMAC: MAC_GW DMAC: MAC_F 2 SIP: IP_A 1 V V SIP: IP_A DIP: IP_F DIP: IP_F 1 2 3 4 SMAC: MAC_V1 SMAC: hop-by-hop DMAC: hop-by-hop DMAC: MAC_V2

SIP: IP_V1 SIP: IP_V1

DIP: IP_V2 DIP: IP_V2 Underlay Underlay Host A UDP 3 MAC_A / IP_A V UDP Host F MAC_F, IP_F VXLAN VNID: 50000 VXLAN VNID: 50000

SMAC: MAC_A SMAC: MAC_GW

DMAC: MAC_GW DMAC: MAC_F

Overlay Overlay SIP: IP_A SIP: IP_A DIP: IP_F DIP: IP_F

RR RR MAC, IP VNI NH VRF MAC, IP VNI NH VRF MAC_A, IP_A 30000 Local 50000 MAC_A, IP_A 30000 Local 50000 Subnet F 30005 IP_V2 50000 Subnet F 30005 E1/4 50000 SMAC: MAC_A SMAC: MAC_GW DMAC: MAC_GW DMAC: MAC_F 2 SIP: IP_A 1 V V SIP: IP_A DIP: IP_F DIP: IP_F 1 2 3 4 SMAC: MAC_V1 SMAC: hop-by-hop DMAC: hop-by-hop DMAC: MAC_V2

SIP: IP_V1 SIP: IP_V1

DIP: IP_V2 DIP: IP_V2 Underlay Underlay Host A UDP 3 MAC_A / IP_A V UDP Host F MAC_F, IP_F VXLAN VNID: 50000 VXLAN VNID: 50000

SMAC: MAC_A SMAC: MAC_GW

DMAC: MAC_GW DMAC: MAC_F

Overlay Overlay SIP: IP_A SIP: IP_A DIP: IP_F DIP: IP_F

Configuration Example for VLAN, VNI, VRF Configuration Example for MP-BGP EVPN # VLAN to VNI mapping (MT-Lite) # BGP Configuration for VRF (routing) vlan 43 router bgp 65535 vn-segment 30000 address-family ipv4 unicast vlan 55 neighbor RR_IP remote-as 65535 vn-segment 30001 address-family ipv4 unicast address-family l2vpn evpn # Allocate VLAN for VRF VNI send-community extended vlan 500 vrf VRF-A vn-segment 50000 address-family ipv4 unicast advertise l2vpn evpn # VRF configuration for “customer” VRF vrf context VRF-A # EVPN Configuration for VNI (bridging) vni 50000 evpn rd auto vni 30000 l2 address-family ipv4 unicast rd auto route-target both auto evpn route-target both auto vni 30001 l2 rd auto route-target both auto

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 89 Optimized Networks with VXLAN Data Center Fabric Properties Extended Namespace Scalable Layer-2 Domains Integrated Route and Bridge Multi-Tenancy

• Forward based on MAC or IP address learnt via Control- IP Services Plane (MP-BGP EVPN)

• VXLAN Routing • Make routing decisions at VTEPs • Distributed Anycast Gateway (requires Overlay • Scale and Multipathing (ECMP) Control-Plane) • Multi-Tenancy • Leverage Layer-3 Gateway capabilities along with Protocol Information • LISP-ish / LISP-like approach for Host/IP Mobility – Location (VTEP), Identifier (MAC, IP of End-Host)

VY VX

2 V2 V

V3 Layer-3 Boundary V3

V1 V1

Centralized Gateway Distributed Gateway • Extra Bridging hop before and after Routing • Route or Bridge at Leaf • Centralized Gateway (Aggregation) for Routing • Distributed Gateway (Anycast) for Routing • Large amounts of state => convergence issues • Disaggregate state by scale out • Scale problem for large Layer-2 domains • Optimal Scalability • Works with VXLAN Flood & Learn or EVPN • Requires VXLAN/EVPN!

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 92 Centralized Gateway (FHRP) VXLAN Routing • Centralized Routing in a Layer-2 VXLAN Network VY – Routing between VNI ( Different VX Subnet) – Bridging within VNI (Same Subnet) V2 • Inter-VXLAN Routing at Core/Aggregation Layer V3 • vPC provides MAC state synchronization and HSRP peering 1 – Redundant VTEPs share Anycast V VTEP IP address in the Underlay Host Y VNI 30001 Host A VNI 30000

Configuration Example for (Vx) Configuration Example for (Vy) # VLAN to VNI mapping (MT-Lite) # VLAN to VNI mapping vlan 43 vlan 43 vn-segment 30000 vn-segment 30000 vlan 55 vlan 55 vn-segment 30001 vn-segment 30001

# Gateway Interface for “BLUE” & “RED”(SVI) # Gateway Interface for “BLUE” & “RED”(SVI) interface vlan 43 interface vlan 43 vrf member VRF-A vrf member VRF-A ip address 11.11.11.2/24 ip address 11.11.11.3/24 hsrp 43 hsrp 43 ip 11.11.11.1 ip 11.11.11.1 interface vlan 55 interface vlan 55 vrf member VRF-A vrf member VRF-A ip address 98.98.98.2/24 ip address 98.98.98.3/24 hsrp 43 hsrp 43 ip 98.98.98.1 ip 98.98.98.1

VPC Configuration not shown

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 94 Distributed IP Anycast Gateway* VXLAN/EVPN • Distributed Routing with IP Anycast Gateway (Integrated Route/Bridge IRB) – Routing between VNI (Different Subnet) – Bridging within VNI (Same Subnet) V2 • Inter-VXLAN Routing Leaf/Access Layer V3 – All Leafs share gateway IP and MAC for a Subnet (No HSRP) V1 – A Host will always find its Gateway

directly attached anywhere it moves Host Y VNI 30001 *Requires EVPN Control-Plane. Host A VNI 30000

Configuration Example for “BLUE” (V1 & V3) Configuration Example for “RED” (V1-3) # VLAN to VNI mapping (MT-Lite) # VLAN to VNI mapping (MT-Lite) vlan 43 vlan 55 vn-segment 30000 vn-segment 30001

# Anycast Gateway MAC, inherited by any interface # Anycast Gateway MAC, inherited by any interface (SVI) using “fabric forwarding” (SVI) using “fabric forwarding” fabric forwarding anycast-gateway-mac fabric forwarding anycast-gateway-mac 0002.0002.0002 0002.0002.0002

# Distributed IP Anycast Gateway (SVI) # Distributed IP Anycast Gateway (SVI) interface vlan 43 interface vlan 55 no shutdown no shutdown vrf member VRF-A vrf member VRF-A ip address 11.11.11.1/24 tag 12345 ip address 98.98.98.1/24 tag 12345 fabric forwarding mode anycast-gateway fabric forwarding mode anycast-gateway

*Requires EVPN Control-Plane. VRF and BGP configuration not shown

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 96 Consistent Configuration with Distributed Gateway VXLAN/EVPN • Logical Configuration instantiated on ALL Leafs (consistent/brute force) • ARP & MAC state of all hosted VLAN/VNI and SVI across the Network V2 • Flooding to ALL Leafs (all Leafs have all VLAN/VNI instantiated) V3

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 97 Scoping Configuration with Distributed Gateway VXLAN/EVPN • Logical Configuration only instantiated at respective Leaf (scoped) • ARP & MAC state only for local hosted VLAN/VNI and SVI V2 • Flooding only to respective Leaf (where VLAN/VNI is instantiated) V3 • Host demands provisioning; two models available V1 – top-down Orchestration, push to Leaf – bottom-up Orchestration, pull by Leaf

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 98 Different integrated Route/Bridge (IRB) Modes VXLAN Routing • Overlay Networks do follow two slightly different integrated Route/Bridge (IRB) semantics ? • Asymmetric – Uses different “path” from Source to V2 Destination and back

• Symmetric V3 – Uses same “path” from Source to Destination and back V1 • Cisco follows Symmetric IRB

Host Y VNI 30001 Host A VNI 30000

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 99 Asymmetric IRB VXLAN Routing • Asymmetric – Similar to Inter-VLAN routing – Source and Destination VNI has to exist on Switch where routing happens – Post Routing traffic shares destination VNI with Bridged traffic V2 – Not very suitable for distributed Routing V3 • From Host A via VLAN/VNI “blue” routed at V1 to VNI “red” reaching destination VLAN “red” 1 • From Host Y via VLAN/VNI “red” routed at V V3 to VNI “blue reaching destination VLAN “blue” Host Y VNI 30001 Host A VNI 30000

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 100 Symmetric IRB VXLAN Routing • Symmetric – Similar to creating a Transit Segment – Regardless of where Source or Destination VNI exists – Post Routing traffic uses different VNI than Bridged traffic V2 – Additional VNI for Routing traffic (per VRF) V3 • From Host A via VLAN “blue” routed at V1 to VNI “purple” reaching destination VLAN “red” 1 • From Host Y via VLAN “red” routed at V3 V to VNI “purple” reaching destination VLAN “blue” Host Y VNI 30001 – Used in Cisco VXLAN/EVPN Host A VNI 30000

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 101 Asymmetric vs. Symmetric IRB VXLAN Routing - Block Diagram • Asymmetric IRB • Symmetric IRB – Used in Cisco VXLAN/EVPN

VRF VNI

Layer-2 VNI

V1 V3 V1 V3

2 2 VNI 2 VNI 2 VNI 2 VNI

- - - -

Layer Layer Layer Layer

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 102 Optimized Networks with VXLAN Data Center Fabric Properties Extended Namespace Scalable Layer-2 Domains Integrated Route and Bridge Multi-Tenancy

• No IP addressing on • VXLAN with distributed Point-2-Point Default-Gateway connections* • End Host Discovery and • No ARP flooding on Distribution (aka Control- Underlay Plane) • Optimized Multi- • Minimized Flood & Learn Destination Topology for across Overlay Scale and Convergence

*Related to VXLAN Layer-3 Underlay (IP Un-Numbered) BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 104 Getting the Puzzle Together! Optimized Networks with VXLAN Underlay

Overlay Integrated (VXLAN) Route/Bridge

BGP (EVPN)

VXLAN/EVPN VXLAN FabricPath

Encapsulation Packet Encapsulation (PE) Packet Encapsulation (PE) Frame Encapsulation (FE) Transport Medium Requirement Layer-3 Layer-3 Layer-1 (mandatory) End-Host Reachability and Distribution MP-BGP EVPN Flood & Learn Flood & Learn (+Conversational Learning)

End-Host Detection Localized Flood & Learn w/ ARP Flood & Learn Flood & Learn suppression Multi-Destination Traffic (BUM*) Multicast (PIM) / Ingress-Replication Multicast (PIM) FabricPath IS-IS forwarding Underlay Control-Plane Any Unicast Routing Protocol Any Unicast Routing Protocol FabricPath IS-IS (static, OSPF, IS-IS, BGP) (static, OSPF, IS-IS, eBGP)

Unique Node Identifier VTEP IP VTEP IP SwitchID Standard Reference RFC 7348 + draft-sd-l2vnp-evpn-overlay RFC 7348 TRILL based (Cisco Proprietary)

*BUM: Broadcast, Unknown Unicast, Multicast

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 106 • Yesterday: VXLAN, yet another Overlay VXLAN – Data-Plane only (Multicast based Flood & Learn) applicability • Today: VXLAN for the creation of scalable DC Fabrics – Intra-DC evolves as the – Control-Plane, active VTEP discovery, Multicast and Unicast (Head-End Control Plane Replication) • Future: VXLAN for DCI – Inter-DC – DCI Enhancements (ARP evolves! caching/suppress, Multi-Homing, Failure Domain isolation, Loop Protection etc.)

• Control-Plane – Learn and Distribute MAC information (no Core Flood&Learn) (Layer-3) OTV/VPLS • Multi-Homing – Automated Multi-Homing for Resiliency • Loop Prevention – Using redundant Path V V – Providing Loop protection V V • Fault Containment – Separate Control-Plane information – Limit Flood (ARP caching) • Transport Agnostic Principles for Interconnecting Networks – Can leverage literally any Transport Technology Do Apply for Ethernet, FabricPath and VXLAN

• Simplified Transport Requirement – Multicast dependent and independent Forwarding Core of BUM* Traffic (no hairpin) (Layer-3) OTV/VPLS • Multicast Optimization – Offers optimized Multicast Forwarding • Path Diversity – Flow based Entropy V V

VXLAN or V V VXLAN or • Multi-Site VXLAN/EVPN VXLAN/EVPN – Provides Site to Multi-Site connectivity V V

V V

PrinciplesV for Interconnecting NetworksV Do Apply for Ethernet, FabricPath and VXLAN

Inter-Pod Connectivity

Plane

Homing Site

- -

Multi Prevention Control Loop Fault Containment Transport Agnostic Multicast Optimization PathDiversity Multi

FabricPath ✖ ✔1 ✔✔ ✖ ✖ ✖ ✔ ✖ Good 1 2 VXLAN (Flood&Learn) ✖ BRKDCT✔ -2049✔ - Overlay✖ Transport✔ Virtualization✔ ✔✔ ✖ VXLAN-EVPN ✔✔ ✔Brian1 Farnham✔2 , Technical✔✔ Marketing✔✔ ✔ Engineer✔✔ ✖ Better VPLS ✖ ✔1 ✔✔ Thursday,✖ 2:30p✖ ✖ ✔ ✔

Best OTV ✔✔ ✔✔ ✔✔ ✔✔ ✔✔ ✔✔ ✔✔ ✔✔

1) Only with Multi-Chassis Link Aggregation (MC-LAG / VPC) 2) Limited Overlay Loop Prevention

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 112 Interconnecting Classic Ethernet Networks (Layer-2) Inter-Pod Connectivity • Interconnecting Classic Ethernet Pods with VXLAN/EVPN is possible Core (Layer-3) – EVPN Control-Plane provides ARP VXLAN/EVPN suppression and avoids Flood&Learn VNI 30000 • Ingress Replication to avoid Multicast

requirement in Underlay eBGP – Depends on various communication V V requirement factors (traffic requirements, V V amount of Sites, etc) • Multi-Homing requires VPC(!)

• Loop in one Classic Ethernet Pod can Not All Principles Satisfied influence the other Pod(s) “Good Enough” Solution

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 113 Interconnecting VXLAN Networks (Layer-2) Inter-Pod Connectivity • Interconnecting VXLAN/EVPN Pods with VXLAN/EVPN is possible Core (Layer-3) – Control-Plane Domains (EVPN) can be separated (iBGP/eBGP) • Data-Plane Encapsulation is End-to-End! – Leaf/TOR knows about all VTEP across the eBGP two Data Centers

– BUM Traffic is across Pods VXLAN/EVPN VXLAN/EVPN VNI 30000 VNI 30000

• Decision on Ingress Replication V V (Unicast) or Multicast is across all Pods being interconnected V iBGP V

V Not All Principles Satisfied V “Good Enough” Solution

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 114 Interconnecting VXLAN Networks (Layer-3) Inter-Pod Connectivity • Interconnecting VXLAN/EVPN Pods with VXLAN/EVPN is possible Core (Layer-3) – Control-Plane Domains (EVPN) can be VXLA/EVPN separated (iBGP/eBGP) VNI 99000 • With Layer-3 interconnect, Data-Plane Encapsulation is separated – Routing decision at DC-Edge results in V V Decapsulation VXLAN or V V VXLAN or VXLAN/EVPN VXLAN/EVPN – Requires a Transit VNI between Sites VNI 30000 VNI 31000 V V • No Layer-2 Interconnect! V V

V Not All Principles Satisfied V “Good Enough” Solution

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 115 Optimized Networks with VXLAN Data Center Fabric Properties Extended Namespace Scalable Layer-2 Domains Integrated Route and Bridge  Inter-Pod connectivity Multi-Tenancy

• Data Center Fabric Properties • Optimized Networks with VXLAN – Overview – Underlay – Control- & Data-Plane – Multi-Tenancy • Optimized Networks with FabricPath • Fabric Management & Automation

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 117 Layer-2 Multi-Tenancy Multi-Tenancy • VNI are utilized for providing isolation at Layer-2 and Layer-3 across VXLAN – Received frames must be mapped to specific VNI for VXLAN transport – The VLAN-to-VNI mapping can be V2 performed

• per-Switch (MT-Lite) 3 • per-Port (MT-Full) level V

• VLANs become locally significant on VLAN the Leaf/Port and 1:1 mapped to a V1 VNI (global) Host B VLAN VNI 43 Host A VLAN 43

• Two different Interface “Mode of VLAN Operation” • MT-Lite (Multi-Tenancy Lite) ethernet VLAN VNI vxlan

– VLAN to Segment ID mapping Multi-Tenancy Lite (MT-Lite)

• MT-Full (Multi-Tenancy Full) – Leverages Ethernet Flow Point (EFP) Bridge- or Virtual Services Instance (VSI) Domain approach – Brings flexibility for Multi-Encap ethernet VLAN VNI vxlan Gateway (VLAN to VXLAN, MPLS etc) Multi-Tenancy Full (MT-Full)

• MT-Lite configuration is simplified Method of Configuration Example MT-Lite mapping a IEEE 802.1Q VLAN ID to a VXLAN # VLAN to VNI mapping (MT-Lite) VNI vlan 43 – VLAN to VNI configuration on a per-Switch based vn-segment 30000 – VLAN becomes “Switch Local Identifier” # Layer-2 Interface Configuration; Trunk (MT-Lite) – VNI becomes “Network Global Identifier” interface Ethernet 1/8 switchport mode trunk – 4k VLAN limitation per-Switch does still apply – 4k Network limitation has been removed # Layer-2 Interface Configuration; Access (MT- – Dependent on VLAN Space! Lite) interface Ethernet 1/12 switchport mode access switchport access vlan 43

VLAN # Layer-3 Instance (MT-Lite) interface Vlan 43

ethernet VLAN VNI vxlan ip address x.x.x.1/24

Multi-Tenancy Lite (MT-Lite)

• MT-Full configuration is extended to allow Per- Configuration Example MT-Full Port VLAN to Bridge-Domain mapping. Bridge- # VLAN to VNI mapping (MT-Full) Domain will be mapped to VXLAN VNI vni 30000 – VLAN to VNI configuration on a per-Port based bridge-domain 100 – VLAN becomes “Port Local Identifier” member vni 30000 – Bridge-Domain becomes “Switch Local Identifier” encapsulation profile vni MyProfile – VNI becomes “Network Global Identifier” dot1q 43 vni 30000 – 4k VLAN limitation resides only on a per-Dot1Q Trunk # Interface Configuration (MT-Full) interface Ethernet 1/12 – Independent from VLAN Space! no switchport service instance 1 vni Bridge- encapsulation profile MyProfile default Domain # Layer-3 Instance (MT-Full)

ethernet VLAN VNI vxlan interface Bdi 100 ip address x.x.x.1/24

Multi-Tenancy Full (MT-Full)

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 121 Layer-3 Multi-Tenancy Multi-Tenancy • VNI utilized for providing isolation at Layer-2 and Layer-3 across VXLAN – Received frames must be mapped to specific VNI for VXLAN transport – The VLAN-to-VNI mapping is performed on Routing V2 • All Routed Traffic uses the VNI assigned to the VRF V3

V1 VLAN

Host Y VLAN VNI 30001 Host A VNI 30000

Configuration Example VRF-A with 2 Subnet

vxlan # VRF configuration for “customer” VRF vrf context VRF-A vni 50000 rd auto address-family ipv4 unicast

route-target both auto evpn VNI50000

# Layer-3 Instance (MT-Lite) interface vlan 43 vrf member VRF-A VRF-A ip address 11.11.11.1/24 fabric forwarding mode anycast-gateway

interface vlan 55 vrf member VRF-A ethernet VLAN 55 VLAN 43 ethernet ip address 98.98.98.1/24 fabric forwarding mode anycast-gateway

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 123 Optimized Networks with VXLAN Data Center Fabric Properties Extended Namespace Scalable Layer-2 Domains Integrated Route and Bridge Multi-Tenancy

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 125 Fabric Control Plane Optimized Networks with FabricPath IS-IS for fabric link state distribution • Fabric node reachability for overlay encapsulation • Building multi-destination trees for IS-IS Control Plane multicast and broadcast traffic • Quick reaction to fabric link/node failure (Layer-2 BFD) • Enhanced for mesh topologies IS-IS Adjacencies Fabric Control Protocol doesn’t distribute • Host Routes • Host originated control traffic • Server subnet information

RR RR

Fabric Host/Subnet External Subnet Route Injection Route Injection MP-BGP Control Plane

N1KV/OVS

MAN/WA N iBGP Adjacencies • Host Route Distribution decoupled from the Fabric link state protocol • Use MP-BGP on the leaf nodes to distribute internal host/subnet routes and external reachability information • MP-BGP enhancements to carry up to 1.2 Million routes and reduce convergence time Note: Route-Reflectors deployed for scaling purposes BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 127 Distributed Gateway at the Leaf Optimized Networks with FabricPath

GW IP: 10.10.10.1 GW IP: 20.20.20.1 GW MAC: 0011:2222:3333 GW MAC: 0011:2222:3333

N1KV/OVS Virtual Switch MAN/WA N Anycast Gateway

• Any Subnet anywhere => Any Leaf can instantiate ANY Subnet – All Leafs share gateway IP and MAC for a Subnet (No HSRP) – ARPs are terminated on Leafs, No Flooding beyond Leaf • Facilitates VM Mobility, workload distribution, arbitrary clustering • Seamless Layer-2 or Layer-3 communication between physical hosts and virtual machines

Layer-2 IS-IS Fabric Layer-3 Underlay FabricPath Encapsulation VXLAN Encapsulation

FabricPath/BGP VXLAN/EVPN End-Host Reachability and Distribution Multi-Protocol BGP Multi-Protocol BGP (EVPN) End-Host Detection ARP/ND & VDP ARP/ND Fabric Multicast Control-Protocol FabricPath IS-IS PIM / Ingress Repl. Fabric Topology Control-Plane FabricPath IS-IS Layer-3 OSPF, IS-IS, BGP Leaf IP/MAC binding distribution within Fabric FabricPath IS-IS Multi-Protocol BGP (EVPN)

VXLAN/EVPN FabricPath/BGP

Encapsulation Packet Encapsulation (PE) Frame Encapsulation (FE) Transport Medium Requirement Layer-3 Layer-1 (mandatory) End-Host Reachability and Distribution MP-BGP EVPN MP-BGP L3VPN End-Host Detection Localized Flood & Learn w/ ARP ARP/ND & VDP suppression Multi-Destination Traffic (BUM*) Multicast (PIM) / Ingress-Replication FabricPath IS-IS forwarding Underlay Control-Plane Any Unicast Routing Protocol FabricPath IS-IS (static, OSPF, IS-IS, BGP)

Unique Node Identifier VTEP IP SwitchID Standard Reference RFC 7348 + draft-sd-l2vnp-evpn-overlay TRILL based (Cisco Proprietary)

*BUM: Broadcast, Unknown Unicast, Multicast

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 130 Forwarding Modes Comparison Anycast-Gateway Proxy-Gateway Anycast-Gateway + ARP Suppression (FabricPath/BGP) (FabricPath/BGP) (VXLAN/EVPN)

VLAN/Subnets stretched between Leafs ✓ ✓ ✓

Common Anycast GW IP across Leafs ✓ ✓ ✓

Common Anycast GW MAC across Leafs ✓ ✓ ✓

✓ ✗ (respond to ARP/ND only if Use local Proxy-ARP/ND ✗ (impersonated ARP the destination is available in response from Host) the RIB)

ARP Flooding in Layer-2 Domain ✗ ✓ ✓ (floods across Network) (ARP suppression)

Always routed Intra-Subnet forwarding Bridged + Flood&Learn Bridged + Protocol Learning (TTL decrement) Silent Host Discovery ✗ ✓ ✓ Non-IP Forwarding ✓ ✓ ✓

BRKDCT-3378 © 2015 Cisco and/or its affiliates. All rights reserved. Cisco Public 132 Simplifying Fabric Management & Optimizing Fabric Visibility Advantages SCP/SFTP XMPP Services Services Device Auto-Configuration TFTP LDAP Services Cabling Plan Consistency Check DHCP Message Services Broker Automated Network Provisioning Common point of fabric access Tenant, Virtual Fabric & Host Visibility

UCS Director Advantages Compute & Storage Policies Open APIs Any workload, Anywhere, Anytime Fabric Management Network Services Controller Open Integration: Orchestration

Network & Network Automated Scalable Provisioning Services Policies Published Schema Workload aware fabric

• Simplify – Do not start with the most difficult task (low hanging Fruits) • Standardize – Find common Denominators and create Templates • Automate repetitive Tasks – Use Templates for Simple Tasks and use Automation (e.g. create VLAN, SVI, VRF) • Abstract – Take a step back and look at the WHOLE – Cisco ACI

• No IP addressing on • VXLAN with distributed • Automated Configuration Point-2-Point Default-Gateway of Tenant and Network connections* • End Host Discovery and • Fabric Management and • No ARP flooding on Distribution (aka Control- Troubleshooting Underlay Plane) • Optimized Multi- • Minimized Flood & Learn Destination Topology for across Overlay Scale and Convergence

Session ID Session Title Speaker Day / Time

• Visit the World of Solutions for – Cisco Campus – Overlay Demos (VXLAN, OTV, LISP) / Programmability Demo – Walk in Labs – LABDCT-2227 (Wednesday & Thursday @ 2:30p – Technical Solution Clinics • Meet the Engineer – Wednesday & Thursday @ 10:00 – 11:00 • Lunch time Table Topics – Always Happy to have Geek Talk during Lunch • DevNet zone related labs and sessions – DevNet1052 (Tuesday, 2:00p) – DevNet-1009 (Thursday, 1:30p) • Recommended Reading: for reading material and further resources for this session, please visit www.pearson-books.com/CLMilan2015

Using TRILL, FabricPath, and VXLAN: Designing Massively Scalable Data Centers (MSDC) with Overlays • Sanjay K. Hooda • Shyam Kapadia • Padmanabhan Krishnan

ISBN-10: 1-58714-393-3

ISBN-13: 978-1-58714-393-9

• Please complete your online session evaluations after each session. Complete 4 session evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt.

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations