BRKACI-2608.Pdf

Multicast in ACI

John Weston, Technical Marketing Engineer Intent Based Networking Group

BRKACI-2608 Cisco Webex Teams

Questions? Use Cisco Webex Teams to chat with the speaker after the session How 1 Find this session in the Cisco Events Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space

BRKACI-2608 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 Keynote Tuesday Wednesday 09:30 BRKACI-2934 08:30 Thursday Troubleshooting BRKACI-2642 08:30 ACI Multi-POD Troubleshooting ACI L3-OUT Friday BRKACI-3545 09:00 ACI Thursday Packet Walk Tuesday BRKACI-2644 09:45 Wednesday Troubleshooting BRKACI-2210 11:00 BRKACI-2641 11:00 ACI Operating Troubleshooting ACI PBR Handbook ACI Endpoints Thursday BRKACI-2271 11:15 Wednesday Monitoring BRKACI-2770 Friday BRKACI-2674 12:15 11:30 Troubleshooting ACI Automating ACI Tuesday BRKACI-1619 14:30 ACI Multi-Site ACI Operations w/Ansible Keynote Thursday Wednesday 17:00 BRKACI-2608 16:45 BRKACI-2645 Tuesday Multicast in ACI 15:45 Cisco Live Troubleshooting Celebration Virtual Machine 18:30 Manager ACI Operations #CLEMEA Keynote Wednesday Tuesday BRKACI-2003 Thursday 09:30 09:00 ACI Multi-POD BRKACI-2690 08:30 Friday ACI in the clouds BRKACI-2301 09:00 ACI micro Segmentation

Wednesday Tuesday BRKACI-2125 11:00 Thursday BRKACI-2210 11:00 ACI Multi-Site BRKACI-2291 11:15 Multi-Pod/Multi-Site BRKACI-3004 ACI Operating ACI Security Handbook build from scratch Deep Dive Friday 11:30

Tuesday BRKACI-2770 BRKACI-2004 14:30 Automating ACI Learn to build Wednesday ACI Fabric BRKACI-2505 16:45 Keynote Kubernetes Thursday integration w/ACI 17:00 BRKACI-3330 Tuesday 17:00 Cisco Live OpenShift Celebration integration w/ACI 18:30 ACI Technology #CLEMEA Session Objectives

At the end of the session, the participants should be able to: . Understand how tenant multicast traffic is forwarded in the ACI fabric

Initial assumption: . The audience already has good knowledge of ACI main concepts (Tenant, BD, EPG, L3Out, Multi-Pod, Multi-Site, etc.) . The audience already has a good understanding of multicast routing (IGMP, PIM)

• Introduction

• Multicast Data Plane

• Multicast Control Plane

• Layer-2 IP Multicast

• Layer-3 Multicast

• Multicast Configuration

• Multicast in a Multi-Pod Fabric

• Multicast in a Multi-Site Architecture

• Inter-VRF Multicast

• Multicast Troubleshooting

• Layer-3 Multicast support (Cloud Scale ASICs) • External RP • Auto-RP/BSR • ASM • SSM • L3 Multicast support for FEX • IGMP/PIM filtering features (No L3out) • MLD snooping • PIM6 2014 2016 2018 2019 2019

1.0(1) 2.0(1) 2.0(2) 3.1(1) 4.0(1) 4.1(1) 4.2(1) 2016 2017 • L3 Multicast with Multi-Site • Layer-2 Multicast • Multicast with Multi-Pod • Inter-VRF multicast • IGMP snooping • RP in the fabric (PIM anycast)* • Fabric IGMP snooping querier • Multicast Route Scale (32k Mroutes)** • External querier • 8,000 multicast group scale (IGMP snooping) *RP in the fabric not supported with Multi-Site **FX only

. ACI is a VXLAN fabric . Unicast packets sent across the VTEP VXLAN IP Payload fabric will be encapsulated in a TEP 10.0.48.64 TEP 10.0.48.67 unicast VXLAN packet . The outer source and destination IP addresses are VXLAN tunnel BD1 VNID: 16383905 endpoints BD1 BD1 . The inner packet carries tenant traffic EP-A EP-B

Src VTEP IP Dst VTEP IP BD VNID 10.0.48.64 10.0.48.67 16383905 EP-B MAC EP-A MAC EP-A IP EP-B IP

Outer Outer Outer Inner Inner VXLAN Src IP Dst IP Payload SRC IP DST IP UDP Dst MAC Src MAC

VRF1 VNID 2129923

VTEP VXLAN IP Payload BD1 VNID 16383905 TEP 10.0.48.64 TEP 10.0.48.67

BD2 VNID 15728630

BD1 BD2 EP-A EP-B

Src VTEP IP Dst VTEP IP VRF VNID 10.0.48.64 10.0.48.67 2129923 EP-A IP EP-B IP

Outer Outer Outer Inner Inner VXLAN Src IP Dst IP Payload SRC IP DST IP UDP Dst MAC Src MAC

• Multicast traffic is also encapsulated in VXLAN across the fabric • Tenant multicast traffic (inner Outer Inner Multicast VXLAN Multicast Payload packet) is encapsulated in an outer VXLAN multicast packet

BD1 BD1 BD2

EP1 EP2 10.10.10.10 10.10.10.11 Multicast Source Receiver for group 239.1.1.1

Note: Also used for other multi-destination BUM traffic (Broadcast, unknown unicast, multicast)

= VTEP (VXLAN Tunnel Endpoint)

ACI Multicast terminology . GIPi: Group IP Inner address. Multicast address in the VTEP VXLAN IP Payload inner VXLAN packet. Tenant multicast traffic running in the tenant VRF . GIPo: Group IP Outer address. Multicast address in the BD1 VNID: 16383905 outer VXLAN packet. This is the multicast address BD GIPo: 225.0.44.144 BD1 BD1 used for distributing multicast traffic across the fabric EP-A EP-B Multicast Source Receiver joined 239.1.1.1

Src VTEP IP BD VNID Dst BD GIPo Multicast EP-A GIPi 10.0.48.64 16383905 EP-A IP 225.0.44.144 MAC MAC 239.1.1.1

Outer Outer Outer Inner Inner VXLAN Src IP Dst IP Payload SRC IP DST IP UDP Dst MAC Src MAC

= VTEP (VXLAN Tunnel Endpoint)

VRF1 APIC allocates a GIPo VNID 2129923 to all BDs and GIPo: 225.1.192.96 multicast enabled VTEP VXLAN IP Payload BD1 VRFs VNID 16383905 GIPo: 225.0.44.144

BD2 VNID 15728630 GIPo: 225.1.170.32 BD1 BD2 EP-A EP-B Multicast Source Receiver 239.1.1.1

Src VTEP IP Dst VRF GIPo VRF VNID GIPi 10.0.48.64 225.1.192.96 2129923 EP-A IP 239.1.1.1

Outer Outer Outer Inner Inner VXLAN Src IP Dst IP Payload SRC IP DST IP UDP Dst Eth Src Eth

Tenant Layer-2 Unicast Packet Outer SRC VTEP DST VTEP BD VNID Dst MAC Src MAC Src IP Dst IP Payload UDP

Tenant Layer-3 Unicast Packet Outer Fabric Fabric SRC VTEP DST VTEP VRF VNID Src IP Dst IP Payload UDP MAC MAC

Tenant Layer-2 Multicast Packet Outer Mcast SRC VTEP BD GIPo BD VNID Src MAC Src IP GIPi Payload UDP MAC

Tenant Layer-3 Multicast Packet Outer Fabric Fabric SRC VTEP VRF GIPo VRF VNID Src IP GIPi Payload UDP MAC MAC

*Multisite, remote leaf, and vPod use Head end replication to send multicast

Underlay Control Plane • ACI uses the GIPo range 225.0.0.0/15 configured during fabric setup by default (configurable) • The underlay multicast groups are IS-IS separate from tenant multicast groups • APIC assigns GIPo addresses to BDs and VRFs • PIM is not used in the underlay • GIPo groups are advertised using IS-IS GM-LSPs

FTAG Root for FTAG Root for FTAG Root for FTAG Root for Underlay Control Plane Tree 1, 5, 9 Tree 2, 6, 10, Tree 3, 7, 11, Tree 4, 8, 12 FTAG 1 FTAG 3 . IS-IS is used to build loop free distribution trees called FTAG trees.

R R . All FTAG trees extend to all nodes (loop free topology) . Multiple trees achieve load balancing across the fabric for multicast traffic . There are 16 FTAG trees but only 12 trees are used for user traffic

Overlay Control Plane • IGMP is used by hosts to join multicast group (version 2, version 3 supported) • COOP is used within the fabric for advertising multicast interest to spines COOP COOP and border leaves • PIM is used on L3Out connections and IGMP IGMP Bridge Domain SVIs PIM

• PIM enabled L3Outs • Non-border leaves must also have see border leaves as loopbacks configured PIM neighbors but • Border leaves form do not send PIM PIM neighbors with hellos themselves PIM other border leaves across fabric

BD1BD3 L3Out L3Out DR BD1 PIM PIM PIM DR PIM PIM PIM • Border leaves run full PIM protocol and peer • Non-border leaves run PIM on with external PIM bridge domain SVIs routers • PIM on non-border leaves runs • PIM enabled L3Outs in passive mode and will not are required for L3 peer with any external devices multicast even if source and receivers are inside the fabric

BRKACI-2608 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 GIPo and FTAG selection . A GIPo address is allocated to BDs and multicast enabled VRFs. . The GIPo address displayed in the GUI is a /28 address. The last four bits of this address are used for the FTAG selection . The actual address sent in the VXLAN packet will be one of the addresses in the /28 range

225.1.22.208 BD1 BD1 EP-A EP-B 1 1 0 1 0 0 0 0

FTAG selection

. When the source leaf sends a multicast Multicast flow hash FTAG 1 VXLAN packet it selects an FTAG using selects FTAG 1. Sends multicast the last four bits in the last octet of the VXLAN packet to group 225.1.122.209 R destination multicast address (GIPo)

225.1.22.209

1 1 0 1 0 0 0 1 EP1 EP2 10.10.10.10 10.10.10.11 Multicast Source Receiver for group 239.1.1.1 FTAG 1

GIPo: 225.1.22.208 FTAG range: 225.1.22.209-220

. When the source leaf sends a multicast Multicast flow hash VXLAN packet it selects an FTAG using selects FTAG 2. FTAG 2 Sends multicast the last four bits in the last octet of the VXLAN packet to group 225.1.122.210 R destination multicast address (GIPo)

225.1.22.210

1 1 0 1 0 0 1 0 EP1 EP2 10.10.10.10 10.10.10.11 Multicast Source Receiver for group 239.1.1.1 FTAG 1

GIPo: 225.1.22.208 FTAG range: 225.1.22.209-220

. Multicast is also used in the overlay for other multi-destination traffic. Non-IP multicast, Spanning tree, Broadcast, Unknown Unicast. The BD GIPo is used for other multi-destination traffic

Non-Multicast Routing Enabled BD Multicast Routing Enabled BD

Broadcast BD GIPo BD GIPo

Unknown Unicast BD GIPo BD GIPo Flood

Multicast BD GIPo VRF GIPo

• In this section layer-2 multicast refers to IP multicast packets forwarded on a layer-2 network segment (BD/subnet)

• It is not Layer-2 non-IP multicast packets. (multicast packets with a destination multicast MAC address without an IP header)

• Also excludes link local multicast (224.0.0.0/24). Link local multicast is always forwarded to all ports in the BD

Supported from release 1.0

Layer-2 multicast is only forwarded within the bridge domain

Forwarded using the BD GIPo address

Layer-2 multicast is supported on all generation leaf and spine switches

BRKACI-2608 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 27 Use case: Layer-2 multicast in bridge domain Multicast sources and receivers in the same subnet (bridge domain) No L3 multicast requirement

• Multicast will be forwarded to all leaf switches where the BD is deployed using the BD GIPo BD GIPo • Each Leaf switch will receive a copy of the multicast stream and forward multicast out front panel Bridge Domain ports based on the BD Unknown Multicast configuration

EP1 EP2 • Flood 10.10.10.10 10.10.10.11 Multicast Source Receiver for • Optimized Multicast Flood group 239.1.1.1

Multicast Receiver source sending sends IGMP multicast report

Unknown Multicast: Flood Multicast is “unknown” when there are no IGMP snooping entries for the group (per leaf)

eth1/1 eth1/2 eth1/1 eth1/2 eth1/1 eth1/2

EP1 10.10.10.10 Multicast Source Bridge Domain

Unknown multicast is forwarded out all ports in the BD

Unknown Multicast: Flood If a group is known (IGMP snooping table entry) streams for that group will be forwarded based on the IGMP snooping table (per leaf)

eth1/1 eth1/2 eth1/1 eth1/2 eth1/1 eth1/2

IGMP Join

10.10.10.10 Receiver for Multicast Source group 239.1.1.1 Bridge Domain

Known multicast forwarded based on IGMP snooping table (per leaf)

Unknown Multicast: Optimized Flood Optimized flood forwards only based on IGMP snooping table across all leaf switches where BD is deployed

eth1/1 eth1/2 eth1/1 eth1/2 eth1/1 eth1/2

IGMP Join

10.10.10.10 Receiver for Multicast Source group 239.1.1.1 Bridge Domain

Unknown multicast is forwarded based on IGMP snooping table

• Optimized Flood is similar to IGMP snooping where multicast is only sent to interested receivers. • Requires an IGMP snooping querier to trigger report messages from hosts

• Flood will forward multicast out all ports in the BD. Receivers will receive multicast in the absence of a querier but can consume bandwidth and result in unnecessary flooding of multicast traffic

Configure an IGMP snooping Enable querier under the BD 1 2 policy for the BD. Enable the subnet querier option

• Fabric supports external querier Leaf# show ip igmp snooping querier Vlan IP Address Version Expires Port • A port that receives an IGMP 12 10.1.1.1. v2 00:04:12 Po1 query or PIM hello will leaf# show ip igmp snooping mrouter become an IGMP snooping Type: S - Static, D - Dynamic, V - vPC Peer Link I - Internal, F - Fabricpath core port mrouter port U - User Configured Vlan Router-port Type Uptime Expires • The external queries will be 12 Po1. D 00:15:24 00:03:36 forwarded out all BD ports • IGMP reports and all multicast traffic will be sent BD1 BD1 BD1 Po1 BD1 BD1 BD1 IGMP Query out mrouter ports PIM Hello

HW Supported spines Scale • All supported spines • Supports up to 8,000 Mroutes (fabric wide Supported Leaf • All 2nd Gen (Cloud Scale) and per leaf) Default • -EX profile • -FX • -GX • Supports up to 32,000 Mroutes (FX only) High Dual Stack profile

• Supported Modes • Inter-VRF multicast 4.0 • PIM-SM (ASM) • Multi-Site support 4.0 • PIM-SSM • PIM and IGMP filters • External Rendezvous Point (RP) • Multicast traffic not filtered by • Static contracts

• Auto-RP • Not supported with remote leaf • BSR 4.0 • Fabric RP (Anycast RP) • No Auto-RP or BSR • Not supported with Multi-Site (coming soon)

• IGMP snooping must be enabled for PIM enabled BDs • Multicast packets will only be forwarded out BD ports if there is an IGMP snooping entry • It is not required to change the L3 Unknown Multicast Flooding setting (this differs from non-PIM enabled BDs)

• L3Outs are a requirement for multicast routing • The L3Out must be configured with a loopback address and have PIM enabled L3Out must have loopback • PIM is supported on L3Outs with routed or routed sub-interfaces and includes Layer-3 port-channels L3Out L3Out PIM PIM BD3 BD3 • PIM PIM Not supported on L3Outs with SVI Multicast Multicast Source Receiver interfaces

L3Out is required even when • For ASM the L3Out provides a path to the source and receiver are inside external RP or can function as the RP (4.0 the fabric feature)

External RP Options

Source leaf BL is always • Static RP (supports multiple RPs) is FHR LHR for fabric PIM Register • Auto-RP (MA Filters)

• BSR (BSR Filters) L3Out L3Out PIM PIM IGMP BD3 BD3 PIM Join Multicast Join Multicast Source Receiver

Source IP for Register packets

Fabric RP

• Anycast RP (equivalent to PIM anycast, does not run MSDP)

RP RP RP RP • Does not support peering anycast L3Out L3Out L3Out L3Out PIM PIM  PIM RP with external RP BD3 BD3 BD3PIM BD3 • All PIM enabled border leaves will become anycast RP members (per VRF)

• Required for inter-VRF multicast

The COOP database maintains multicast group interest in the fabric and is distributed to all active border leaves in the fabric

One BL will be selected as the stripe winner for a multicast group and will be the designated forwarder for the group COOP COOP G1 G2

L3Out L3Out L3Out L3Out PIM PIM PIM Join IGMP IGMP PIM Join BD3 BD3 Join G1:239.10.10.10 G2: 239.11.11.11 Multicast Multicast Join Receiver Receiver G1:239.10.10.10 G2: 239.11.11.11 RP

Fast Convergence is a feature where all border leaves (stripe winners and non stripe winners) will send joins for (*,G) or (S,G). Only the stripe winner will forward the multicast traffic into the fabric

COOP COOP G1 G2

L3Out L3Out L3Out L3Out PIM PIM PIM Join IGMP IGMP PIM Join BD3 BD3 Join G1:239.10.10.10 G2: 239.11.11.11 Multicast Multicast Join G2: 239.11.11.11 G1: 239.10.10.10 Receiver Receiver G1:239.10.10.10 G2: 239.11.11.11 RP/Source

External network has multicast state. Improves convergence time after a failure

ACI switches do not continuously BD1 GIPo send register packets to the RP. Only 225.1.22.208 a single packet is sent to the RP. The BD2 GIPo leaf will send periodic null registers to 225.0.163.208 the RP. VRF GIPo The border leaf seeing that the 225.1.192.64 source is local to the fabric (RPF is COOP VRF GIPo fabric interface) will send a PIM prune 225.1.192.64 towards the RP Register

L3Out L3Out BD1 BD1 BD2 BD2 PIM PIM PIM PIM PIM PIM BD3 BD3 IGMP Join

10.10.10.10 Receiver for PIM group 239.1.1.1 Multicast Join Source VRF1 RP

Multicast in BD without PIM enabled  IGMP Querier (fabric or external)  OMF mode

BD3 GIPo 225.0.163.208 BD3 GIPo 225.0.163.208

L3Out L3Out BD1 BD1 BD3 BD3 BD2 BD2 PIM PIM PIM PIM PIM PIM BD3 BD3

Multicast Multicast Source Receiver

VRF1 RP

BD1 GIPo 225.1.22.208 VRF GIPo 225.1.192.64

VRF GIPo 225.1.192.64

L3Out L3Out BD1 BD1 BD1 BD1 PIM PIM PIM PIM PIM PIM BD3 BD3 IGMP Join

10.10.10.10 Multicast Multicast Receiver PIM Source Join VRF1 RP

Layer-2 multicast forwarding behavior for PIM enabled BDs • Routing first approach. All IP multicast is routed

• TTL will be decremented

SRC MAC TTL SRC IP DST IP Payload SRC MAC TTL SRC IP DST IP Payload VXLAN VXLAN twice. Once on the ingress BD1 MAC 2 10.10.10.10 239.1.1.1 Payload BD1 MAC 1 10.10.10.10 239.1.1.1 Payload node and once on the egress Decrement TTL BD1 VRF1 BD1 Decrement TTL node (regardless of number of transit nodes)

• SRC MAC TTL SRC IP DST IP Payload SRC MAC TTL SRC IP DST IP Payload RP must be defined for ASM EP1 MAC 3 10.10.10.10 239.1.1.1 Payload BD1 MAC 1 10.10.10.10 239.1.1.1 Payload EP2 • EP1 Source MAC will be rewritten 10.10.10.11 10.10.10.10 Receiver for Multicast Source to BD MAC group 239.1.1.1

*Excludes link local multicast (224.0.0/24)

BD1 GIPo 225.1.22.208 VRF GIPo 225.1.192.64

Decrement TTL VRF GIPo Decrement TTL 225.1.192.64

L3Out L3Out BD1 BD1 BD1 BD1 PIM PIM PIM PIM PIM PIM BD3 BD3

10.10.10.10 Multicast Multicast Receiver Source VRF1 RP

BD1 GIPo PIM-SSM Support 225.1.22.208 BD2 GIPo • Default SSM multicast range 225.0.163.208 232.0.0.0/8 VRF GIPo 225.1.192.64 • Configuration of different VRF GIPo COOP 225.1.192.64 SSM range supported

• PIM enabled BDs run

L3Out L3Out IGMPv2. Must enable BD1 BD1 BD2 BD2 PIM PIM PIM PIM PIM PIM BD3 BD3 IGMPv3 in BD IGMP policy IGMPv3 Join • Supports IGMP SSM 10.10.10.10 Receiver for Multicast group 239.1.1.1 Translate (allows IGMP Source Source 10.10.10.10 VRF1 version 2 hosts to join SSM groups)

Multicast traffic is not filtered by contracts Contracts are required only for the routing requirement

1. BD Subnet is not programmed on BL. BL cannot advertise BD subnet out L3Out 2. BD subnet will be programmed on 10.1.1.0/24 static via spine proxy BL only if policy allows traffic to BL 192.168.1.0/24 BGP via BL1 TEP 192.168.1.0/24 via L3out subnet 192.168.1.0/24 BGP via BL2 TEP 10.1.1.0/24 static via spine proxy • Contract • vzAny Contract L3Out L3Out L3Out BD1 PIM PIM • Preferred Group PIM BD BD3subnet: 10.1.1.254/24 BD3 BD3 • Unenforced VRF

192.168.1.0/24

EPG1 C Ext EPG

• MLD snooping support added in release 4.1(1)

• Supports MLDv1 and MLDv2*

• Supports up to 2,000 IPv6 multicast groups across the fabric

• Not supported on 1st generation leaf switches (EX/FX/FX2 only)

• Supports the same flooding modes as IPv4 (Flood/Optimized Flood)

• IPv6 Multicast routing (PIMv6) supported from 4.2(1) release

* MLDv2 forwarded in hardware based on (*,G) lookup

• L3Outs are required for multicast routing • L3Outs must have a Global IPv6 unicast address configured (can be the same loopback used for IPv4/router- id) • IPv6 tenant multicast is carried across BD BD L3Out L3Out PIMv6 PIMv6 PIMv6 PIMv6 BD3 BD3 the fabric tunnel (VRF GIPo IPv4 multicast address) Multicast Multicast • Dual stack will require both an IPv4 and Source Source RP IPv6 unicast loopback address • Only external RP supported • Only external receivers supported in the 4.2(1) release Multicast in a Multi-Pod Fabric ACI Multi-Pod – Underlay Control Plane IPN Configured PIM Bidir for fabric GIPo range vrf context ipn Spines send IGMP joins for ip pim rp-address 10.101.200.1 group-list 225.0.0.0/15 bidir ip pim rp-address 192.168.100.2 group-list 239.255.255.240/28 bidir BD and VRF GIPo groups towards IPN A designated forwarder IPN link is selected across all IGMP IGMP available IPN links PIM BiDir

APIC Cluster

BD1 BD2 L3Out L3Out L3Out L3Out BD2 BD1 PIM PIM PIM PIM PIM PIM PIM PIM

Multicast Multicast Receiver Receiver Group1: Group2: 239.10.10.10 239.20.20.20 Core

BGP EVPN Type 6 route

IPN

PIM BiDir

APIC Cluster COOP COOP COOP

BD1 BD2 L3Out L3Out L3Out L3Out BD2 BD1 PIM PIM PIM PIM PIM PIM PIM PIM PIM PIM (*,G) join (*,G) join Multicast Multicast Receiver Receiver 239.10.10.10 239.20.20.20 Group1: Group2: 239.10.10.10 239.20.20.20 Core

Spine 2 link was IPN selected to send Spine 2 link is the IGMP join for VRF PIM BiDir DF for VRF GIPo GIPo

APIC Cluster

BD1 BD2 L3Out L3Out L3Out L3Out BD2 BD1 PIM PIM PIM PIM PIM PIM PIM PIM

Multicast Multicast Multicast Receiver Receiver Source Group1: Group2: 239.10.10.10 239.20.20.20 Core

Three (or four) steps to enable L3 multicast

1. Enable Multicast at the VRF

2. Enable multicast for BDs where Multicast is required

3. Enable multicast for the L3Out

4. Configure and RP (for ASM)

1 Enable Multicast at the VRF level

2 Enable PIM for 3 Enable PIM for Bridge Domains L3out

4 Configure an RP address (ASM)

Fabric RP Configuration

Auto-RP and BSR options are also supported

Multicast routing is enabled by enabling the VRF, BD, L3out and RP configuration for ASM but additional multicast features are supported

IGMP Features PIM Features PIM Authentication IGMP Report Policy PIM timers IGMP static join PIM Join/Prune filters IGMP fast leaves PIM neighbor filters IGMP state-limit PIM multicast domain boundary IGMP SSM translate Auto-RP BSR

• Stretched BDs with BUM Traffic Enabled (no PIM configuration required) • Within a site the L2 multicast is sent to the BD GIPo multicast address (unique per site)  reaches all the spines and the leaf nodes where the BD is defined (configuration driven) • Spine elected as Designated Forwarder (DF) replicate the stream to each remote sites where the BD is stretched • At the receiving spine the multicast will be sent down the FTAG tree to the receiving site BD GIPo multicast address Inter-Site Network

HREP tunnel destination: 10.100.102.200

Site 1 BD1 VNID  16514962 BD1 GIPo  225.0.195.240

Site 2 BD1 VNID  16711545 BD1 GIPo  225.1.128.160 BD1 BD1 BD1 BD1 BD1 Site 2 Site 1 Source Receiver NOT Receiver Receiver

• Built as Routing-First Approach (decrement TTL at source and destination ACI leaf node) • L3 Multicast is always sent to the VRF GIPo within a site (existing behavior) • Between sites it is sent over the HREP tunnel to the Multicast TEP of the remote sites where the VRF is stretched (the VXLAN header will include the source site VRF VNID) • L3 Multicast at the receiving site will be sent in the VRF GIPo of the receiving site Inter-Site Network

Site1 VRF VNID  2293762 HREP tunnel dest: 10.100.102.200

Site 2 Decrement TTL Decrement TTL VRF VNID 2621443 VRF VRF GIPo  225.1.248.16 Site 1  L3Out VRF VNID 2293762 L3Out  VRF VRF GIPo 225.1.248.32 VRF

BD1 BD2Decrement TTL BD1 BD2 BD2 Site 1 VRF VRF VRF VRF Site 2 Source R1 R3 R4 RP

• Local L3Out must be used to receive traffic from an external source • Multicast traffic from external sources dropped on the spines (to avoid traffic duplication)

Multicast traffic from external Multicast traffic from external Inter-Site sources is dropped on spine. sources is dropped on spine. Network Not sent over HREP tunnels Not sent over HREP tunnels

O-MTEP O-MTEP

Site 2 Decrement TTL VRF VNID 2621443 VRF VRF GIPo  225.1.248.16 Site 1  L3Out VRF VNID 2293762 L3Out  VRF VRF GIPo 225.1.248.32 VRF Decrement TTL BD2Decrement TTL BD1 BD2 BD2 Site 1 VRF VRF VRF Site 2 R1 R3 R4 RP Source BRKACI-2608 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 66 Inter-VRF multicast Inter-VRF Multicast

Allows multicast receivers in one VRF to receive multicast traffic from sources in another VRF

Always forwarded in the source VRF across the fabric

Receiving switch responsible for crossing VRFs

The RPF lookup for the source/RP is done in the source VRF Requires source VRF to be present on all switches where receiver VRF is located

VRF1 VRF2 Note:BL11 The source VRF is not automatically programmed on the switch where the receiver VRF is 20.1.1.10 30.1.1.30 located. Requires user to configure source an EPG on the switch in the source multicast receiver VRF 239.23.1.1

Multicast sent across fabric in source VRF. (VRF1 Source VRF must be GIPo/VRF1 VNID) deployed on all leaf switches where there are receivers

L3out L3out L3out L3out

VRF1 VRF2 VRF1 VRF2 multicast packets arrive VRF1 VRF1 VRF2 at the egress leaf in the source VRF and are Source Receiver Receiver forwarded to receivers in the receiver VRF

VRF1 VRF2 Requires RP in the fabric configured in Static RP configured

the source VRF 20.1.1.10 30.1.1.30 in receiver VRF source multicast receiver Fabric RP: 239.23.1.1 123.1.1.1 RP: 123.1.1.1

Anycast RP in source VRF RP 123.1.1.1

L3out L3out L3out L3out

VRF1 VRF2 VRF1 VRF2

VRF1 VRF1 VRF2

Source Receiver Receiver

30.1.1.30 multicast receiver

If the source VRF is not deployed on the switch multicast traffic will not be received

L3out L3out L3out L3out

VRF1 VRF2 VRF1 VRF2

VRF1 VRF2

Source Receiver Receiver

20.1.1.10 30.1.1.30 multicast receiver RP:123.1.1.1

Source VRF does not need to be deployed Receiver VRF does not on all leaf switches. need to be deployed on Only where there are source leaf receivers in other VRFs

L3out L3out L3out

VRF2 VRF1 VRF2

VRF1 VRF1 VRF2 VRF2 VRF2 no multicast Source Receiver no multicast receivers Receiver receivers

Fabric RP in source VRF

Configure route-map with multicast group range and the VRF where the multicast sources are located

Note: sources for a multicast group cannot be in different VRFs

Leaf-1# show ip mroute 239.10.10.10 vrf CL:vrf IP Multicast Routing Table for VRF "CL:vrf"

(10.10.1.10/32, 239.10.10.10/32), uptime: 00:00:14, ip pim Incoming interface: Tunnel25, RPF nbr: 10.1.184.66 (pervasive) Outgoing interface list: (count: 0)

1. FHR leaf will show (S,G) mroute NBL NBL BL BL with empty OIF for “show ip mroute” command L3Out L3Out 2. The “show forwarding distribution multicast route” command will VRF1 VRF1 display the outgoing tunnel Source interface Receiver 10.10.1.10 10.10.2.10 LeafGroup-1# 239.10.10.10 vsh -c 'show forwarding distributionRP multicast route vrf CL:vrf'

(10.10.1.10/32, 239.10.10.10/32), RPF Interface: Tunnel25, flags: O Received Packets: 2 Bytes: 130 Number of Outgoing Interfaces: 1 Outgoing Interface List Index: 8202 Tunnel25

BRKACI-2608 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 75 Layer-3 ASM Multicast States Last Hop Router (LHR) Leaf-2# show ip mroute 239.10.10.10 vrf CL:vrf IP Multicast Routing Table for VRF "CL:vrf"

(*, 239.10.10.10/32), uptime: 17:08:54, igmp ip pim Incoming interface: Tunnel24, RPF nbr: 10.1.176.64 Outgoing interface list: (count: 1) Vlan104, uptime: 17:08:54, igmp

NBL NBL BL BL

L3Out L3Out

VRF1 1. LHR will onlyVRF1 have a (*,G) mroute entry. Source 2. Does notReceiver require (S,G) because 10.10.1.10 multicast10.10.2.20 is always sent on VRF GIPoGroupto all 239.10.10.10 leaves RP 3. No (S,G) on LHR is expected

Leaf-4# show ip mroute 239.10.10.10 vrf CL:vrf IP Multicast Routing Table for VRF "CL:vrf"

Group not found

NBL NBL BL BL

L3Out L3Out Leaf-3# show ip mroute 239.10.10.10 vrf CL:vrf 1. An (S,G) mroute will be created on IP Multicast Routing Table for VRF "CL:vrf" the border leaf with the (*,G) VRF1 VRF1 mroute (*, 239.10.10.10/32), uptime: 1d14h, ngmvpn ip pim 2. The (*,G) when there is multicast Incoming interface:Source Ethernet1/16.89, RPFReceiver nbr: 192.168.7.1 Outgoing interface10.10.1.10 list: (count: 1) (Fabric10.10.2.20 OIF) interest (Internal or external) Tunnel17, uptime: 1d14h, ngmvpnGroup 239.10.10.10 RP 3. When the receiver is in the fabric the BL selected for the (*,G) is the stripe winner (10.10.1.10/32, 239.10.10.10/32), uptime: 1d14h, pim mrib ip Incoming interface: Tunnel17, RPF nbr: 10.1.184.66 (pervasive) Outgoing interface list: (count: 1) Tunnel17, uptime: 1d14h, mrib, (RPF)

NBL NBL BL BL

L3Out L3Out

Leaf-4# show ip mroute 239.10.10.10 vrf CL:vrf IP Multicast Routing Table for VRF "CL:vrf" VRF1 VRF1

(10.10.1.10/32, 239.10.10.10/32),Source uptime: 00:00:10,Receiver pim ip Incoming interface:10.10.1.10 Tunnel17, RPF nbr: 10.1.184.6610.10.2.20 (pervasive) Outgoing interface list: (count: 1) Receiver Group 239.10.10.10 RP Ethernet1/16.155, uptime: 00:00:10, pim

Stripe winner for group 239.10.10.10 1. Command to find stripe winner can be executed on any leaf 2. Hash basedNBL on group and VRF VNIDNBL BL BL

L3Out L3Out

Leaf-3# vsh -c 'show ip pim internal stripe-winner 239.10.10.10 vrf CL:vrf' PIM Stripe WinnerVRF1 info for VRF "CL:vrfVRF1" (BL count: 2) (*, 239.10.10.10)Source Receiver BLs: 10.10.1.10 10.10.2.20 Group hash 1262394260 VNID 3047428 Group 239.10.10.10 1.1.1.103 hash: 1948335499 (local) RP 1.1.1.104 hash: 725726570 Winner: 1.1.1.103 best_hash: 1948335499

Leaf-1# show ip mroute vrf CL:vrf IP Multicast Routing Table for VRF "CL:vrf"

(*, 232.0.0.0/8), uptime: 2d11h, pim ip Incoming interface: Null, RPF nbr: 0.0.0.0 Outgoing interface list: (count: 0)

NBL NBL BL BL 1. For SSM multicast the FHR can send SSM multicast on the fabric L3Out L3Out tunnel immediately 2. SSM does not use an RP so there VRF1 VRF1 is no RP lookup needed (no Source register packets Receiversent for SSM) 10.10.1.153. No (S,G) state on10.10.2.20 source leaf is expected sinceGroup all232.30.30.30 multicast is RP Source: 10.10.1.5 forwarded on fabric tunnel

BRKACI-2608 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 80 Layer-3 SSM Multicast Leaf-2# show ip mroute vrf CL:vrf Last Hop Router (LHR) IP Multicast Routing Table for VRF "CL:vrf"

(*, 232.0.0.0/8), uptime: 2d11h, pim ip Incoming interface: Null, RPF nbr: 0.0.0.0 Outgoing interface list: (count: 0)

(10.10.1.5/32, 232.30.30.30/32), uptime: 00:16:14, igmp ip pim Incoming interface: Tunnel24, RPF nbr: 10.1.184.66 (pervasive) Outgoing interface list: (count: 1) Vlan104, uptime: 00:16:14, igmp NBL NBL BL BL

L3Out L3Out

1. Last Hop Router leaf will have (S,G) mrouteVRF1 entry VRF1

2. SourceThe (S,G) state is created Receiverby the 10.10.1.15(S,G) IGMP join 10.10.2.20 Group 232.30.30.30 RP Source: 10.10.1.5

BRKACI-2608 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 81 Layer-3 ASM Multicast Troubleshooting Is source leaf forwarding multicast? Leaf-2# show ip mroute 239.10.10.10 detail vrf CL:vrf IP Multicast Routing Table for VRF "CL:vrf"

(10.10.1.10/32, 239.10.10.10/32), uptime: 00:00:37, ip(0) pim(0) Data Created: Yes Received Register stop Pervasive ? VPC Flags 1. If register stop is not received this RPF-Source Forwarder indicates a possibleNBL configuration NBL Stats: 0/0 [Packets/Bytes],BL 0.000BL bps Incoming interface: Tunnel25, RPF nbr: 10.1.184.66 (pervasive) or routing issue L3Out L3Out 2. The FHR will not forward multicast Outgoing interface list: (count: 0) if the RP is not configured ”show ip pim rp” commandVRF1 to check VRF1Leaf-2# show ip mroute 239.10.10.10 detail vrf CL:vrf 3. The FHR will not forward multicast IP Multicast Routing Table for VRF "CL:vrf" Source Receiver if the there is10.10.1.15 no route to the RP 10.10.2.20 “show ip route” command to (10.10.1.10/32, 239.10.10.10/32), uptime: 00:00:37, ip(0) pim(0) Data Created: Yes RP check Pervasive 4. The FHR will forward multicast if VPC Flags the RP is configured and there is a RPF-Source Forwarder route to the RP Stats: 0/0 [Packets/Bytes], 0.000 bps Incoming interface: Tunnel25, RPF nbr: 10.1.184.66 (pervasive) Outgoing interface list: (count: 0)

Multicast not received Leaf-2# show ip mroute 239.10.10.10 vrf CL:vrf IP Multicast Routing Table for VRF "CL:vrf" 1. Receiver has joined multicast group (sent IGMP join report) (*, 239.10.10.10/32), uptime: 00:00:12, igmp ip pim 2. Incoming interface is Null indicating Incoming interface: Null, RPF nbr: 0.0.0.0 multicast is not being received Outgoing interface list: (count: 1) 3. RPF is 0.0.0.0 ? Vlan104, uptime: 00:00:12, igmp NBL NBL BL BL

L3Out L3Out

Multicast received Leaf-2# show ip mroute 239.10.10.10 vrf CL:vrf VRF1 VRF1IP Multicast Routing Table for VRF "CL:vrf" 1. Incoming interface shows fabric tunnel interfaceSource Receiver(*, 239.10.10.10/32), uptime: 00:03:13, igmp ip pim 10.10.1.15 2. RPF neighbor is spine anycast proxy 10.10.2.20Incoming interface: Tunnel24, RPF nbr: 10.1.176.64 address. Outgoing interface list: RP(count: 1) Vlan104, uptime: 00:03:13, igmp

COOP COOP NBL BL BL

L3Out L3Out

1. Receiver sends IGMP join to leaf IGMP Join VRF1 VRF1 PIM Join 2. LHR sends COOP message to spine notifying spine of interest Receiver 10.10.2.20 RP 3. Spine notifies BL of multicast Group 239.20.20.20 interest 4. BLs run hash to determine stripe Source winner for group/VRF 192.168.1.1

Leaf-1# show ip pim neigh vrf CL:vrf 1. No PIM neighbors will be formed over the fabric PIM Neighbor information for Dom:CL:vrf Neighbor Interface Uptime Expires DRPriority Bidir BFDState COOP COOP NBL BL BL

L3Out L3Out

1. Receiver sends IGMP join to leaf Leaf-103#IGMP show Join ip pim internal stripe-winner 239.10.10.11 vrf VRF1 VRF1CL:vrf 2. LHR sends COOP message to spine PIM Stripe Winner info for VRF "CL:vrf" (BL count: 0) notifying spine of interest Receiver(*, 239.10.10.11) 10.10.2.20 BLs: RP 3. Spine notifies BL of multicast Group 239.20.20.20 interest Group hash 158879015 VNID 3047428 Winner: 255.251.249.253 best_hash: 0 4. No stripe winner. Join not sent Source towards external RP/Source 192.168.1.1

Leaf-1# show ip pim neigh vrf CL:vrf 1. PIM neighbors formed over the PIM Neighbor information for Dom:CL:vrf Neighbor Interface Uptime Expires DRPriority Bidir BFDState fabric 1.1.1.104/32 tunnel25 00:00:08 00:01:36 1 no n/a 1.1.1.103/32 tunnel25 00:00:08 00:01:42COOP 1 no n/a COOP NBL BL BL

L3Out L3Out

1. Receiver sends IGMP join to leaf Leaf-103#IGMP show Join ip pim internal stripe-winner 239.20.20.20 vrf VRF1 VRF1 2. LHR sends COOP message to spine CL:vrf PIM Stripe Winner info for VRF "CL:vrf" (BL count: 2) notifying spine of interest Receiver 10.10.2.20(*, 239.20.20.20) 3. Spine notifies BL of multicast BLs: RP interest Group hash 738275646 VNID 3047428 4. BLs run hash to determine stripe 1.1.1.103 hash: 426404817 (local)Source winner for group/VRF 1.1.1.104 hash: 1470373220192.168.1.1 Winner: 1.1.1.104 best_hash: 1470373220

Outer destination address is 225.1.193.33 (FTAG 1)

BRKACI-2608 © 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public 91 Complete your online session • Please complete your session survey survey after each session. Your feedback is very important.

• Complete a minimum of 4 session surveys and the Overall Conference survey (starting on Thursday) to receive your Cisco Live t-shirt.

• All surveys can be taken in the Cisco Events Mobile App or by logging in to the Content Catalog on ciscolive.com/emea.

Cisco Live sessions will be available for viewing on demand after the event at ciscolive.com.

Demos in the Walk-in labs Cisco campus

Meet the engineer Related sessions 1:1 meetings