#CLUS Packet Journey Inside ASR 9000 Mike Mikhail, Solutions Arch [email protected] BRKARC-2017 #CLUS Acknowledgement • Content • Jeff Byzek, Technical Marketing Engineer • Gawel Mikolajczyk, Technical Solutions Architect • David Pothier, Enterprise Architect • Xander Thuijs, Principal Engineer • Mei Zhang, Technical Leader • Review • David Pothier, Enterprise Architect • Matt Breneisen, Solutions Integration Architect • Neel Shah • Xander Thuijs, Principal Engineer #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 #CLUS © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public Cisco Webex Teams Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session How 1 Find this session in the Cisco Events App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space Webex Teams will be moderated cs.co/ciscolivebot#BRKARC-2017 by the speaker until June 18, 2018. #CLUS © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Agenda System architecture overview: Control & forwarding paths Control and exception traffic: Internal forwarding, and security Transit frame forwarding: L3/L2 unicast/multicast in hardware MPLS operation: Forwarding & service labels in hardware Troubleshooting: Counters, discards, and packet/frame capture #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 Introduction • About me • Mike Mikhail, Solutions Architect, [email protected] • Available at “Meet the Engineer” for 1:1 discussions • Interests: SP platforms, SP technologies, QoS, SDN, NFV • ASR 9000 today • Very popular SP/WAN edge router • High bandwidth Ethernet services platform • Hw forwarding, high density: ~1B pps per LC, >1Tbps/slot • Distributed processing and distributed forwarding • Continued dev, busy roadmap, a wealth of new features David #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 Agenda System architecture overview: Control & forwarding paths Control and exception traffic: Internal forwarding, and security Transit frame forwarding: L3/L2 unicast/multicast in hardware MPLS operation: Forwarding & service labels in hardware Troubleshooting: Counters, discards, and packet/frame capture #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 1 System Architecture 1.5 Tbps/slot ASR 9000 Family Chassis, cards, power, air flow “Fixed” hw: RP+Fabric +LC+ports ASR 9901 ASR 9904 ASR 9906 ASR 9010 ASR 9910 ASR 9912 ASR 9922 RP Built-in 1+1 RSP 1+1 RSP 1+1 RSP 1+1 RSP 1+1 RP 1+1 RP Fabric Built-in 2x RSP 6+1 2x RSP 6+1 6+1 6+1 Line cards 16x1G + 24x1/10G 2 4 8 8 10 20 or ports + 2x100G Rack units 2 6 14 21 21 30 44 Power modules 2x AC or 2x DC 4x AC or 4x DC 3x AC or 4x DC 8x AC or 8x DC 8x AC or 8x DC 12x AC or 12x DC 16x AC or 16x DC Air flow Front to back Right to left Front to back Front to back Front to back Front to back Front to back #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 1.5 Tbps/slot ASR 9000 Models Chassis, cards, power, air flow “Fixed” hw: RP+SP+LC+ ports+bays ASR 9001 ASR 9904 ASR 9006 ASR 9010 ASR 9910 ASR 9912 ASR 9922 RP Built-in 1+1 RSP 1+1 RSP 1+1 RSP 1+1 RSP 1+1 RP 1+1 RP Fabric Built-in 2x RSP 2x RSP 2x RSP 6+1 6+1 6+1 Line cards & 4x SFP+ 2 4 8 8 10 20 ports 2x MPA Rack units 2 6 10 21 21 30 44 Power modules 2x AC or 2x DC 4x AC or 4x DC 4x AC or 4x DC 8x AC or 8x DC 8x AC or 8x DC 12x AC or 12x DC 16x AC or 16x DC Air flow Right to left Right to left Right to back Front to back Front to back Front to back Front to back #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 1.5 Tbps/slot ASR 9000 Models Chassis, cards, power, air flow With optional baffle “Fixed” hw: RP+SP+LC+ ports+bays ASR 9001 ASR 9904 ASR 9006 ASR 9010 ASR 9910 ASR 9912 ASR 9922 RP Built-in 1+1 RSP 1+1 RSP 1+1 RSP 1+1 RSP 1+1 RP 1+1 RP Fabric Built-in 2x RSP 2x RSP 2x RSP 6+1 6+1 6+1 Line cards & 4x SFP+ 2 4 8 8 10 20 ports 2x MPA Rack units 2 10U 2-post 23” 10 21 21 30 44 Power modules 2x AC or 2x DC 4x AC or 4x DC 4x AC or 4x DC 8x AC or 8x DC 8x AC or 8x DC 12x AC or 12x DC 16x AC or 16x DC Air flow Right to left Front to back Right to back Front to back Front to back Front to back Front to back #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 System Architecture 9904, 9006, 9010 . Distributed control Line Card plane RSP – L2 protocols, ARP, BFD, CFM, Netflow run on LC CPU C CPU P BITS/DTI . Distributed data U FIA plane FIC – Forwarding distributed to NP’s . Active-active switch fabric – Each RSP houses “half” of the fabric Switch Fabric #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 The Switch Fabric 9010, 9006 . 3-Stage fabric – Allows for variable number of FIA’s and FIA links on LC . Super-framing for unicast – Super-frame same-priority same-egress frames in a jumbo frame . 880[440] Gbps per slot – 4x 110[55] Gbps links per slot per RSP – 440[220] Gbps if an RSP is removed 110G links [RSP880] . Fabric load sharing 110G links [RSP880] 55G55G links links [RSP440] [RSP440] – Unicast: per super-frame – Multicast: per flow #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 The Switch Fabric 9922, 9912, 9910, 9906 Fabric Stage Fabric Fabric Stage Fabric Linecard . 3-Stage fabric Linecard – Allows for variable number of FIA’s and FIA links on LC . Super-framing for unicast – Super-frame same-priority same-egress frames in a jumbo frame Fabric Stage Fabric . Bandwidth per slot: Stage Fabric Linecard Linecard – 2x 110[55] Gbps links per slot per fabric card – 2 fabrics on RSPs for 9910, 9906 . Fabric load sharing – Unicast: per super-frame Fabric cards 110G links [RSP880] 55G links [RSP440] – Multicast: per flow #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 The Switch Fabric 9922, 9912, 9910, 9906 On RSPs Fabric Stage Fabric Fabric Stage Fabric Linecard . 3-Stage fabric Linecard – Allows for variable number of FIA’s and FIA links on LC . Super-framing for unicast – Super-frame same-priority same-egress frames in a jumbo frame Fabric Stage Fabric . Bandwidth per slot: Stage Fabric Linecard Linecard – 2x 110[55] Gbps links per slot per fabric card – 2 fabrics on RSPs for 9910, 9906 . Fabric load sharing – Unicast: per super-frame Back of Fabric cards 110G links [RSP880] mid-plane 55G links [RSP440] – Multicast: per flow #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 Slot Address Internal slot address, slot mask, and fabric group ID Slot Slot Mask Slot Slot Mask Logical Physical Binary Hex Logical Physical Binary Hex LC7 9 1000000000 0x0200 LC3 5 0000100000 0x0020 9006 LC2 4 0000010000 0x0010 LC6 8 0100000000 0x0100 LC1 3 0000001000 0x0008 LC5 7 0010000000 0x0080 LC0 2 0000000100 0x0004 LC4 6 0001000000 0x0040 RSP1 1 0000000010 0x0002 RSP0 5 0000100000 0x0020 9010 RSP0 0 0000000001 0x0001 RSP1 4 0000010000 0x0010 Slot Slot Mask LC3 3 0000001000 0x0008 Logical Physical Binary Hex LC2 2 0000000100 0x0004 LC19 21 10000 00000000 0x10 00000000 0000 LC1 1 0000000010 0x0002 9912/9922 LC1-18 3-20 LC0 0 0000000001 0x0001 LC0 2 0000000100 0x0004 . Follows the sequence of slots in chassis RP1 1 0000000010 0x0002 . 4/6/12/22 RP slots: 000011 which is 0x0003, decimal 3 RP0 0 0000000001 0x0001 . 9010 RSP slots: 0000110000 which is 0x0030, decimal 48 #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 Fabric Link status RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers fabric arbiter linkstatus 3 location 0/RSP0/CPU0 Wed Nov 20 20:44:29.615 EST Slot 3 PG:9 up RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers fabric arbiter linkstatus 0 location 0/RSP0/CPU0 Wed Nov 20 20:44:22.995 EST Slot 0 PG:24 up Internal slot number RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers fabric crossbar link-status instance 1 location 0/RSP0/CPU0 Tue Dec 17 02:59:30.110 EST PORT Remote Slot Remote Inst Logical ID Status ====================================================== 02 0/3/CPU0 00 0 Up 16 0/0/CPU0 00 0 Up LC3 in 9010 18 0/2/CPU0 00 1 Up LC0 in 9010 20 0/2/CPU0 00 0 Up 24 0/3/CPU0 00 1 Up #CLUS BRKARC-2017 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 Fabric Link statistics Instance 0 Instance 1 RP/0/RSP0/CPU0:rasr9000-2w-a#show controllers fabric crossbar statistics instance 1 location 0/RSP0/CPU0 Tue Dec 17 02:59:36.376 EST Port statistics for xbar:1 port:2 ============================== Hi priority stats (unicast) =========================== Ingress Packet Count Since Last Read : 17347357500 Ingress Channel Utilization Count : 5 Output Buffer Queued Packet Count : 1 Egress Packet Count Since Last Read : 19006087016 Egress Channel Utilization Count : 4 .