ASR9000 Selected Topics and Troubleshooting

BRKSPG-2904

Aleksandar Vidakovic, Emeritus CCIE 5697 Technical Leader (Engineering, SP Routing) Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

How 1. Find this session in the Cisco Live Mobile App 2. Click “Join the Discussion” 3. Install Spark or go directly to the space 4. Enter messages/questions in the space

cs.co/ciscolivebot#BRKSPG-2904

• cXR to eXR migration • QOS: shared and flow aware policers

• Fabric interworking 5 vs 7 • Software Management

• Netflow • Automatic Intf Shutdown on LC Insert

• PWHE Load Balancing and • Dealing with NP Lockups Troubleshooting XR Embedded Packet Tracer XR Embedded Packet Tracer

• XR Embedded Packet Tracer (EMT) empowers users to: • Follow a given packet flow through the router • Perform in-depth triaging of packet forwarding issues in data, punt and inject path

• Common Packet Tracer framework used by all XR platforms • Common slow path tracing capabilities across all platforms • Data-path tracing capabilities differ between platforms

• Key functionalities: • Trace packets through the router. • User defined conditions to match packets of interest. • Network Protocol agnostic (supports offset/value/mask as condition) • Works from user (cisco-support privilege) mode (no debug nor configuration required) • Provide insight into the features executed on the packet, with timestamps and feature processing result (phase II)

Packets Marked : 1000 NPU Packets Dropped : 293 Data-path elements Packets To Fabric: 707 Data-path elements registered with the transparent to the Packet Packet Tracer Tracer framework framework (implicit information) (explicit information) Fabric

Packets From Fabric : 649 NPU Packets Dropped : 478 Packets Punted to LC CPU: 50 Packets to Interface : 121

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 Detailed Trace Packet seq #1 Marked on Te0/1/0/0 1 at Oct 31 14:35:54.579, pass Packet seq #1 QoS Classification at Oct 31 14:35:54.580, classified against class class-default, pass Packet seq #1 ACL processing at Oct 31 14:35:54.580, access-list name My_ACL_In, pass Packet seq #1 L3 Forwarding decision at Oct 31 14:35:54.580, next-hop Te0/6/0/0, pass Packet seq #1 sent to Fabric at Oct 31 14:35:54.581, pass Sequence number Action Timestamp Attribute

Fabric

Packet seq #1 Received from Fabricat Oct 31 14:35:54.582, pass ... Packet seq #1 QoS Classification at Oct 31 14:35:54.582, classified against class voice, pass Packet seq #1 Egress policer at Oct 31 14:35:54.582, drop

• Every marked packet is assigned a unique sequence number

• In detailed mode only one packet marking element allowed in the system

• Not all platforms will support this mode! (fabric header must carry the seq#)

• Syntax: packet-trace condition

• Supported filter types: • Raw (Offset/Value/Mask): packet-trace condition interface Te0/2/0/0 packet-trace condition offset 12 value 0x8847 mask 0xFFFF packet-trace condition offset 14 value 0x04231000 mask 0xFFFFF000 Logical AND packet-trace condition offset 18 value 0x03E81000 mask 0xFFFFF000

• Canonical: packet-trace condition interface Te0/2/0/0 packet-trace condition ipv4 offset 22 packet-trace condition ipv4 source 10.10.10.10/32 Logical AND packet-trace condition ipv4 destination 10.20.20.0 mask 255.255.252.0 packet-trace condition tcp port source 80

• Platforms: asr9k

• Capabilities: data-path tracing only, basic counters

How useful do you find this feature? Leave us your feedback via Cisco SPARK or supporforums

• Evolved XR -- Linux 64bit kernel • More memory per process (BGP?) with VM’s • Linux tools (install using RPM) • Starting with 6.1.x, eXR and cXR images are built for ASR9000. • New hardware (e.g. Lightspeed line card for asr9000) • NCS5500 runs eXR only • Certain functionality (e.g. ISSU, • Typhoon and Tomahawk together Golden ISO) can only run classic XR 32 bit.

• eXR requires: • Tomahawk or Lightspeed line cards • RSP880 / RP2

• Also used by Cisco test teams for migration, upgrade and patch install

• Documentation included in the CSM Server installation

• CSM Server Demo: Great overview in • https://www.youtube.com/watch?v=isxN08x-mr4 only 4 minutes! • CSM Server documentation: • https://supportforums.cisco.com/document/13154846/cisco-software-manager-csm-33-overview-documentation

• Migration documentation: • http://www.cisco.com/c/en/us/td/docs/routers/asr9000/migration/guide/b-migration-to-ios-xr-64-bit.html

• Video "ASR9K IOS XR 32 bit to 64 bit Migration using CSM Server": • https://youtu.be/RVgR0TdbpVw NEW!!!

• Allows line rate performance for 12X100GE Skyhammer linecard • 5 fabrics (230 Gbps each, total 1150 Gbps) would not suffice to fill all 12 ports

• Gives better fabric redundancy for 8X100GE linecards • System can lose two more fabrics before throughput gets impacted

• Enables line rate interoperability between 12X100GE and 8X100GE linecards in corner cases • When 12X100GE card sends traffic to 5-fabric 8X100GE cards, only the first five fabrics can be used. So the 12X100GE card can not send 1.2 Tbps linerate to 5-fabric 8X100GE cards.

• A9K-… linecards are 5 fabric (48x10, MOD200/400, 4x100, A9K-8x100)

• A99K… linecards are 7 fabric (A99-8x100, 12x100)

• If Ingress and Egress Linecards are 7-Fabric cards: • Unicast traffic will use all 7 fabric cards

• If Ingress and/or Egress Linecard is 5-Fabric card: • Unicast traffic will use the first 5 fabric cards

• With mixed 5/7-Fabric linecards present, the system will *not* go down to the lowest common denominator!

The system always makes the best use of the available resources!

Interlaken 2x 8x15G = 240Gbps raw BW 2x 16x 10.9375Gbps 3x Fencer ports = 350G raw BW

QSFP 0 PHY NP FIA QSFP 1 CPU

QSFP 2 PHY NP FIA QSFP 3

QSFP 4 PHY NP FIA QSFP 5 Switch Fabric (SM15) …

QSFP 6 PHY NP FIA Up to 14x115G QSFP 7

QSFP 8 PHY NP FIA QSFP 9

QSFP 10 PHY NP FIA QSFP 11

• No external TCAM on this card. Only 5Mb internal TCAM • Due to limited TCAM only L3 Transport/LSR features supported

Traffic is using the first 5 fabric cards Each arrow represents 230 Gbps because both linecards can not reach fabric card 5&6.

5-Fabric 5-Fabric Tomahawk Tomahawk

Line Card SM15 SM15 Line Card

SFC2 Fabric cards

Traffic is using the first 5 fabric Each arrow represents 230 Gbps cards because the ingress linecard can not reach fabric card 5&6.

5-Fabric 7-Fabric Tomahawk Tomahawk

Line Card SM15 SM15 Line Card

Fabric cards 5&6 are still used to reach other 7-fabric linecards! SFC2 Fabric cards

Traffic is using the first 5 fabric cards because the egress linecard can not reach Each arrow represents 230 Gbps fabric card 5&6.

7-Fabric 5-Fabric Tomahawk Tomahawk

Line Card SM15 SM15 Line Card

Fabric cards 5&6 are still used to reach other 7-fabric linecards! SFC2 Fabric cards

Each arrow represents 230 Gbps Traffic is using all fabric cards!

7-Fabric 7-Fabric Tomahawk Tomahawk

Line Card SM15 SM15 Line Card

SFC2 Fabric cards

• Mixed 5/7-Fabric Linecards present in a chassis • Multicast traffic will use the first 5 fabric cards

• Only 7-Fabric Linecards present in a chassis • Per default, Multicast traffic will still use the first 5 fabric cards • System can be configured through CLI to use all 7 fabric cards • If 5-Fabric card is inserted into chassis later on, the card will not boot up

RP/0/RSP1/CPU0:ASR9K-2(admin-config)#fabric enable mode ? A99-highbandwidth A99 High bandwidth cards only highbandwidth High bandwidth cards only

• Default: • Max 1024 VQI per system • Multicast traffic uses the first 5 fabric cards

• Highbandwidth: • Max 2048 VQI per system ( only Tomahawk and RP2 allowed) • Multicast traffic uses the first 5 fabric cards

• A99-highbandwidth: • Max 2048 VQI per system • Multicast traffic uses all 7 fabric cards ( only A99 Tomahawk and RP2 allowed)

5x230G = 1.15Tbps 12x120G = 1.44Tbps 12x100G = 1.2Tbps

8x100G 5-Fabric Tomahawk Line Card FIA NP 8x100G 5-Fabric FIA NP Tomahawk Line Card FIA NP

8x100G 5-Fabric SM15 FIA NP Tomahawk Line FIA NP Card FIA NP

12x100G Tomahawk Line Card Each arrow represents 230 Gbps 1.61Tbps

SFC2 Fabric cards

5x230G = 1.15Tbps 12x120G = 1.44Tbps 12x100G = 1.2Tbps

8x100G 5-Fabric FIA shaper must be adjusted Tomahawk Line Card FIA NP 8x100G 5-Fabric FIA NP Tomahawk Line Card FIA NP

8x100G 5-Fabric SM15 FIA NP Tomahawk Line FIA NP Card FIA NP

12x100G Tomahawk Line Card Each arrow represents 230 Gbps 1.61Tbps

SFC2 Fabric cards

• A99 chassis with 5 fabric cards or more: • 83Gbps per 100G port (total of 996 Gbps; fabric conn 5x230Gbps = 1.15Tbps Gbps)

• Any chassis with 4 fabric cards (asr9010, asr9006 with dual RSP880): • 71Gbps per 100G port (total of 852 Gbps; fabric connection 4x230Gbps = 920 Gbps)

• Syslog: • LC/0/0/CPU0:Dec 27 12:05:16.429 EST: pfm_node_lc[299]: %FABRIC-FIA-1- RATE_LIMITER_ON : Set|fialc[163907]|0x1072000|Insufficient fabric capacity for card types in use - FIA egress rate limiter applied

• Checking the shaper rate: • show controllers fabric fia information location • show controller fabric fia trace location | include “shape_RL"

RP/0/RSP1/CPU0:WEST-PE_ASR9K-2#admin show inventory | i Chassis NAME: "chassis ASR-9010-AC", DESCR: "ASR 9010 8 Line Card Slot Chassis with V1 AC PEM" RP/0/RSP1/CPU0:ASR9K-2#show controllers fabric fia information location 0/7/CPU0 ********** FIA-0 ********** Category: bandwidth-0 One port in admin down BW if0 120 BW if1 23  Full BW allowed to the ********** FIA-1 ********** other port Category: bandwidth-1 BW if0 23 BW if1 120 ********** FIA-2 ********** Category: bandwidth-2 BW if0 0 Slice powered down BW if1 0 ********** FIA-3 ********** Category: bandwidth-3 BW if0 71 BW split between two ports BW if1 71

• FIA policer can be disabled on A99 chassis

• Configuration command (note: not an admin command): • hw-module fia-intf-policer disable

• When can I use it? • When most traffic is between 12x100G line cards • E.g.: Only one line card in chassis is a 5-fabric card • E.g.: All 5-fabric cards in the chassis are MOD200

ASR9010/ASR9006 with 2x RSP440 ASR9904 with 2x RSP440

Fabric BW per Slot Fabric BW per Slot 440Gbps 550Gbps w Dual RSP440s w Dual RSP440s

Total LC Data BW Total LC Data BW 400Gbps 480Gbps

Slice, CPAK Port Slice, CPAK Port BW w 4 Slices in 100Gbps 50Gbps BW w 4 Slices in 124.5Gbps 62Gbps Use Use Slice, CPAK Port Slice, CPAK Port BW w 3 Slices in 130Gbps 65Gbps BW w 3 Slices in 166Gbps 83Gbps Use Use Slice, CPAK Port Slice, CPAK Port BW w 2 Slices in 200Gbps 100Gbps BW w 2 Slices in 200Gbps 100Gbps Use Use

RSP0 Crossbar Fabric 5: credit return 1: Fabric Request ASIC Crossbar Fabric ASIC

Arbitration Fabric Interface Fabric Interface and VOQ and VOQ 2: Arbitration

Crossbar Fabric 3: Fabric Grant ASIC Crossbar 4: load-balanced Fabric transmission across ASIC fabric links Arbitration RSP1

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 32 System QoS Refresh (3) – Backpressure and VoQ Mechanism Egress NP congestion   backpressure to ingress FIA  Packet is en-queued in the dedicated VoQ  No impact of the packet going to different egress NP  One VoQ set (4 queues) for each 10G entity in the system No head-of-line-block issue (10/100G in Tomahawk) Backpressure: egress NP  egress FIA  fabric Arbiter  ingress FIA  VoQ

Ingress side of LC1 Egress side of LC2 CPU CPU 1 10GbpsPHY NP0 NP0 PHY 2 5Gbps PHY NP1 5Gbps NP1 PHY FIA FIA PHY NP2 Switch NP2 PHY 3 Fabric PHY NP3 NP3 PHY

Packet going to different egress NP put into different VoQ set  Congestion on one NP won’t block the packet going to different NP

ASR 9922/9912 ASR 9904 ASR 9010/9006 ASR 9910 RP1 or RP2 LC Type SFC1 SFC2 RSP440 RSP880 RSP440 RSP880 RSP/SFC 9910 8x100GE Y Y Y Y Y Y Y 4x100GE Y Y Y Y Y Y Y 400GE IPoDWDM Y Y Y Y Y Y Y MOD400/MOD200 Y Y Y Y Y Y Y 12x100GE (7-fab) N Y N Y N N Y 8x100GE (7-fab) Y Y Y Y N N Y 24x10/1G / 48x10/1G Y Y Y Y Y Y Y Typhoon + Tomahawk Y Y Y Y Y Y Y Trident + Tomahawk N N N N N N N SIP-700 + Tomahawk N N Y Y Y Y N VSM Y Y Y Y Y Y Y

ASR 9922/9912 ASR 9904 ASR 9010/9006 ASR 9910 RP1 or RP2 LC Type SFC1 SFC2 RSP440 RSP880 RSP440 RSP880 RSP/SFC 9910 8x100GE Y Y Y Y 9904Y doesn’t haveY Y fabric cards, but has 4x100GE Y Y Y Y Y Y Y some extra fab 400GE IPoDWDM Y Y Y Y chansY to the 2 Yslots Y MOD400/MOD200 Y Y Y Y Y Y Y 12x100GE (7-fab) N Y N Y N N Y Trident/RSP28x100GE (7-fab) no Y Y Y Y N N Y longer24x10/1G supported / 48x10/1G Y Y Y Y Y Y Y Typhoonafter 534 + Tomahawk Y Y Y Y Y Y This quad “N”Y is because the 90xx Trident + Tomahawk N N N N N N N chassis only wire for SIP-700 + Tomahawk N N Y Y Y Y “5”fab lanes. 2N are SIP700 doesn’t VSM Y Y Y Y Y Y not reachable.Y understand all these fabric cards

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 35 Netflow Netflow High Level Architecture Every packet sampled gets looped. PPS utilization of netflow directly related to sampling rate. 1:1 sampling is ~50% NPU perf degradation. Cache resides on LC CPU (regardless phy or bundle)

• Sampled packets punt rate 200kpps on Ty/Th (shared) between all NPs

• Packets that exceed the punt rate are dropped by LPTS

Queueing Parse Search Resolve Modify Scheduling

To LC CPU • Resolve takes care of the sampling and looping

• Resolve is heavily loaded responsible for feature application (policer, mark, pw resolution, acl actions, fib next hop etc).

• Netflow replication is heavy, the sampling algorithm is also.

• Implemented by a counter for very specific deterministic sampling providing very high accuracy! • Affects racetrack performance

• At sample levels of 2^n a random number method is used improving performance but less accuracy in the sampling period. E.g. sampling rate 65536, 2048, 4096, 8192, 16384, 32768. • Use -1/+1 on these values to get back to number based sampling for higher accuracy.

• ALL packets are subject to sample, drop or forward or punt!

Netflow Netflow Manager RP MA Show/Clear Command Show/Clear Command

Control Plane Netflow MA LC (nfma) Config Path

Netflow EA (nfea) n Data Plane nfea PD DLL Flow Path

NetFlow Server (nfsvr) NETIO

PI Component Netflow Producer (nf_producer) FIB mgr

PD Component SPP/SPIO

Hardware Flow Packets Program CEF Program uidb Export Packets to From NP the external collector

Parameter Location Purpose Pros Cons

(higher) sampling NPU How often packets get sampled for High values (e.g. 1:1) provide Low values reduce visibility of rate correlation in the cache most accurate detail but overrun flow details (e.g. pps). the NPU punt path to the LC CPU Short flows may not be seen. (larger) cache LC CPU Defines the number of entries the cache can Large cache prevents premature Consumes more memory size Memory hold export; useful when number of (300 bytes per entry) flows is large Permanent cache LC CPU Defines whether a learned flow should ever Provides very accurate packet rate A once learned flow never Memory expire (be removed from the cache) details on every export expires; always occupies the spot in the cache list (longer) Active LC CPU If a flow is still active, when should the details Reduces export rate Overuses the cache (keep the timer of the flow be exported to the collector entry longer) (longer) Inactive LC CPU If a flow is inactive (no packet seen for this Reduces export rate Keeps "stale" entries longer in timer period), when should the entry be retired cache (exported)

• Which interfaces is my map applied to? #show flow platform nfea monitor BGPDNLD eg loc 0/5/cPU0 Monitor : BGPDNLD Bundle-Ether100 Total interface : 1

• Full bgp as-path and community in netflow records • Configure: "bgp attribute download" under "router bgp" RP/0/RP0/CPU0:ASR9922#show cef 1.0.128.0/19 detail Updated May 17 12:04:50.009 BGP Attribute: id: 0x14c38, Local id: 0x5, Origin AS: 9737, Next Hop AS: 64530 ASPATH : 64530 109 6939 4766 38040 9737 Community: 100:200

• Physical bundle member (ingress netflow can report ingress member, egress netflow can report egress member) flow monitor-map BGPDNLD option outphysint

• Fragmentation is an exception path on all XR platforms. • Taking an exception path can lead to packets out of order: • Packet A.1 from flow A requires fragmentation, takes exception path. • Packet A.2 received shortly after A.1 does not require fragmentation. • Packet A.2 is forwarded before A.1 because exception path is slower.

• Fragmentation can be supported only on IPv4, PPPoE and MPLS packets. • IPv6 protocol does not define fragmentation.

• ASR9k is a two-stage forwarding platform  fragmentation is an egress feature.

• Packets that exceed MTU are punted to LC CPU through LPTS static policers: IPV4_FRAG_NEEDED_PUNT,PPPOE_FRAG_NEEDED_PUNT,MPLS_FRAG_NEEDED_PUNT

• NetIO fragments the packet and takes the forwarding decision based on SW FIB.

• L2 rewrite in NetIO (Adjacency database) LC CPU • Packet is injected "to port" NetIO • Benefits performance • Sub-interface services don't apply LPTS Te0/1/1/1 Te0/0/0/0 Packet (4kB) Packet (5kB) NP NP (MTU 9k) LC Fabric LC (MTU 4k) (1kB)

• Restrictions: • Works only on main interfaces • Works only on clear IPv4 packets

• Related NP counters: • MODIFY_CANNOT_FRAGMENT_DROP • Frames dropped due to errors during fragmentation • MODIFY_FRAGMENT_PROCESSING • Frames entering fragmentation processing code • MODIFY_FRAGMENTS_SENT

CE-PE L3 link over PW PWHE Internet L2 PW virtual Peering Access PE interface CE (A-PE) Service PE (S-PE)

L2 (port or Aggregation Core / Internet MPLS L2VPN L3PE vlan) Business L3 Core CE VPNs

• Unified MPLS end-to-end transport architecture

• Flexible service edge placement with virtual PWHE interface • L2 and L3 interface/sub-interface • Feature parity as regular L3 interface: QoS, ACL, Netflow, BFD, etc • CE-PE routing is over MPLS transport network. It doesn’t need direct L3 link any more

• CE-PE virtual link is protected by the MPLS transport network

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 48 Pseudo-Wire Head-End Configuration Examples Bind PW-Ether to MPLS Interfaces Bind PW-Ether to L2 VC interface pw-ether 200 l2vpn xconnect xconnect MPLS vrf vrf0001 xconnect group pwhe PW ipv4 address 11.0.0.1/24 p2p pwhe-red ipv6 address 2001:da1::1/64 interface pw-ether 100 CE PE PE attach generic-interface-list pwhe_gi_2 neighbor 100.100.100.100 pw-id 1 ! PWHE L3/L2 Main generic-interface-list pwhe_gi_2 Interface Example interface Bundle-Ether100 interface TenGigE0/5/0/12

xconnect MPLS interface pw-ether 100.100 l2vpn PW encap dot1q 100 xconnect group cisco vrf vpn-red p2p service2 CE ipv4 address 10.1.1.2/24 PE interface PW-Ether100.200 neighbor ipv4 1.1.1.1 pw-id 22 PE interface pw-ether 100.200 l2transport encap dot1q 200 bridge group group1 PWHE L3/L2 Sub- bridge-domain domain2 interface Example ! interface pw-ether 100.300 l2transport interface PW-Ether100.300 encap dot1q 300 vfi cisco neighbor 192.0.0.1 pw-id 100 neighbor 192.0.0.2 pw-id 100

• Only Per-VC load-balancing* PW-Ether Generic interface list • L2VPN load balancing config is ignored S-PE BE/ • No support for Flow Label* ECMP • Flow label config in pw-class is ignored A-PE1 A-PE2 • No support for BGP MP Per- VC LB *We are working on a solution!!! 1G 10G 1G 1G • What kind of load-balancing scheme does your PWHE Per- Per- CPE CPE CPE CPE deployment require? VC LB VC LB • (L2, L3+L4, any subset of the two) University • What QoS implications are acceptable? campus • Today PWHE QoS works perfectly thanks to per-VC LB!!!

RP/0/RP0/CPU0:ASR9922#sh cef 192.168.31.2 hardware ingress detail location 0/14/CPU0 192.168.31.2/32, version 186, attached, subscriber, internal 0x1000041 0x0 (ptr 0x83f30360) [1], 0x0 (0x83f0297c), 0x0 (0x0) Updated Dec 22 13:56:53.133 <...> LW-LDI-TS Dec 22 13:56:53.137 SUBS-INFO[0x85aeb050 IFH=0x920 (Subs I/F PW-Ether300.304.ip1) NH=0x853eff94 Flags=0x28] via PW-Ether300.304.ip1, 2 dependencies, weight 0, class 0 [flags 0x8] path-idx 0 NHID 0x0 [0x853efe68 0x0] local adjacency <...> RX H/W Result on NP:0 [Adj ptr:0x25 (BE)]: <...> FLAGS gre_adj: 0 pwhe_adj: 1 gre_ipv6_transp: 0 pwhe_vc_ll : 0x84bb00 (8698624) pwhe_vc_hash : 0x15a111a7 (LE) if_handle : 0x5e0

RP/0/RP0/CPU0:ASR9922#show interface pw-ether 300 <..> Generic-Interface-List: pwhe_gi_2 Find the related generic interface list

RP/0/RP0/CPU0:ASR9922#show generic-interface-list name pwhe_gi_2 Topology? Wed Dec 27 12:25:42.062 EST generic-interface-list: pwhe_gi_2 (ID: 3, interfaces: 5) Is this the full list of interfaces on which Bundle-Ether100 - items pending 0, downloaded to FIB the MPLS packets from A-PE can be TenGigE0/5/0/12 - items pending 0, downloaded to FIB received? TenGigE0/5/0/13 - items pending 0, downloaded to FIB TenGigE0/5/0/14 - items pending 0, downloaded to FIB Is this the full list of output interfaces TenGigE0/5/0/15 - items pending 0, downloaded to FIB towards PW tail-end? Number of items: 1 List is downloaded to FIB Check route towards tail-end RP/0/RP0/CPU0:ASR9922#show l2vpn xconnect interface PW-Ether 300 XConnect Segment 1 Segment 2 Group Name ST Description ST Description ST ------pwhe_lb pwhe_lb UP PE300 UP 8.8.8.8 300 UP ------

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 52 PWHE Troubleshooting show generic-interface-list [name ] show im database interface pw-ether verbose show interface pw-ether detail show proc pwhe_ma show l2vpn ea pwhe summary location show l2vpn pwhe interface pw-ether detail show l2vpn generic-interface-list private show l2vpn ma pwhe interface pw-ether private show l2vpn ma pwhe peer private show l2vpn ma pwhe trace show mpls lsd forwarding pw-list

• Punt packet: • NetIO process: • Any packet that NP hands over to the • Netio process runs on every RP and LC or RP CPU is a punt packet LC CPU • Examples: packets destined to the • Handles all protocol packets router; packets with destination IP that • 1 high and 1 normal priority output requires ARP resolution; etc. queue for injected packets

• Inject packet: • SPP: • Any packet sent to NP from LC or RP • Software Packet Path CPU • Optimized packet processing • CPU pipe-lining of instructions and data cache usage

OSPF HSRP BGP RIB

Socket (control)

RAW UDP TCP LPTS PA LPTS FM

Packet BCDL IPv4-IO Manager FIB/IFIB

Protocol DLL Switching DLL Protocol DLL Switching DLL

Driver DLL NetIO Driver DLL

fabric SPP

• HSRP runs at normal priority • BFD process is designed for Fast lightweight generic failure detection. • Not designed for short failure detection • Hello packet Rx/Tx always on LC CPU (bfd agent) • HSRP Tx hello packet path: • Threads of BFD process • hsrp  udp  ipv4_io  netio  responsible for hello packet spp  punt-fpga  fia_rsp  xbar exchange run at higher priority  fia_lc  np  wire • Session state maintenance and • Performance of the inject path interaction with clients is on RP (bfd depends on HSRP scale and other server) processes running on the RP • BFD does NOT transmit packets through S/W stack (netio)

BFD Server PRM: Platform resource manager RP FRR: Fast Reroute LC IM: Interface Manager FRR AIB: Adjacency Information Base BFD Agent IM SPP: Software Packet Path ChkPoint µIDB: Microcode Interface Descriptor Block Session Intf NetIO Packet NP: Network Processor DB (PD) Dispatcher AIB (PD)

PRM µIDB Manager SPP

NP0 NPn

• Current MQC allows for QoS action (policer/shape etc) on a set of statically configured classifiers (dscp, COS, pre configured ACL matching etc)

• Flow aware QoS solution on ASR9k provides a framework to enable QoS actions at a flow level.

• Detection of flows and enforcing QoS action is guided by existing CLI

• Flows are learned dynamically on a per-class, per-interface, per-direction level and QoS action / decisions are applied on a per-flow basis.

• Flexibility: • User defined flow definition – SrcIP, DstIP, 5 tuple • Configurable CAC / flow bandwidth to decide how many flows to allow • Redirection of non-admitted flows to default queue • Configurable flow entry idle-timeout to tune as per use case / traffic profile and ensure service fairness.

• Flow aware QoS with per-flow policer action also known as: • Micro flow policer • UBRL (User-based Rate Limiting) • Flow aware Policer

• Flow aware QoS with per-flow Call Admission Control (CAC) action is also known as: • Video Q / Video CAC • Admit CAC local • Flow aware CAC

• UBRL ensures single flow can’t starve bandwidth assigned to its matching class.

• CAC ensures specific number of flows are protected by either dropping or redirecting exceeding flows. CAC redirection provides a powerful tool for value added best effort services that dynamically adapt to load on the guaranteed pipe.

Regular QoS processing Source 1 (10.1.1.1) QoS Classification QoS Child Action QoS Parent Action Policer – 30 mbps Same policer applied on Source 2 Match on prec under both flows (11.1.1.1) same class-map Both flows policed at 30 mbps total

Flow Aware QoS processing

Flow Flow aware QoS module Lookup Source 1 flow Src 1 policed at 30 mbps Src 2 Source 1 Source 2 flow (10.1.1.1) QoS Classification policed at 30 mbps QoS Parent Action Same policer applied on Flow Match on prec under both flows Source 2 Table (11.1.1.1) same class-map QoS per-flow Child Action Policer – 30 mbps

Source 1 Regular QoS processing (10.1.1.1) QoS Classification QoS Child Action QoS Parent Action Source 2 Shape 10 mbps Same policer applied (11.1.1.1) Match on dscp under applied on all flows: on all flows Source 3 same class-map 10mbps in total and so (12.1.1.1) source 4 causes congestion Source 4 Each flow is 3 Mbps (13.1.1.1) Flow Aware QoS processing Flow aware QoS module Flow Source 1 Src 1 QoS Child Action (10.1.1.1) Lookup Shape 10mbps Src 2 Source 2 Applied on only (11.1.1.1) Src 3 QoS Classification admitted flows QoS Parent Action So, no congestion ! Source 3 Flow Same policer applied (12.1.1.1) Match on dscp under Table on all flows same class-map Source 4 Src 4 either dropped or (13.1.1.1) redirected to default queue Each flow is 3 Mbps

Requirement Mandatory Minor / Good to have Flow mask 1. Src IP based flow mask a. 3 tuple flow mask (SrcIP, DstIP, Protocol) flexibility 2. Dst IP based flow mask b. Src IP + Dst IP based flow mask (per-class) 3. Full 5 tuple flow mask (for IPv4 only) c. IPv6 5 tuple support Direction Ingress UBRL + Egress CAC Both direction support for both features Interface Bundle, and 1/10/40/100 G Typhoon/Tomahawk LC intf. BVI, BNG, PWHE and Satellite intf. End user interface C3PL/MQC complaint config & policer action support Per-class show commands with per-flow statistics / / CLI MIBs Transport / Service IPv4 / IPv6 UC + L4 (TCP, UDP, ICMP) All transport / services - Multicast, GRE, VPN Scale and 256k/496k unique flows per Ty/To NP (SE card) for UBRL and Any performance / scale improvement possible in performance 64k per NP and 16k per class for CAC with at least IMIX size phases line rate performance. Exhaustion / High rate (100 Gbps) Catchall policer for UBRL. Full QoS action support on reclassification fallback behavior Drop and reclassification of rejected CAC flows Tunable knob for cathcall policer CAC specific User configurable Flow / CAC BW to derive static max flow Syslog, ECN, Traps requirements count & configurable flow cache age timer (min of 30s) Dynamic BW calculation and enforcement Age min value from 1s Feature interaction ACL, Netflow for UBRL & ACL, Vidmon for CAC Full Feature matrix support on Typhoon.

• Flow – Packet traffic stream identifiable by a unique pattern in the packet header

• Flow mask – User config defining what fields constitute or differentiate a flow

• Flow tuple – Individual fields that can be included in the flow mask.

• Per-flow action • Requires per-flow resource allocation (first release supports only policer for UBRL and admit / reject action for CAC)

• Aggregate action • (Same as standard ASR9k QoS) • Can be regular mark / set actions, enforced on each flow, but are common for all flows. • Can be an aggregate parent policer / queuing action enforced on all flows

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 67 Flow Aware QOS Configuration UBRL Configuration: CLI class-map match-all match flow-key [ src-addr | dst-addr | 5-tuple ] match flow-cache { idle-timeout { | none }} | {max-count }

Example: class-map match-all cmap1 match flow-key 5-tuple flow-cache idle-timeout 120 end-class-map ! policy-map pmap1 class cmap1 police 100Mbps

Notes: 1. When flow-key is omitted, “5-tuple” is default. 2. “none” is the new keyword introduced for idle-timeout that indicates the flow doesn’t expire. 3. When idle-timeout is not configured, platform dependent value is set.

• Match flow requires match-all semantics. • A combination of match flow with match-any will be rejected.

• flow-key could be one of these options: • 5-tuple | src-ip | dst-ip | combination of src-ip and dst-ip

• Policer rate must be absolute (i.e. percentage not allowed)

• ASR9k does not support max-count in shipping releases (hidden in cli).

• CLI will reject policy classes with UBRL action which do not contain policing/marking. All other actions can also be configured in a UBRL class.

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 71 CAC Configuration: CLI class-map match-all match cac { admitted | unadmitted } [local] end-class-map ! policy-map class admit cac [local] [ flow idle-timeout ] flow rate  max per-flow rate rate  max rate for all flows

policy-map cac-redirect-policy class-map match-all cac-admit Only one of the two can be used class video match cac admitted local in a single CAC service-policy cac-child-policy end-class-map child policy admit cac local flow idle-timeout 30 class-map match-all cac-unadmit flow rate 1 mbps  max per-flow rate match cac unadmitted local rate 10 mbps  max rate for all flows end-class-map ! shape average 20 mbps policy-map cac-child-policy ! class cac-admit end-policy-map set precedence 2 shape average 10 mbps ! class class-default shape average 10 mbps ! end-policy-map

• Admit CAC sub-mode options: • idle timeout - optional • flow rate - mandatory • rate - mandatory

• A single level CAC policy is supported without the "match admit/unadmit" classification..

• In 2 level CAC policy, the CAC submode and "match admit/unadmit" must be 1 level away with the latter always being the child policy of the CAC submode.

• The flow rate must be less than rate.

• There is no per flow statistics support in the first release, and the work is in progress. The aggregate per class statistics o/p is supported as is today for any QoS policies.

• No deviations/exceptions for XML.

• There is no per flow statistics support in the first release, hence, there is no new PI MIB support for supporting flow aware stats. Existing aggregate per class level statistics and QoS MIBs are fully supported.

• QoS tech support is useful for debugging.

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 76 Platform resource allocation CLI units of 1k • Admin configuration command: hw-module flow-qos location max-flow-count <1-496>

• Used to configure the maximum number of flows that can be learnt on a specific line card based on the user requirement.

• Carve value is programmed on all NPs of the line card and require a linecard reload for memory carving on boot.

• Granularity of carve is 1k – reserves so many policers, NP search memory structure space for dynamic flow learning later on.

• On TR card, only up to 4k would be allowed

• If not enabled: 'qos-ea' detected the 'warning' condition 'Flow Aware QoS is not enabled on this LC.'

• Before LC reload RP/0/RSP0/CPU0:asr9k#sh controllers np struct FLOW-QOS-HASH np1 location 0/0/CPU0 Node: 0/0/CPU0: ------NP: 1 Struct 131: FLOW_QOS_HASH_STR Struct is a PHYSICAL entity Reserved Entries: 0, Used Entries: 0, Max Entries: 1024

• After LC reload and with 256 active flows: RP/0/RSP0/CPU0:asr9k#sh controllers np struct FLOW-QOS-HASH np1 location 0/0/CPU0 Node: 0/0/CPU0: ------NP: 1 Struct 131: FLOW_QOS_HASH_STR Struct is a PHYSICAL entity Reserved Entries: 0, Used Entries: 256, Max Entries: 131072

• User configurable per NP from 1k (default) to 256k/496k in 1k granularity per LC

• SE card: 256k/496k unique flow entries supported per NP for per flow policing

• TR card: 4k unique flow entries supported per NP for per flow policing

• Resource is per NP and so applies to both directions if traffic / UBRL policy applied bi directionally on same NP

• Scale limited by number of hardware policers supported by Typhoon/Tomahawk LC.

• Same table and scale can be supported as UBRL (flow table implementation is reused).

• For first release only 64k is claimed per NP and 16k per class

• 16k per class is hard limit. Per NP scale could be characterized and bumped up to 256k or more.

• TR card support is 4k per NP

• Per class CAC count can be oversubscribed to sum up to more than per NP limit. Configuration is allowed but global limit if reached per NP will be enforced first.

• Existing QoS class and policy-map scale stated in will remain (pmaps, classes, queues etc)

• There is no limit on number of UBRL classes and they can be as many as classes supported by regular QoS on ASR9k per system

• Similarly, there is no limit for CAC classes with reject drop action.

• If it is CAC redirect action, then up to 4k such classes are supported per NP (TR & SE)

• Requirement is to support line rate at IMIX packet size.

• Measured performance numbers from feature Integration Testing

No. Feature Line rate achieved @ L2 packet size in bytes 1 UBRL Input policy + IPv4 traffic 105 2 UBRL Output policy + IPv4 traffic 110 3 CAC Input policy + Reject drop + IPv4 traffic 110 4 CAC Output policy + Reject drop + IPv4 traffic 110 5 UBRL Input policy + IPv6 traffic 149 6 UBRL Output policy + IPv6 traffic 141 7 CAC Redirect Input policy + IPv4 traffic 244 8 CAC Redirect Output policy + IPv4 traffic 241

• CAC for IPv6 traffic is not supported. UBRL does not support 5-tuple flows with IPv6 traffic due to address length constraints.

• Not supported on L2 forwarding interfaces. Similarly, no support for PW-HE, BVI, BNG subscriber interfaces and satellite interfaces in first release.

• In bundle cases, flows are learnt and policed per-member and not on aggregate traffic of all the members

• There will be no support for L3VPN over MPLS / IP cores and related various imposition/disposition scenarios. Multicast support will be considered by DT end.

• There will be no per flow statistics support in the first releases. Hence there will be no new MIB support. Existing aggregate per class level statistics and QoS MIBs are fully supported.

• There will be no information given to the user on admission/rejection of flows for CAC through syslog or any other means

• No user-specified tuple support for CAC. It is assumed that CAC will be supported with 5 tuples

• CAC redirect will always require 2 level policies with at most of only 2 classes (either admit / unadmit and default) at the child level

• No support for policer action in leaf CAC class.

• UBRL per-flow policers are not shared with either SPI or shared policer. Only CAC count can be shared.

• UBRL and CAC actions are not supported in the same class or (same child level for CAC redirect)

• UBRL and CAC actions are supported only at the leaf level. CAC submode for a redirect action can be only at parent level.

• For CAC redirect action, the child level admit-classes / unadmit-classes support no other match criteria other than CAC admit / unadmit statements.

• Flow learning will not happen for feature drops such as ACL, no route etc. which occur before QoS processing. However, flow learning and resource allocation will happen for packets which are possibly dropped further down due to congestion, egress feature, uRPF etc.

• Similarly, flow match and age refresh will happen as long as packet is not dropped out of feature path before NP search stage since Flow hash table uses h/w refresh mechanism. So, if a flow is learnt, even if a newly configured ACL starts dropping the packets, the flow entry will not age out.

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 86 Flow Aware QOS Show / Debug Commands Show commands show qos-ea int <> in / out detail show qos int <> in / out show controller np struct FLOW-QOS-HASH summary location !Flow table details show controller np struct FLOW-QOS-HASH detail all location !Flow table details sh controller np registers block 12 location !Learn MREG content sh controller np registers block 20 location !Resolve MREG content show runn class-map show runn policy-map show runn int show policy-map int in / out !Collect several times show controller np cou location !Collect several times

DBG_PRS_EP_L_PRS_PL_FLOWQOS Packets entering loopback pass for Flow QoS learning

RSV_GET_CTX_TMOUT_FLOWQOS_CTX (Drop count) CAC Redirect frames dropped - Internal context timeout - software error

DBG_L_RSV_FLOWQOS_LRN_CNT Count of frames identified as new flow and being considered for learning into the Flow QoS entry table per NP

RSV_FLOWQOS_FLOW_CNT_EXCEED Flow QoS entry table per NP (UBRL) or per class (CAC) is full. No more flow entries can be learnt until older entries age out / get deleted

DBG_L_RSV_FLOWQOS_PASS2_ADD_CNT Replica frames finishing up 2nd pass Flow QoS learningThese are copies of original frames just for learn and discard

RSV_DROP_CAC_REJECT (Drop count) Ingress or Egress frames dropped because of per-class Flow aware CAC flow count exceed

LRN_FLOWQOS_CAC_CNT_EXCEED Per class h/w entry table for Flow aware CAC is full. No more flow entries can be learnt until older entries age out / get deleted

LRN_FLOWQOS_TBL_INDEXQ_EMPTY H/w programming inconsistency - Per flow hardware resourcequeue is empty. No more flow entries can be learnt and allocatedper-flow resource until queue is replenished with flow age/delete

• CSM Application (v2.x) • Standalone Java application • Runs on Linux, MS Windows, MacOS • Optimises SMU list per platform/release

• CSM Server (v3+) • Open source distribution • Runs on Linux

Install 2 Classify Reply Reply Manager •5 Resolve Dependency Send Report •6 Create SMU List

•7 Conformance 8 XR Router Report Test, Certify, Deploy 9 NOC BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 93 What CSM Does Solves For: • Software Management: • Time consuming, manual, • Automated and Simplified image (releases and SMUs) retrieval, laborious, repetitive, error- reporting and alerts prone SW installation • Pushes image to one or many devices • Complicated patch • End to end SW management dependencies • Patch recommendation, and conformance reporting • High costs • Migration from 32-bit XR to 64-bit XR Big Wins: • Operations Simplification: • Huge time and resource • Auto-updates: you can schedule installation, pre- and post- savings installation verifications • Up to 90% time savings on • Easier access to image and patch details (documentation) SW upgrades • Multi-platform and multi-OS support

• Inventory Management: • Visibility into hardware, cards, slots, S/N, optic types

• Manages thousands of devices by regions • Detects supported platforms and chassis • Daily software inventory retrieval • Connects via telnet/ssh and through a jump server

• Manages TFTP/FTP/SFTP server repositories • Browses/downloads/uploads software packages

• Manages users with different security privileges • Authenticates through CSM database or LDAP server

• IOS XR 32-bit (ASR9K, CRS)

• IOS XR 64-bit (ASR9K-64, NCS1K, NCS5K, NCS5500, XRv9000)

• NG-XR (NCS6K)

• IOS-XE (ASR902, ASR903, ASR907, ASR920)

• IOS (ASR901)

Device to upgrade

Actions to perform

Software source

Pick what you want to push

When to perform operation

Device to upgrade

Actions to perform

Software source

Pick what you want to push

When to perform operation

• Supports RESTful APIs using token based authentication.

• Ability to create, query, update, and delete hosts.

• Ability to query software information from CCO.

• Ability to create, query, delete scheduled installations

Download CSM: How to download https://software.cisco.com/download/release.html?mdfid=282423206&softwareid= 284777134&release=3.5&relind=AVAILABLE&rellifecycle=&reltype=latest Supported on: CSM Server Documentation: • CRS https://supportforums.cisco.com/document/13154846/cisco-software-manager-33- overview-documentation • NCS (1K, 5K, 6K)

CSM Server Videos: • 9XX Series Introduction to CSM Server: • ASR 9000 How to use https://youtu.be/isxN08x-mr4 Getting Started with CSM Server: • More coming soon! https://www.youtube.com/watch?v=omdpr3uP_b4 ASR9K IOS XR 32 bit to 64 bit Migration using CSM Server: https://youtu.be/RVgR0TdbpVw

CSM Application Video: https://www.youtube.com/watch?v=PYO2Om-nUKQ

Support forum https://supportforums.cisco.com/community/5996/xr-os-and-platforms

• Main principles: • Interface auto-shutdown is applied on interfaces without any configuration • On router platforms interfaces must be in shutdown status by default • Only explicit configuration is permanent • Otherwise the consistency of commit rollback may be disrupted

• Unwanted consequence best explained with this sequence: 1. New LC is inserted 2. Every interface on the LC is shutdown via auto-config 3. Users apply some configuration to the interface (for example interface description) 4. System or LC is reloaded 5. Interface is NOT shutdown via auto-config because it has some configuration

• Main principles remain: • Interface auto-shutdown is applied on interfaces without any configuration • On router platforms interfaces must be in shutdown status by default • Only explicit configuration is permanent • Otherwise the consistency of commit rollback may be disrupted

• Auto-shutdown config is saved into a file • .snd_shut__

• Upon rollback this file is consulted and config is merged •  consistent behaviour

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 103 Dealing with NP Lockups What is an NP lockup? Worker busted, infinite loop When all workers in a stage are locked up  the whole pipeline stalls Detection: health monitor (diagnostic) and TOP watchdog (listens to each worker) worker

Search Modify Queueing Parse Resolve Scheduling

•L2/L3 header •Performs QoS •Processes Search •Adds internal •Queuing, packet parsing in and ACL results: system headers Shaping and TCAM lookups in •ACL filtering •Egress Control Scheduling •Builds keys for TCAM tables •Ingress QoS Header (ECH) functions ingress ACL, •Performs L2 classification and •Switch Fabric •All packets go QoS and and L3 lookups policing Header (SFH) through this forwarding in RLDRAM •Forwarding (egress stage lookups (uCode) SFP determined) • Performs L2 MAC learning

CPU SDD

3x 10G Punt CPU 3x10GE Typhoon SFP + FPGA 3x 10G FIA 3x10GE Typhoon SFP +

3x 10G 3x10GE Typhoon USB SFP + Switch Fabric FIA Disk0/1 3x 10G FIA Switch 3x10GE Typhoon SFP + ASIC Fabric 3x 10G I/O FPGA 3x10GE Typhoon SFP + Typhoon 3x 10G FIA 3x10GE Typhoon SFP + NVRAM Boot Flash 3x 10G 3x10GE Typhoon SFP + CPU Complex 3x 10G FIA 3x10GE SFP + Typhoon Both active and standby RP send diag packets • (Active: unicast, Standby: multicast)

BRKSPG-2904 © 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 106 Interpreting the message %PLATFORM-DIAGS-3-PUNT_FABRIC_DATA_PATH_FAILED : Set|online_diag_rsp[237686]|System Punt/Fabric/data Path Test(0x2000004)|failure threshold is 3, (slot, NP) failed: (0/2/CPU0, 0)

• Syslog message formatted

• Set or Clear

• Specific test case failure

• processID and test ID (not important)

• Physical linecard reporting the error and the NPU towards which the path is broken. • If a single NPU is listed: it may be an NP lockup • If multiple or all NPs are listed: troubleshoot the fabric path to the line card

LC CPU PRM PRM: Primitive Resource Manager Health Mon packet PRS_HEALTH_MON counter

PRS SRCH RSV MDF TM (Traffic (Parse) (Search) (Resolve) (Modify) Manager)

• Common causes for DIAG failures are • NP Lockup (more detail to follow) • NP hw failure RP/0/RSP0/CPU0:A9K-BNG#sh contr np count np0 loc 0/0/CPU0 | i DIAG

• Bridge/FIA failure 235 PARSE_RSP_INJ_DIAGS_CNT 3268 0 • Fabric issues 248 PARSE_LC_INJ_DIAGS_CNT 3260 0 788 DIAGS 3260 0 902 PUNT_DIAGS_RSP_ACT 3268 0

• How to interpret failures reported Should trip multiple NPU’s! • Asses if there are multiple messages • From the same LC? • From NPU’s Connecting to the same FIA? • Reported by active and or standby RSP? • Noticing any other forwarding issues?

• A single health monitor packet is injected directly into each NP after LC boot.

• If this packet is dropped or lost, then a new health monitor packet re-inject is attempted three times.

• Failure detection time is ~12-13 seconds (polling interval + original health drop + 3 re-inject health drops) ( 1 sec + 3 sec + 9 sec).

• Error message: LC/0/0/CPU0: Jan 20: 11:09:22:122 : prm_server_to: NP-DIAG health monitoring failure on NP1

• Error recovery with NP fast reset: LC/0/3/CPU0:Feb 6 04:03:43.899 : prm_server_to[299]: %PLATFORM- NP-4-FAULT : Fast Reset NP1 - successful auto-recovery of NP

• Driver monitor all packet processing engines (PRS, SRH, RSV & MDF)

• Engine inactivity is an indicative of microcode bug or engine got stuck.

• Failure detection time is approximately 100ms

• Error message: LC/0/3/CPU0: Jan 20: 19:07:12:242 : prm_server_to: NP-DIAG failure on NP2, STS-0 PRS2-level cause 0x10 LC/0/3/CPU0: Jan 20: 19:07:12:242 : prm_server_to: NP-DIAG failure on NP2, STS-0 RSV2-level cause 0x20 LC/0/3/CPU0: Jan 20: 19:07:12:242 : prm_server_to: NP-DIAG failure on NP2, STS-0 MDF2-level cause 0x30

• Error recovery with NP fast reset: LC/0/3/CPU0:Feb 6 04:03:43.899 : prm_server_to[299]: %PLATFORM-NP-4-FAULT : Fast Reset NP1 - successful auto-recovery of NP

• NP Lockup is declared when the system detects that a path through the NP microcode is blocked

• NP lockup detection: • TOP inactivity (watchdog): lockup detection within milliseconds • Online-diagnostics failure: lockup detection 12s on Typhoon/Tomahawk

• What IOS XR does after detecting an NP lockup: • Generate NPdatalog and NPdrvlog files on harddisk:/np/ • System keeps logs of the last five NP lockup occurrences per NP • Perform NP fast reset • If 3 fast resets fail to clear the lockup condition, reload the LC

• Total traffic timeout: • Detection time + Fast Reset (~150ms) + NPdatalog collection (~13s)

• NP Lockup is detected with a delay  data captured in NPdatalog and NPdrvlog files may not be sufficient to understand the root cause

• Good problem specification and reproduction are keys to finding the root cause • When was the lockup observed first? • What configuration changes preceded the lockup? • What network event or traffic patter change preceded the lockup? • etc.

• What to collect: • Problem specification • All NPdrvlog and NPdatalog files • cXR: harddisk:/np/ ; eXR: /misc/scratch/np/ • show running-config, show install active summary, show logging

• Starting with IOS XR release 5.3.x, standard IOS XR package includes alternative NP microcode with more debug hooks (Typhoon, Tomahawk)

• Benefit: early detection of conditions that may lead to NP lockup

• Side effects: performance impact

• Loading the debug NP µcode requires a fast reset of the NP (not LC reload)

• Should be used only under guidance of Cisco TAC engineer

Questions? Use Cisco Spark to communicate with the speaker after the session

How 1. Find this session in the Cisco Live Mobile App 2. Click “Join the Discussion” 3. Install Spark or go directly to the space 4. Enter messages/questions in the space

cs.co/ciscolivebot#BRKSPG-2904

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public • Please complete your Online Complete Your Online Session Evaluations after each session Session Evaluation • Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt • All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Tech Circle

• Meet the Engineer 1:1 meetings

• Related sessions