Troubleshoot Catalyst 9800 Wireless Controllers Serviceability Enhancements of the New Platform

BRKEWN-3013

Troubleshoot Catalyst 9800 Wireless Controllers Serviceability enhancements of the new platform

Nicolas Darchis, CX Technical Leader Session objectives

• Understand the Catalyst 9800 WLC architecture in order to be able to know when to use which troubleshooting tool • software • hardware • relationship between these two

• Understand how features process packets through IOS-XE

• Understand how to easily debug the platform • presentation of recent serviceability enhancements • spare memorizing – focus on understanding • not “tips & tricks” but debugging strategy and tools

• C9800 general architecture • Hardware • Software • Life of a packet

• IOS-XE logging architecture • General concepts • Logging features and techniques (multiple parts)

• Packet captures and tracing

• Useful commands and tools

• Conclusion

• 1-2 years serviceability effort (and • Sudha Katgeri, US TAC TL more to come) • Nicolas Darchis, EMEA TAC TL • Testing and feedback since 16.7 code • Patnership with BU escalation (internal only) and engineering Tech Leads • TAC involved in serviceability requests 2 releases in advance • Cross-technology partnership within TAC and between BUs

Instructions unclear, got stuck in washing machine

Debugging process is extremely different… …for the better ! … …once you understand why !

• Serviceability is a journey with a clear destination … that is not yet in sight • Cisco AND customers want less TAC cases and faster/easier TAC cases

• A single way to enable debugs, not having to remember and enable dozens of debug commands

• Capacity to trace the path and time of a packet through the platform, including all the features it hits on the way

• Obtaining debug logs of past event in their context even without having enabled any debug manually

• Being able to verify things at every layer of the platform (control or data plane)

• Hardware • Software • Life of a Packet

• RP – Route Processor

• FP – Forwarding Processor = ESP (Embedded Service Processor)

• CPP – Cisco Packet Processor Complex= QFP (Quantum Flow Processor)

• PPE – Packet Processing Engine

• IOCP – I/O Control Processor

• FECP – Forwarding Engine Control Processor

• SPA – Shared Port Adapter

• SIP – SPA Interface Processor

• IOSd – IOS image that runs as a process on the RP

• FMAN – Forwarding manager (FMAN-RP, FMAN-FP)

• btrace – binOS tracing – the binary logging system used by binOS processes

• EOBC = Ethernet Out of Band Channels – Packet Interface for Card to Card Control Traffic

• IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP

• Doppler (UADP) hardware chipset. FED (Forwarding Engine Driver) architecture for packet forwarding

• Software & control plane stays the same but dataplane is completely different

• We will not cover this dataplane type. More details in BRKARC-2035

• Cloud and appliances : same CPP architecture !

• Cloud just virtualizes the dataplane

• Appliance have the QFP (Quantum Flow Processor) hardware for the CPP.

• No differences apart from the performance (and Manufacture Installed Certificate)

• More details in BRKARC-3147 (ASR1k architecture and troubleshooting)

• While appliances have a Manufacture-Installed Certificate, virtual controllers can only rely on Self-Signed Certificate.

• It is generated by the day-0 wizard once you enabled the wireless management interface and configured the country.

• In case of issues, it can be re-generated with this CLI : #wireless config vwlc-ssc key-size 2048 signature-algo sha256 password

• As of 16.10, all SSC have the same Serial Number which can cause browser issues when you have multiple C9800-CL. You can install another certificate for UI management to work around the problem.

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 C9800 WLC hardware Cisco “Quantum Flow Processor” • Packet Processing Engine (QFP-PPE) – 64 Packet Processors with 4 threads per core – 1.5GHz Tensilica ISA processors + DRAM packet memory – Single 80M TCAM4 I/F – C-language for feature development; extensive development support tools Multi-Core (64) Packet Processor – HW assist for flow-locks, look-ups, stats, WRED, policers, range lookup, crypto, CRC

• Buffer/queue subsystem (QFP-BQS) – HW hierarchical 3-parameter (min, max & excess) scheduler – Fully configurable # of layers based on HQF – Priority propagation through the multiple layers

Traffic Manager (BQS) BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 C9800 WLC hardware Example layout of a 9800-80 • 9800-80 is actually capable of 100Gbps QFP 1 QFP 0

• 2 load-balanced QFPs (1 in 9800-40) eUSB • 1 crypto chip TCAM • 12-core CPU in 9800-80 PCH • 8-core CPU in 9800-40

CPU

SSD

Block X86 CPU Block Crypto chip C complex D

C9800-40: Single QFP QFP Astro C9800-80: Dual QFP Block B Buffering

Ingress ASIC (NP5c/Ezchip) buffering

10G PHY Block A

SFP+ x4 or x8

• Hardware • Software • Life of a Packet

• IOS(d) is a process like another. IOSd takes care of routing, CLI, multicast and interfaces IOS-XE Polaris 16.10

IOSd IOSd Hosted subsystemIOSd IOSd subsystemIOSd Apps Blob subsystem WNCd WNCd HA WNCd Management interface RRMd Module drivers Rogued Kernel Wireshark BinOS

• SANET : client AAA policies

• SISF : client IP learning SANET SISF • WNCd : “controller” process managing APs and clients WNCd

Legend database Wireless process

WNCmgr

WNCd WNCd smd wstatsd rogued mobilityd

Horizontally scaled

RRM NMSPd

Legend database Wireless process

• To take real profit of multi-core systems, otherwise:

AP distribution across WNCd processes: • APs of the same site join the same WNCd process • Exception is on the default site where APs are load-balanced

IOSd

FMAN RP WNCd Wireless WNCd WNCd processes FMAN ASIC FP driver

Legend database Wireless Programmable interface Polaris infra process Punt path

IOSd

FMAN WNCd RP WNCd WNCd Wireless FMAN processes ASIC FP Linux LSMPI LFTS TCP/IP

Dataplane (CPP/ Doppler) Legend database Wireless Linux Polaris infra Programmable interface process kernel Punt path

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 Software architecture rd Cisco Web DMI 3 Prime Management party DNA-C UI procs REPM IOSd infra DBM CLI ODM FMAN agent RP WNCd WNCd Wireless WNCd FMAN processes ASIC FP Linux LSMPI LFTS TCP/IP

Dataplane (CPP/ Doppler)

Legend database Wireless Linux Management Programmable interface Polaris infra process kernel access Crimson access Punt path

• Hardware • Software • Life of a Packet

X86 CPU complex Crypto chip

Traffic Forwarding Path: Astro Faceplate Ports  EZChip QFP NP5c  Astro  Yoda Complex Buffering

Ingress ASIC buffering

Block A/B/C/D troubleshooting 10G PHY commands in following hidden slides

X86 CPU complex Crypto chip

Traffic Forwarding Path: Faceplate Ports  EZChip QFP Astro NP5c  Astro  Yoda Complex Buffering

Ingress ASIC buffering

Block A/B/C/D troubleshooting 10G PHY commands in following hidden slides

IOSd Other wireless processes FMAN RP WNCd WNCd WNCd RRMd FMAN FP Linux LSMPI LFTS TCP/IP Dataplane (CPP/ Doppler)

Wireless Polaris Linux Legend processes infra kernel RRM packet path

Block A

Interface Stats (to check for link errors, pkt rate/bps, backpressure indication etc) show interfaces

EZChip Backpressure indication commands [Drops in Rx indicate Backpressure from Yoda] show platform hardware port 0/0/7 ezman statistics

show platform hardware port ezman info | buffer

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 30 Block A : Interface / ASIC CLI show platform hardware port 0/1/0 ezman show interfaces tenGigabitEthernet 0/0/5 statistics RX Counters Input queue: 0/375/0/0 (size/max/drops/flushes); Total MAC Filter drop:0 Unknown Vlan Drop:0 output drops: 0 Queueing strategy: fifo High Priority Output queue: 0/40 (size/max) Pass Pkt:93191 Bytes:16243813 5 minute input rate 9090609000 bits/sec, 1410628 Drop Pkt:0 Bytes:0 packets/sec Low Priority 5 minute output rate 1435073000 bits/sec, 121063 Pass Pkt:8447296407 Bytes:6805697149012 packets/sec Drop Pkt:172546135654 181442777729 packets input, 146180407969360 bytes, Bytes:139012797667814 Backpressure indicators.. 0 no buffer QFP PPEs not able to Received 1865 broadcasts (7052 multicasts) TX Counters 0 runts, 0 giants, 0 throttles High Priority process 1174099533 input errors, 0 CRC, 0 frame, 1174099533 Pass Pkt:0 Bytes:0 fast enough, causing overrun, 0 ignored Drop Pkt:0 Bytes:0 random drops 0 watchdog, 7052 multicast, 0 pause input Low Priority 15220486562 packets output, 22554826373594 bytes, Pass Pkt:15182029085 Bytes:22497843030408 0 underruns Drop Pkt:0 Bytes:0

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 31 Block B (PPE) : Crypto commands WiredWireless B1.2 CAPWAP crypto Wireless MAC decrypt/ Crypto D B1.2 encrypt crypto

Crypto Stats from PPE show platform hardware chassis active qfp feature wireless dtls datapath statistics show platform hardware chassis active qfp feature ssl data drop

PPE drops due to Crypto Backpressure [Hitting Cavium limits] show platform hardware chassis active qfp statistics drop | inc ssl

Individual DTLS Session Stats|Summary [Shows stats for both directions:Encrypt | Decrypt] show platform hardware chassis active qfp feature wireless dtls stats [ NOTE: Session id obtained from the cpp-client command below]

DTLS Session Plumbing Details (including getting the session-id) show platform hardware chassis active qfp feature wireless dtls cpp-client statistic|summary

Other Crypto related Commands on the PPE: show platform software dtls chassis active F0 statistics show platform hardware chassis active qfp feature ipsec data memory

Crypto Stats show platform hardware crypto-device registers packet-count show platform hardware crypto-device statistics

Crypto Backpressure Status show platform hardware crypto-dev registers backpressure

Crypto Utilization show platform hardware crypt-dev utilization

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 33 Crypto command output show platform hardware crypto-dev registers backpressure show platform hardware crypto-dev utilization

Forwarding Manager Encryption- Past crypto device utilization: 1 min (percentage) : 0% processor Registers (decrypt pkt): 37 (encrypt pkt): 37 5 min (percentage) : 0% transmitting backpressure: false (decrypt pkt): 271 (encrypt pkt): 271 receiving backpressure: false 15 min (percentage) : 0% (decrypt pkt): 771 (encrypt pkt): 781

QFP core drops due to Crypto backpressure High crypto utilization / transmit backpressure flag being set (true) or the sslVpnTailDrops counter going up : These are indicators that we’re hitting the crypto processing limit

QFP Punt routine X86 CPU /inject callback

B1.2 punt_inject

Wireless Punt Stats for various reasons show platform hardware chassis active qfp feature wireless punt statistics Global Punt Stats (those with LFTS are the ones punted to WNCd) show platform hardware chassis active qfp infra put statistics type per-cause | in LFTS Inject Stats ( check for the ones injected into LFTS) show platform hardware chassis active qfp infra punt statistics type inject-drop Punt Policer show platform hardware chassis active qfp infrastructure punt policer summary To configure Punt Policer Rate for a Specific Cause show platform hardware chassis active qfp infrastructure punt config

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public B 1.2 : Punt/Inject Command Output Show plat hard chass active qfp infra punt statistics type Show pl hardware chass active qfp infra punt statistics type inject- punt-drop | inc LFTS drop | inc LFTS

097 Packets to LFTS 0 043 Applications Injecting Pkts using LFTS 0 129 wls 802.11 Packets to LFTS 0 130 wls CAPWAP Packets to LFTS 0 043 Applications Injecting Pkts using LFTS 0 131 wls MOBILITY Packets to LFTS 0 132 wls SISF Packets to LFTS 0 043 Applications Injecting Pkts using LFTS 0 043 Applications Injecting Pkts using LFTS 0

For each cause, the inject drop Show plat hard chass active qfp infra puntmstatistics per-cause counter for those injected via LFTS is shown above. For each punt cause, we look for the stats to LFTS

B1.2 C punt_inject QFP Punt routine X86 CPU /inject C1.1 callback C1.1 punt_inject at punt_inject at LSMPI/LFTS LSMPI/LFTS

LSMPI Stats [INJECT] show platform software infrastructure lsmpi inject

LSMPI Stats show platform software infrastructure lsmpi

C1.2 B1.2 C punt_inject punt_inject QFP Punt at SISF routine X86 CPU /inject C1.2 callback punt_inject at SISF

SISF database (IP reachability information) show wireless device-tracking database ip

SISF injected L2 / DHCP pkt stats show wireless device-tracking counters l2 | dhcp

• General concepts • Always-on tracing • Trace-on-failure • Conditional debugging : Radioactive tracing • Non-conditional debugging : Specific component debugging

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 39 IOS-XE logging architecture bTrace (i.e. binOS Tracing system) • All binOS (i.e. non IOSd) processes log to files in flash/disk

• Each process has his own log file

• Files are written in memory first then on disk (bootflash:/tracelogs/)

• When a log file reaches its maximum size, it rotates and creates a new one

• Logs are written in binary and then compressed for archiving

• This means that live debugging (old IOS-like) is not available for now

• Logs are written using syslog-like severity levels

• IOSd still uses IOS logger. Migration to btrace in progress.

WNCd-0 tracelog (wncd_x_R0-0.2280_41.20181009080530.bin) Client join L2 Authentication Key Exchange Start. EAP type: PEAP, Resolved VLAN: 185, Audit Session id:ABCD WNCd-0 messages EAP Key management successful. AKM:FT-DOT1X Cipher:CCMP WPA2 Btrace Library Mobility discovery triggered. Client mode: Local ADD MOBILE sent. Client state flags: 0x72 BSSID: MAC: abcd.abcd.cdef capwap IFID: 0x1234 Client IP learn successful. Method: IP Snooping IP: 10.0.0.1 Client state transition: S_CO_IP_LEARN_IN_PROGRESS -> S_CO_RUN Final RUN state message

Syslog, VTY (term mon), console, … IOSd Oct 9 09:12:15.363 UTC: %CLIENT_ORCH_LOG-6-CLIENT_ADDED_TO_RUN_STATE: Chassis 2 R0/0: wncd: Username entry (bob) joined with ssid (foo) for device with MAC: 1234.1234.5678 IOS Logger Btrace Library IOSd traceIog (IOSRP_R0-0.14671_21.20181009041228.bin)

Oct 9 09:12:15.363 UTC: %CLIENT_ORCH_LOG-6-CLIENT_ADDED_TO_RUN_STATE: Chassis 2 R0/0: wncd: Username entry (bob) joined with ssid (foo) for device with MAC: 1234.1234.5678

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 41 “That means I’m going to have to check a whole list of log files in different folders, linux-style right ? The skeptical network admin Company XYZ

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 42 IOS-XE logging architecture How do we consult logs ? • ERROR level represent abnormal situations. We want to raise the user attention to these

• WARNING represent an incident that could potentially to an error (or not…)

• NOTICE is the default logging level. It represents a condition, even 2-Critical 3-Error normal, but having significance (a client connecting for example) 4-Warning 5-Notice • INFO contains details about state machines and the communication flow 6-Info 7-Debug • DEBUG contains enough logging information for developers to root 8-Verbose cause a possible bug most of the time

• VERBOSE : logging madness is unleashed. Are you sure you want this ?

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 43 IOS-XE logging architecture How do we consult logs ? #show logging continues to give us the log history of IOSd events #show logging profile wireless to-file collates (and decodes) to the destination filename in flash all the wireless-relevant logs from ALL the log files on disk 2-Critical Since there is a lot of logging, we have options like : 3-Error 4-Warning #show logging profile wireless level to-file 5-Notice 6-Info INFO level is as verbose as a debug was on AireOS.Use this ! 7-Debug 8-Verbose Debug level is perfect to throw thousands of lines to those bored TAC engineers though …

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 44 IOS-XE logging architecture How do we consult logs ? #show logging profile wireless level info filter mac to-file #show logging profile wireless start timestamp “MM/DD/YYYY HH:MM:SS” level info filter mac to-file

Without the timestamp option, the logs will be collected since the last system boot, which can be very very far away back in time

• General concepts • Always-on tracing • Trace-on-failure • Conditional debugging : Radioactive tracing • Non-conditional debugging : Specific component debugging

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 46 Introducing always-on tracing Logs even without enabling debugs • Each process constantly writes logs down to NOTICE level to give some context about normal but significant events happening

• We are tracking client connections (even successful) and their state machine changes

• Target : each process can log for at least 48 hours (on a fully loaded box)

• You can debug what happened to a client AFTER it happened and without having enabled anything beforehand ! 2-Critical 3-Error 4-Warning 5-Notice 6-Info 7-Debug 8-Verbose

[client-orch-sm] [24632]: (note): MAC: 0040.96b9.b5c4 Association received. BSSID 0038.df25.f12f, old BSSID 0000.0000.0000, WLAN 1, Slot 1 AP 0038.df25.f120, AP0038.DF24.62A8 [client-orch-state] [24632]: (note): MAC: 0040.96b9.b5c4 Client state transition: S_CO_INIT ->S_CO_ASSOCIATING [dot11] [24632]: (note): MAC: 0040.96b9.b5c4 Association success. AID 1, Roaming = 0, WGB = 0, 11r = 0, 11w = 0 [client-orch-state] [24632]: (note): MAC: 0040.96b9.b5c4 Client state transition: S_CO_ASSOCIATING ->S_CO_L2_AUTH_IN_PROGRESS [client-auth] [24632]: (note): MAC: 0040.96b9.b5c4 ADD MOBILE sent. Client state flags: 0x71 BSSID: MAC: 0038.df25.f12f capwap IFID: 0xf90400004 [client-auth] [24632]: (note): MAC: 0040.96b9.b5c4 L2 Authentication initiated. method DOT1X, Policy VLAN 1,AAA override = 0 [ewlc-infra-evq] [24632]: (note): Authentication Success. Resolved Policy bitmap:11 for client 0040.96b9.b5c4 [client-auth] [24632]: (note): MAC: 0040.96b9.b5c4 L2 Authentication Key Exchange Start. EAP type: PEAP, Resolved VLAN: 16, Audit Session id: 22100A090000000E89D69B30 [client-keymgmt] [24632]: (note): MAC: 0040.96b9.b5c4 EAP Key management successful. AKM:DOT1X Cipher:CCMP WPA2 [client-orch-sm] [24632]: (note): MAC: 0040.96b9.b5c4 Mobility discovery triggered. Client mode: Local [client-orch-state] [24632]: (note): MAC: 0040.96b9.b5c4 Client state transition: S_CO_L2_AUTH_IN_PROGRESS - >S_CO_MOBILITY_DISCOVERY_IN_PROGRESS [client-auth] [24632]: (note): MAC: 0040.96b9.b5c4 ADD MOBILE sent. Client state flags: 0x72 BSSID: MAC: 0038.df25.f12f capwap IFID: 0xf90400004 [client-orch-state] [24632]: (note): MAC: 0040.96b9.b5c4 Client state transition: S_CO_MOBILITY_DISCOVERY_IN_PROGRESS - >S_CO_DPATH_PLUMB_IN_PROGRESS [dot11] [24632]: (note): MAC: 0040.96b9.b5c4 Client datapath entry params - ssid:dot1x_j,slot_id:1 bssid ifid: 0x0, radio_ifid: 0xf90400002 [dpath_svc] [24632]: (note): MAC: 0040.96b9.b5c4 Client datapath entry created for ifid 0xfa0000001 [client-orch-state] [24632]: (note): MAC: 0040.96b9.b5c4 Client state transition: S_CO_DPATH_PLUMB_IN_PROGRESS - >S_CO_IP_LEARN_IN_PROGRESS [client-iplearn] [24632]: (note): MAC: 0040.96b9.b5c4 Client IP learn successful. Method: DHCP IP: 9.10.16.121 [client-orch-state] [24632]: (note): MAC: 0040.96b9.b5c4 Client state transition: S_CO_IP_LEARN_IN_PROGRESS ->S_CO_RUN

[client-orch-sm] [32269]: (note): MAC: 0000.0a34.0001 Association received. BSSID 000b.cd00.060f, old BSSID 0000.0000.0000, WLAN 1, Slot 1 AP 000b.cd00.0600, EWLC-AK-SIM-AP-5 2018/09/18 08:58:40.149 {wncd_x_R0-1}{1}: [client-orch-state] [32269]: (note): MAC: 0000.0a34.0001 Client state transition: client_orch_sm_state___none -> S_CO_ASSOCIATING [dot11] [32269]: (note): MAC: 0000.0a34.0001 Association success. AID 1, Roaming = 0, WGB = 0, 11r = 0, 11w = 0 [client-orch-state] [32269]: (note): MAC: 0000.0a34.0001 Client state transition: S_CO_ASSOCIATING -> S_CO_ASSOCIATED_TR [client-orch-state] [32269]: (note): MAC: 0000.0a34.0001 Client state transition: S_CO_ASSOCIATED_TR -> S_CO_L2_AUTH_IN_PROGRESS [client-auth] [32269]: (note): MAC: 0000.0a34.0001 ADD MOBILE sent. Client state flags: 0x71 BSSID: MAC: 000b.cd00.060f capwap IFID: 0xf90400003 [client-auth] [32269]: (note): MAC: 0000.0a34.0001 L2 Authentication initiated. method DOT1X, Policy VLAN 1,AAA override = 1 [ewlc-infra-evq] [32269]: (note): Authentication Success. Resolved Policy bitmap:11 for client 0000.0a34.0001 [ewlc-infra-evq] [32269]: (ERR): SANET_AUTHC_FAILURE - Cred Fail username wpa2eapfast, audit session id 22100A09000002C8EBE72A99, [client-orch-sm] [32269]: (note): MAC: 0000.0a34.0001 Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_CLIENT_CREDENTIAL_FAILURE [client-orch-state] [32269]: (note): MAC: 0000.0a34.0001 Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_DELETE_IN_PROGRESS [client-orch-state] [32269]: (note): MAC: 0000.0a34.0001 Client state transition: S_CO_DELETE_IN_PROGRESS -> S_CO_DELETED

show log profile wir filter mac to-file output.txt

Unsupported AP [apmgr-capwap-join] [1263]: UUID: 0, ra: 0, TID: 0 (ERR): d824.bde8.3690 Join request not accepted: Unsupported AP Model AIR-LAP1142N-A-K9

Reg Domain failure [apmgr-capwap-config] [1394]: UUID: 10000000002ed, (ERR): f44e.0597.fb50 Failed to verify reg domain slot. validation of country code(UX) to regulatory domain(-A) error:1 [apmgr-capwap-config] [1394]: UUID: 10000000002ed, (ERR): f44e.0597.fb50 Failed to get ap default country code. Get default country code for AP error. [apmgr-capwap-config] [1394]: UUID: 10000000002ed, (ERR): f44e.0597.fb50 Failed to set reg domain check status. country code US is not configured on WLC

Cert Failure [apmgr-capwap-config] [1394]: UUID: 10000000002ed, (ERR), %PKI-3-CERTIFICATE_INVALID_NOT_YET_VALID: Certificate chain validation has failed. The certificate (SN: 6B4F09560000001763DF) is not yet valid Validity period starts on 22:48:43 IST Sep 9 2014

Discovery to non wireless mgmt interface 2017/09/22 01:51:02.168 {wncmgrd_R0-0}{2}: [capwapac-srvr] [16320]: UUID: 0, ra::0, TID: 0 (ERR): IP:3.3.3.1[5246], Discovery to non wireless mgmt interface

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 50 “That sounds good, but that’s still reactive and I have to wait for users to complain to the helpdesk” The skeptical network admin Company XYZ

• General concepts • Always-on tracing • Trace-on-failure • Conditional debugging : Radioactive tracing • Non-conditional debugging : Specific component debugging

• Predefined failure codes are sh wireless stats trace-on-failure tracked 001. AP radio reset...... : 0 002. AP reset...... : 0 • You can pull statistics … ===> 003. Client disjoin due to AP radio reset...... : 0 • Or see indexed recent failures. It 004. Client disjoin due to AP reset...... : 0 allows you to quickly see the 005. Export client MMIF...... : 0

latest issue on the box and have 006. Export client MM...... : 0

a precise timestamp + UUID 007. Export client generic...... : 0

pointer to the exact logs ! 011. AP join failure...... : 0

012. AP initial configuration failure...... : 44335

…..

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 53 IOS-XE Logging architecture Show logging trace-on-failure summary Time UUID Log ------2018/09/21 04:43:52.773 0x1000000004c93 2048.2000.0300 AP_CFG_STATUS_FAIL : Apmgr failure reason : Regulatory 2018/09/21 04:43:52.990 0x1000000004cbf 2048.2000.0500 AP_CFG_STATUS_FAIL : Apmgr failure reason : Regulatory 2018/09/21 04:43:53.030 0x1000000004ccc 2048.2000.0400 AP_CFG_STATUS_FAIL : Apmgr failure reason : Regulatory 2018/09/21 04:43:53.068 0x1000000004ce5 2048.2000.0200 AP_CFG_STATUS_FAIL : Apmgr failure reason : Regulatory 2018/09/21 04:43:53.226 0x1000000004d05 2048.2000.0700 AP_CFG_STATUS_FAIL : Apmgr failure reason : Regulatory 2018/09/21 04:43:53.270 0x1000000004d17 2048.2000.0600 AP_CFG_STATUS_FAIL : Apmgr failure reason : Regulatory 2018/09/21 04:43:55.626 0x1000000004e61 2048.2000.1200 AP_CFG_STATUS_FAIL : Apmgr failure reason : Regulatory 2018/12/12 12:26:35.406 0x10000000cd09b 8875.56c6.f000 AP_JOIN_FAIL : Apmgr failure reason : Unsupported ap, 2018/12/17 13:18:32.097 0x10000002c7428 08cc.68b4.4660 CAPWAPAC_HEARTBEAT_EXPIRY

2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [ewlc-infra-evq] [3862]: (note): Data type : Message handle 2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [apmgr-capwap-join] [3862]: (ERR): 8875.56c6.f000 Join request not accepted: Unsupported AP Model AIR-CAP3602I-E-K9 2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [apmgr-capwap-join] [3862]: (ERR): 8875.56c6.f000 Failed to process join request. Unable to decode apmgr join response 2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [apmgr-ap-global] [3862]: (ERR): 8875.56c6.f000 Failed to handle ap sm join request. Unable to process apmgr join request 2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [ewlc-infra-evq] [3862]: (ERR): 8875.56c6.f000 AP_JOIN_FAIL : Apmgr failure reason : Unsupported ap, Policy tag : , Site tag : , Rf tag : default-rf-tag 2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [apmgr-db] [3862]: (ERR): Failed to get ap name mac map record for delete. Name: AP3602I-E-K9. Reason: No such file or directory 2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [apmgr-db] [3862]: (ERR): 8875.56c6.f000 Delete ap name map record from the apmgr failed: 2 2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [capwapac-smgr-sess-fsm] [3862]: (ERR): Session-IP: 192.168.17.146[57187] Mac: 8875.56c6.f000 Unmapped previous state in transition S_JOIN_PROCESS to S_END on E_AP_INTERFACE_DOWN 2018/12/12 12:26:35.406 {wncd_x_R0-3}{1}: [apmgr-db] [3862]: (ERR): 8875.56c6.f000 Mismatch in session handles. Record already deleted and recreated

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 55 “And I suppose that for active debugging, I need to remember hundreds of set platform trace commands ?” The skeptical network admin Company XYZ

• General concepts • Always-on tracing • Trace-on-failure • Conditional debugging : Radioactive tracing RA-trace • Non-conditional debugging : OFF Specific component debugging

RA-trace ON

• Debug platform condition interface/mac/ipv4/ipv6

• Debug platform software cond-debug verbose

• Show platform conditions

• Clear platform condition all. Drawbacks : • every independent process in the flow must evaluate the condition separately • Some process do not have access to the data required to evaluate the condition BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 58 RA tracing on incoming packets example

• Every entry process checks if the flow matches the conditional debugging

• If so, it sets a radioactive flag, obtain a UUID (code execution flow unique identifier) and passes it on with to all the functions called

• When the flow ends, the radioactive flag is reset

WNCD evlib WNCD capwapdata WNCD client orch • Receives assoc req • Queries if it has to be • Flag is on, logging at from DP traced “debug” for this flow • Passes to capwap • Sets RA flag and UUID • Passes on to dot11 data function • Passes to client function orchestrator

• Every entry process checks if the flow matches the conditional debugging

• If so, it sets a radioactive flag, obtain a UUID (code execution flow unique identifier) and passes it on with to all the functions called

• When the flow ends, the radioactive flag is reset

• Every entry process checks if the flow matches the conditional debugging

• If so, it sets a radioactive flag, obtain a UUID (code execution flow unique identifier) and passes it on with to all the functions called

• When the flow ends, the radioactive flag is reset

• All intermediate processes will be debugged at the same level without having to verify the original condition

• RA trace level is independent of the btrace levels already configured for each component

• On top of MAC or IP, it is also possible to trigger RAtracing based on specific flows

RRM DCA TPC • Picks certain AP macs • Makes DCA to evaluate for calculations for that AP • Makes TPC RRM/DCA • Traces in debug mode calculations for • Verifies if specific APs when the RA flag is that AP should be traced seen • Trace in debug • Sets RA flag and UUID mode when the RA flag is seen BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public RA tracing conditional debugging same commands for AP or client troubleshooting • debug platform condition feature wireless mac Client troubleshooting • debug platform condition start (reproduce issue) • debug platform condition stop • show logging profile wireless (start timestamp “Date&time”) level debug filter mac to-file • more flash: ”Clear platform condition all” when done • debug platform condition feature wireless mac AP troubleshooting • debug platform condition start (reproduce issue) • debug platform condition stop • show logging profile wireless (start timestamp “Date&time”) level debug filter mac to-file • more flash:

WLC# debug wireless mac aaaa.bbbb.cccc ? Automatically stops ! ftp-server Move log file to FTP server, default storage: "flash:/" monitor-time Max time to trace the condition, Default: 30min internal Collect all logs.(Default: only customer curated logs) This is a macro that runs the commands from the previous slides in a single command. Can be run for a certain or stopped with the “no” version of the command.

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 64 Conditional debugging / RA tracing It’s a “debug client” that works with Aps too and … ? • Clients • APs • Mobility peers • In a short future : multicast ? NMSP ?

• Debugging a mac or an IP doesn’t give the same result, depending on what information processes have access to.

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 65 IOS-XE logging architecture Conditional debugging – difference between mac and IP? • If you debug an AP by its radio mac address, you will miss the DTLS tunnel establishment, because at that stage, the WLC has no idea of the AP mac : the tunnel is built on the IP address of the AP and the source mac is probably the gateway.

• If you debug an AP by its IP address, you will get the DTLS tunnel establishment but might miss some debugs that are focused on the radio mac (RRM, etc…)

• General concepts • Always-on tracing • Trace-on-failure • Conditional debugging : Radioactive tracing • Non-conditional debugging : Specific component debugging

RRM (for all APs)

• set platform software trace wireless chassis active r0 rrm all debug (reproduce issue) • show logging process rrm to-file • more flash: Set back to “NOTICE” level when done ! • set platform software trace nginx chassis active r0 all debug Web UI (reproduce issue) • show logging process nginx to-file • more flash:

RRM (for all APs): set platform software trace wncd chassis active R0 radio-history-channel info set platform software trace wncd chassis active R0 radio-history-reset info set platform software trace wncd chassis active R0 radio-history-radar info

NMSP : set platform software trace nmspd chassis active R0 all-modules debug

Verifying current trace levels: show platform software trace level chassis active r0

• Packet tracing : Following the path of specific packets

• Packet captures : PCAPs as you know them

Exhibit A : Packet tracing in action

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 71 Packet tracing Capwap data packet PPE Processing: Wireless -> Wired B1.1 CAPWAP SVI (L3 IP B1.2 Sub L2 ( src mac Capwap Encapsulated lookup to lookup and tunnel Block terminal learn) lookup (5 Pkt Entry Point tunnel) tuple hash) B1 DTLS, Re- assembly optional

Re-Write header Wireless B1.4 B1.3 Client Feat (strip outer l2, ip, capwap, 801.11 -> lookup ures 802.3) ACL, QoS …

Flood

B1.1 Dst Mac OR 802.3 pkt lookup Unicast L2 switch BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 72 Packet tracing FIA Tracing: Wireless -> Wired SVI (L3 IP lookup to To CAPWAP features (next slide) L2 ( src mac lookup and learn) terminal tunnel) Feature: SWPORT_PM_INPUT_CHECK Entry : Input - 0x7001247c Input : Port-channel1 Feature: IPV4_INPUT_DST_LOOKUP_ISSUE Feature: IPV4_INPUT_LOOKUP_PROCESS Output : Entry : Input - 0x70012220 Entry : Input - 0x70012228 Lapsed time : 629 ns Input : Vlan10 Input : Vlan10 Feature: IPV4_INPUT_ARL Output : Output : Entry : Input - 0x70012964 Lapsed time : 149 ns Lapsed time : 2016 ns Input : Port-channel1 Feature: IPV4_INPUT_ARL_SANITY Feature: IPV4_INPUT_IPOPTIONS_PROCESS Output : Entry : Input - 0x70012610 Entry : Input - 0x700127a8 Lapsed time : 394 ns Input : Vlan10 Input : Vlan10 Feature: SWPORT_PM_FWD_VLAN Output : Output : Entry : Input - 0x700124a0 Lapsed time : 608 ns Lapsed time : 85 ns Input : VLAN-CPPIF-0010 Feature: IPV4_INPUT_DST_LOOKUP_CONSUME Feature: IPV4_INPUT_GOTO_OUTPUT_FEATURE Output : Entry : Input - 0x70012280 Entry : Input - 0x700127c8 Lapsed time : 960 ns Input : Vlan10 Input : CAPWAP-IF-0x0091000004 Feature: SWPORT_VLAN_LEARN Output : Output : Entry : Input - 0x700124e4 Lapsed time : 192 ns Lapsed time : 938 ns Input : VLAN-CPPIF-0010 Feature: IPV4_INPUT_FOR_US_MARTIAN Output : Entry : Input - 0x70012284 Lapsed time : 2304 ns Input : Vlan10 Feature: SWPORT_VLAN_BRIDGING Output : Entry : Input - 0x700123b8 Lapsed time : 4917 ns Input : Vlan10 Output : Lapsed time : 2624 ns

Capwap Wireless Client From SVI L2/SWPORT Dst lookup (src mac & tunnel lkp Mac lookup (5 tuple vlan lookup) hash) Feature: SWPORT_VLAN_BRIDGING Entry : Input - 0x700123b8 Input : WLCLIENT-IF-0x00a0000002 Feature: CAPWAP_INPUT_REASS_FEATURE Feature: LAYER2_IPV4_INPUT_ARL_SANITY Output : Entry : Input - 0x70012988 Entry : Input - 0x70012968 Lapsed time : 3189 ns Input : CAPWAP-IF-0x0091000004 Input : WLCLIENT-IF-0x00a0000002 Feature: MPASS_SET_IN_VECTOR Output : Output : Entry : Input - 0x700129fc Lapsed time : 502016 ns Lapsed time : 458 ns Input : WLCLIENT-IF-0x00a0000002 Feature: Feature: WLCLIENT_INGRESS_IPV4_FWD Output : CAPWAP_TUNNEL_INGRESS_FEATURE Entry : Input - 0x7001299c Lapsed time : 53 ns Entry : Input - 0x7001298c Input : WLCLIENT-IF-0x00a0000002 Feature: Input : WLCLIENT-IF-0x00a0000002 Output : L2_MC_INPUT_REPLICATION_MODULE Output : Lapsed time : 490 ns Entry : Input - 0x700be7f8 Lapsed time : 2965 ns Input : WLCLIENT-IF-0x00a0000002 Output : Lapsed time : 20864 ns

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 74 PPE Processing: Wired -> Wireless B1.1 If wireless B1.3 L2 (802.3) pkt L2 ( src Sub on vlan mac Apply mac Dst mac egress Block lookup lookup features Pkt Entry Point and learn) If wired B1 mac

Build hdr L2 Switched (802.11, Tunnel l3 L2 lookup capwap, lookup outer ip) B1.2 ACL, QoS …

Flood

B1.1 L2 Tx(Phy port) OR 802.3 pkt Unicast L2 switch

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 75 FIA Tracing: Wired -> Wireless Uidb L2 ( src switch Build hdr mac Dst mac (wlcien (802.11, lookup lookup t capwap, and learn) outer ip)

Feature: CAPWAP_UDP_PORT_SAVE_FEATURE Entry : Output - 0x700129d4 Feature: SWPORT_PM_INPUT_CHECK Feature: SWPORT_VLAN_LEARN Input : VLAN-CPPIF-0011 Entry : Input - 0x7001247c Entry : Input - 0x700124e4 Output : CAPWAP-IF-0x009000000c Input : Port-channel1 Input : VLAN-CPPIF-0011 Lapsed time : 266 ns Output : Output : Feature: CAPWAP_ENCAP_FEATURE Lapsed time : 341 ns Lapsed time : 2720 ns Entry : Output - 0x700129dc Feature: IPV4_INPUT_ARL Feature: SWPORT_VLAN_BRIDGING Input : VLAN-CPPIF-0011 Entry : Input - 0x70012964 Entry : Output - 0x700123b8 Output : CAPWAP-IF-0x009000000c Input : Port-channel1 Input : VLAN-CPPIF-0011 Lapsed time : 106 ns Output : Output : WLCLIENT-IF-0x00a0000016 Feature: CAPWAP_OUTPUT_FRAG_FEATURE Lapsed time : 288 ns Lapsed time : 1536 ns Entry : Output - 0x700129e4 Feature: SWPORT_PM_FWD_VLAN Feature: IPV4_VFR_REFRAG Input : VLAN-CPPIF-0011 Entry : Input - 0x700124a0 Entry : Output - 0x700125bc Output : CAPWAP-IF-0x009000000c Input : VLAN-CPPIF-0011 Input : VLAN-CPPIF-0011 Lapsed time : 245 ns Output : Output : WLCLIENT-IF-0x00a0000016 Feature: CAPWAP_ENCAP_IP_FEATURE Lapsed time : 885 ns Lapsed time : 32 ns Entry : Output - 0x700129f8 Input : VLAN-CPPIF-0011 Output : CAPWAP-IF-0x009000000c Lapsed time : 917 ns

Tunnel L3 L2 lookup After CAPWAP Lookup encap

Feature: IPV4_INPUT_LOOKUP_PROCESS Entry : Output - 0x70012228 Feature: IPV4_INPUT_DST_LOOKUP_ISSUE Input : VLAN-CPPIF-0011 Entry : Output - 0x70012220 Output : Vlan10 Input : VLAN-CPPIF-0011 Lapsed time : 618 ns Output : CAPWAP-IF-0x009000000c Feature: IPV4_INPUT_GOTO_OUTPUT_FEATURE Lapsed time : 74 ns Entry : Input - 0x700127c8 Feature: IPV4_INPUT_ARL Input : VLAN-CPPIF-0011 Feature: SWPORT_VLAN_ADD_TAG Entry : Output - 0x70012964 Output : Vlan10 Entry : Output - 0x700123b4 Input : VLAN-CPPIF-0011 Lapsed time : 266 ns Input : Vlan10 Output : CAPWAP-IF-0x009000000c Feature: IPV4_VFR_REFRAG Output : Vlan10 Lapsed time : 309 ns Entry : Output - 0x700125bc Lapsed time : 693 ns Feature: IPV4_INTERNAL_DST_LOOKUP_CONSUME Input : VLAN-CPPIF-0011 Feature: SWPORT_VLAN_BRIDGING Entry : Output - 0x70012224 Output : Vlan10 Entry : Output - 0x700123b8 Input : VLAN-CPPIF-0011 Lapsed time : 42 ns Input : Vlan10 Output : CAPWAP-IF-0x009000000c Feature: IPV4_OUTPUT_L2_REWRITE Output : Vlan10 Lapsed time : 64 ns Entry : Output - 0x7000ad84 Lapsed time : 2410 ns Feature: IPV4_INTERNAL_FOR_US Input : VLAN-CPPIF-0011 Entry : Output - 0x70012958 Output : Vlan10 Input : VLAN-CPPIF-0011 Lapsed time : 576 ns Output : CAPWAP-IF-0x009000000c Feature: IPV4_OUTPUT_FRAG Lapsed time : 74 ns Entry : Output - 0x700128b4 Input : VLAN-CPPIF-0011 Output : Vlan10 Lapsed time : 42 ns

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 77 B1.1: L2/SWPort Commands WiredWireless B1.1 Wireless-> Dst Wired Wireless MAC Apply L2 Src Mac mac egress Lookup/learn lookup features B1.1

B1.1 Hierarchy show platform hardware chassis active qfp feature swport

Per-VLAN Stats (will also include Injected/drop stats for the specific VLAN) show platform hardware chassis active qfp feature swport datapath vlan

Injected / Drop Stats across all VLANs show platform hardware chassis active qfp feature swport datapath system statistics

MAC table show platform hardware chassis active qfp feature swport datapath mac-table all

VLAN Plumbing Data show platform hardware chassis active qfp feature swport client vlan | mac-table | pm

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 78 B1.1 L2 / Swport Command Output show platform hardware chassis active qfp feature swport datapath vlan Vlan 11 statistics ------Total unknown unicast pkt :49703551112, bytes : 63453389847952 Total broadcast pkt :180837, bytes : 10850220 Total wls_pasv_client bcast pkt :0, bytes : 0 Total ingress multicast pkt :0, bytes : 0 Total non-ip multicast pkt :0, bytes : 0 L2 SISF injected unicast arp request pkt :0, bytes : 0 L2 SISF injected bcast arp request pkt :0, bytes : 0 L2 SISF injected arp reply pkt :0, bytes : 0 L2 SISF injected dhcp discovery pkt :0, bytes : 0 L2 SISF injected dhcp offer pkt :0, bytes : 0 L2 SISF injected dhcp request pkt :0, bytes : 0 L2 SISF injected dhcp ack pkt :0, bytes : 0 L2 SISF injected dhcp nak pkt :0, bytes : 0 L2 SISF injected dhcp others pkt :0, bytes : 0 L2 SISF injected dhcpv6 solicit pkt :0, bytes : 0

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 79 B1.1 L2 / Swport Command Output Show pl hard chass active qfp feature swport datapath mac-table all VLAN MAC Address Type AgeIdx InPkt Interface ------10 0011.1100.03f0 Dynamic 387 116 Port-channel1 10 0011.1100.08f0 Dynamic 391 97 Port-channel1 14 00dd.dddd.0001 WlClient 0 0 WLCLIENT-IF-0x00a0000011 13 00cc.cccc.0003 WlClient 0 0 WLCLIENT-IF-0x00a0000002 10 0011.1100.05f0 Dynamic 403 13 Port-channel1 14 00dd.dddd.0002 WlClient 0 0 WLCLIENT-IF-0x00a0000007 10 0011.1100.07f0 Dyna 14 00dd.dddd.0008 WlClient 0 0 WLCLIENT-IF-0x00a0000015 14 00dd.dddd.0004 WlClient 0 0 WLCLIENT-IF-0x00a0000014 13 00cc.cccc.0006 WlClient 0 0 WLCLIENT-IF-0x00a0000004 14 00dd.dddd.0007 WlClient 0 0 WLCLIENT-IF-0x00a0000006 13 00cc.cccc.0005 WlClient 0 0 WLCLIENT-IF-0x00a0000003 14 00dd.dddd.0006 WlClient 15 14 00dd.dddd.0006 WlClient 0 0 WLCLIENT-IF-0x00a0000008 13 00cc.cccc.0004 WlClient 0 0 WLCLIENT-IF-0x00a000000a 10 bc16.f502.2446 Dynamic 2 1240661004 Port-channel1

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 80 B1.2: CAPWAP Commands WiredWireless B1.2 Build hdr (802.11, Tunnel L3 Wireless-> Wired Capwap tunnel lkp (5 capwap, lookup tuple hash) outer ip) B1.2

B1.2

CAPWAP Tunnel Drops [For various reasons] show platform hardware chassis active qfp feature wireless capwap datapath statistics drop

CAPWAP Fragmentation/Reassembly Stats (Frag: towards wireless Reassembly: from wireless] show platform hardware chassis active qfp feature wireless capwap datapath statistics fragmentation | reassembly

Individual CAPWAP Tunnel Stats|Summary [Shows stats for both directions: Rx =upstream; Tx=Downstream] show platform hardware chassis active qfp feature wireless capwap datapath mac-addr statistics start|stop|clear [ NOTE: RX and TX Stats NEED to be enabled with start|stop; Drop Stats are always enabled ]

CAPWAP Tunnel Plumbing Details (including getting the mac-addr) show platform hardware chassis active qfp feature wireless capwap cpp-client <…..>

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 81 CAPWAP Reassembly Command Output B1.2 show platform hardware chassis active qfp feature wireless capwap datapath statistics [reassembly (Upstream) | fragmentation (Downstream)] CPP Wireless Reassembly Packet stats: (outstanding pkt_cnt 0)

Description Packet Count Octet Count ------Capwap Reassembled Packets 105284695862 162564170194361 Shows packet counts Capwap Fragments Received 210599192415 168694420344989 Capwap Fragments Consumed (Saved) 105299081329 159383610078619 for fragmented Capwap Fragments Dropped 0 0 packets and how Capwap Reassembly Timeouts 14385468 14385468 Error - Early-drop fragments 10221624 4487789787 many have been Error - Invalid packet size 0 0 Error - Fragment size too big 0 0 recycled Error - Too many fragments 0 0 Error - Ovlp/Dupl fragments 5193600 5193600 Error - Allocate Info chunk memory 0 0 Error - Free Info chunk memory 0 0 May indicate heavy Error - Hash bucket threshold 0 0 Error - Packet Reassembly Error 0 0 QFP core load of reassembly packets

Apply Wireless-> Wired Dst mac Wireless MAC Wireless egress Client lookup lookup B1.2 (src mac & B1.3 features vlan lookup) B1.3

B1.3

Hierarchy show platform hardware chassis active qfp feature wireless wlclient

WlClient Plumbing Details show platform hardware chassis active qfp feature wireless wlclient cpp-client <….>

Individual Client Stats|Summary [Shows stats for both directions: Rx =upstream; Tx=Downstream] show platform hardware chassis active qfp feature wireless wlclient datapath mac-addr statistics start|stop|clear [NOTE: Rx/Tx Stats will be shown when enabled via the start|stop command; Drop stats are always enabled]

All Client Drops show platform hardware chassis active qfp feature wireless wlclient datapath statistics drop

Capture Point B FIA Tracing at PPE (with Pkt in and out Copy)

clear platform hardware chassis active qfp feature packet-trace QFP clear platform condition all debug platform condition interface [internal-RP | tenGigE | port-channel] both debug platform packet-trace packet 8192 fi cir debug platform packet-trace copy packet both l2 size 256

debug platform condition start To ASIC and PHY debug platform condition stop

show pl ha cha act qfp feat packet-trace summary show pl ha cha act qfp feat packet-trace packet show pl ha ch act qfp fe packet-trace pac all dec | red bootflash:pkt-trace.txt

For a network wide packet tracer, see Cisco True Trace in BRKARC 3147 !

• Web interface to the existing EPC CLI “monitor capture …”

• One click start/stop/download

• Physical and VLAN interfaces can be selected

Capture Point B

QFP Embedded Pkt capture(EPC) at PPE (select the interface where capture is needed)

monitor capture test interface GigabitEthernet2 both monitor capture test control-plane both monitor capture test match any To ASIC and PHY monitor capture test buffer size 100 circular monitor capture test limit pps 1000000 config ter platform punt-policer epc 65000 monitor capture test start monitor capture test stop monitor capture test export bootflash:test.pcap

Ping Traceroute Debug bundle Show techs Specific cases tips

• show tech wireless

• show tech wireless ap

• show tech wireless datapath ap/client mac-address

• show tech wireless multicast

• show tech wireless qos

• show tech wireless client

• show tech wireless fabric

event manager applet catchall event cli pattern ".*" sync no skip no action 1 syslog msg "$_cli_msg

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 96 Useful commands and tools Troubleshooting High Availability #show wireless stats redundancy summary #show wireless stats redundancy history #show wireless stats redundancy statistics #show chassis

• See HA RA tracing: # show logging process stack_mgr internal

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 97 Useful commands and tools Verifying hardware ------show inventory ------(...) NAME: "subslot 0/0 transceiver 1", DESCR: "GE T" PID: ABCU-5710RZ-CS4 , VID: , SN: AGM135322W7 (...)

#show hw-module subslot 0/0 transceiver 1 idprom detail (big output on that SFP)

Discovery requests received from total number of APs : 3

AP Radio MAC AP Ethernet MAC IP Address Last Success time Last failure type Last failure time ------0062.ec06.8d10 0062.ec4a.59b8 10.48.39.177 01/15/19 03:27:43 None NA 00be.75ba.1220 7069.5a3b.1fd0 192.168.61.74 01/15/19 06:34:22 None NA 700f.6a41.cf60 0000.0000.0000 0.0.0.0 01/01/70 00:00:00 Non-wireless Mgmt interface NA

Number of APs: 2

Base MAC Ethernet MAC AP Name IP Address Status Last Failure Type Last Disconnect Reason ------

0062.ec06.8d10 0000.0000.0000 NA NA Not Joined Dtls NA 00be.75ba.1220 0000.0000.0000 NA NA Not Joined Dtls NA 7c0e.cea0.7680 58f3.9cc4.4864 AP58f3.9cc4.4864 192.168.16.92 Not Joined NA Heart beat timer expiry 84b8.021d.1c70 64f6.9d58.5d3c 2702I-sniffer 192.168.16.198 Joined Join Wtp reset config cmd sent a80c.0ddb.c720 a80c.0dd2.1fa8 APa80c.0dd2.1fa8 192.168.18.52 Joined NA DTLS alert from AP

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 100 Useful commands and tools Verifying clients #show wireless stats client detail Total Number of Clients : 0 Protocol Statistics ------Protcol Client Count 802.11b 0 802.11g 0 802.11a 0 802.11n-2.4 GHz 0 802.11n-5 GHz 0 802.11ac 4 802.11ax-5 GHz 0 802.11ax-2.4 GHz 0

Monitoring interval : 10 minute(s) Current client state statistics: ------Authenticating : 0 Mobility : 0 IP Learn : 0 Webauth Pending : 0 Run : 4

Client Summary ------Current Clients : 3 Excluded Clients : 0 Disabled Clients : 0 Foreign Clients : 0 Anchor Clients : 0 Local Clients : 3 client global statistics: ------Total association requests received : 30 Total association attempts : 20 Total FT/LocalAuth requests : 0 Total association failures : 2 Total association response accepts : 28 Total association response rejects : 2 Total association response errors : 0 Total association failures due to blacklist : 0 Total association drops due to multicast mac : 0 Total association drops due to throttling : 0

Total association drops due to unknown bssid : 0 Total association drops due to parse failure : 0 Total association drops due to other reasons : 0 Total 11r ft authentication requests received : 0 Total 11r ft authentication response success : 0 Total 11r ft authentication response failure : 0 Total 11r ft action requests received : 0 Total 11r ft action response success : 0 Total 11r ft action response failure : 0 Total roam attempts : 0 Total CCKM roam attempts : 0 Total 11r roam attempts : 0 Total 11i fast roam attempts : 0 Total 11i slow roam attempts : 0 Total other roam type attempts : 0 Total roam failures in dot11 : 0

(100 lines more of these)

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 103 Useful commands and tools Verifying clients (part 4) Latency Distribution (ms) 1 - 100 : 0 100 - 200 : 0 200 - 300 : 2 300 - 600 : 1 600 - 1000 : 0 1000+ : 3

Webauth HTTP Statistics ------Intercepted HTTP requests : 0 IO Read events : 0 Received HTTP messages : 0 IO write events : 0 Sent HTTP replies : 0 IO AAA messages : 0 SSL OK : 0 SSL Read would block : 0 SSL write would block : 0 Socket opens : 0 Socket closes : 0

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 104 Useful commands and tools Verifying clients (part 5) Webauth HTTP status counts ------HTTP 200 OK : 0 HTTP 201 Created : 0 HTTP 202 Accepted : 0 HTTP 203 Provisional Info : 0 HTTP 204 No Content : 0 HTTP 300 Multiple Choices : 0 HTTP 301 Moved Permanently : 0 HTTP 302 Moved Temporarily : 0 HTTP 303 Method : 0 HTTP 304 Not Modified : 0 HTTP 400 Bad Request : 0 HTTP 401 Unauthorized : 0 HTTP 402 Payment Required : 0 HTTP 403 Forbidden : 0 HTTP 404 Not Found : 0 HTTP 405 Method Not Allowed : 0 HTTP 406 None Acceptable : 0 HTTP 407 Proxy-Auth Required : 0 HTTP 408 Request Timeout : 0 HTTP 409 Conflict : 0 ….

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 105 Useful commands and tools Verifying clients (part 6) Total client delete reasons ------Unknown :0 Deauthentication or disassociation request :0 Session Manager :0 L3 authentication failure :0 Delete received from AP :0 WLAN down :0 AP down/disjoin :0 Connection timeout :0 MAC authentication failure :0 Datapath plumb :0 Due to SSID change :0 Due to VLAN change :0 Admin deauthentication :0 QoS failure :0 WPA key exchange timeout :0 WPA group key update timeout :0 802.11w MAX SA queries reached :0

(100 lines more….)

Number of APs: 2

AP Name AP Mac Site Tag Name Policy Tag Name RF Tag Name Misconfigured Tag Source ------2702I-sniffer 64f6.9d58.5d3c default-site-tag default-policy-tag default-rf-tag No Default APa80c.0dd2.1fa8 a80c.0dd2.1fa8 default-site-tag default-policy-tag default-rf-tag No Default

Sep 19 14:17:09.787: %CONFIG_VALIDATOR_MESSAGE-5-EWLC_GEN_ERR: Chassis 2 R0/0: wncmgrd: Error in AP MAC: 4001.7ab2.c41e Applied policy-tag : noexiste definition does not exist

#sh ap tag summary

LabAP 4001.7ab2.c41e default-site-tag default-policy-tag default-rf-tag Yes Static

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 107 Useful commands and tools AP tags verification #show ap name APa80c.0dd2.1fa8 tag detail AP Name : APa80c.0dd2.1fa8 AP Mac : a80c.0dd2.1fa8

Tag Type Tag Name ------Policy Tag default-policy-tag RF Tag default-rf-tag Site Tag default-site-tag

Policy tag mapping ------WLAN Profile Name Policy Name VLAN Central Switching IPv4 ACL IPv6 ACL ------dot1x-test default-policy-profile VLAN0711 ENABLED Not Configured Not Configured

Site tag mapping ------Flex Profile : default-flex-profile AP Profile : default-ap-profile Local-site : Yes

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 108 Conclusion : troubleshooting recap Step 1 : show log Dec 18 13:38:18.228: %LINEPROTO-5-UPDOWN: Line protocol on Interface Capwap1, changed state to down Dec 18 13:38:18.205: %CAPWAPAC_SMGR_TRACE_MESSAGE-3-EWLC_GEN_ERR: Chassis 1 R0/0: wncd: Error in Session-IP: 192.168.16.134[5264] Mac: 7069.5a51.46e0 Heartbeat timer expiry for AP. Close CAPWAP DTLS session Dec 18 13:38:18.231: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: 4802paolo, MAC: 4c77.6d9e.60e4 Disjoined Dec 21 06:19:45.425: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the maximum limit(500)as specified by the platform. ..Dec 21 06:20:00.748: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the maximum limit(500)as specified by the platform. .Dec 21 06:20:00.785: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the maximum limit(500)as specified by the platform. .Dec 21 06:20:15.616: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the maximum limit(500)as specified by the platform.

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 109 Conclusion : troubleshooting recap Step 2 : show logging trace-on-failure summary 2018/12/14 13:08:12.049 0x10000001af2cc 8875.56c6.f000 AP_JOIN_FAIL : Apmgr failure reason : Unsupported ap, 2018/12/14 13:50:07.891 0x10000001b3ae0 08cc.68b4.4660 AP_IMG_DWLD_FAIL : Image download failure : Unknown

-> show logging profile wireless filter uuid to-file -> show logging profile wireless filter mac to-file

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 110 Conclusion : troubleshooting recap Step 3 : show logging profile wireless filter-mac to-file Run the (always-on) logs for the given MAC/IP you are troubleshooting.

You will see NOTICE level logs for 24/7 since as long as the max storage time of the WLC allows (depends on load) and any debug level if RA trace was enabled at any point.

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 111 Conclusion : troubleshooting recap Step 4 : problem reproducible ? Need more logging ? RA-trace WLC# debug wireless mac aaaa.bbbb.cccc monitor-time 10

WLC#more bootflash:

BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 112 Conclusion : troubleshooting recap Step 5 : TAC case • RA-trace output (internal level, while we’re at it) or show logging profile wireless of always-on output filtered for the problematic mac or timestamp

• Relevant show techs (at least show tech + show tech wireless)

• Your observations from “show logging” or “show logging trace-on-failure summary” (timestamps, affected macs)

• Core dump files from the web UI troubleshooting page (if the problem is a crash)

Questions? Use Cisco Webex Teams (formerly Cisco Spark) to chat with the speaker after the session How 1 Find this session in the Cisco Events Mobile App 2 Click “Join the Discussion” 3 Install Webex Teams or go directly to the team space 4 Enter messages/questions in the team space cs.co/ciscolivebot#BRKEWN-3013

• Please complete your Online Session Survey after each session

• Complete 4 Session Surveys & the Overall Conference Survey (available from Thursday) to receive your Cisco Live T- shirt

• All surveys can be completed via the Cisco Events Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on demand after the event at ciscolive.cisco.com

Related Demos in Walk-in Meet the sessions the Cisco self-paced engineer Showcase labs 1:1 meetings