BRKEWN-3013 Troubleshoot Catalyst 9800 Wireless Controllers Serviceability enhancements of the new platform Nicolas Darchis, CX Technical Leader Session objectives • Understand the Catalyst 9800 WLC architecture in order to be able to know when to use which troubleshooting tool • software • hardware • relationship between these two • Understand how features process packets through IOS-XE • Understand how to easily debug the platform • presentation of recent serviceability enhancements • spare memorizing – focus on understanding • not “tips & tricks” but debugging strategy and tools BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 3 Agenda • Introduction • C9800 general architecture • Hardware • Software • Life of a packet • IOS-XE logging architecture • General concepts • Logging features and techniques (multiple parts) • Packet captures and tracing • Useful commands and tools • Conclusion BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 4 Introduction • 1-2 years serviceability effort (and • Sudha Katgeri, US TAC TL more to come) • Nicolas Darchis, EMEA TAC TL • Testing and feedback since 16.7 code • Patnership with BU escalation (internal only) and engineering Tech Leads • TAC involved in serviceability requests 2 releases in advance • Cross-technology partnership within TAC and between BUs BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 5 Introduction Instructions unclear, got stuck in washing machine Debugging process is extremely different… …for the better ! … …once you understand why ! BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 6 Introduction • Serviceability is a journey with a clear destination … that is not yet in sight • Cisco AND customers want less TAC cases and faster/easier TAC cases BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 7 Introduction Platform objectives • A single way to enable debugs, not having to remember and enable dozens of debug commands • Capacity to trace the path and time of a packet through the platform, including all the features it hits on the way • Obtaining debug logs of past event in their context even without having enabled any debug manually • Being able to verify things at every layer of the platform (control or data plane) BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 8 Catalyst 9800 general architecture • Hardware • Software • Life of a Packet BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 9 Acronyms • RP – Route Processor • FP – Forwarding Processor = ESP (Embedded Service Processor) • CPP – Cisco Packet Processor Complex= QFP (Quantum Flow Processor) • PPE – Packet Processing Engine • IOCP – I/O Control Processor • FECP – Forwarding Engine Control Processor • SPA – Shared Port Adapter • SIP – SPA Interface Processor • IOSd – IOS image that runs as a process on the RP • FMAN – Forwarding manager (FMAN-RP, FMAN-FP) • btrace – binOS tracing – the binary logging system used by binOS processes • EOBC = Ethernet Out of Band Channels – Packet Interface for Card to Card Control Traffic • IOS-XE (BinOS) = Linux Based Software Infrastructure That Executes on MCP BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 10 “So this Catalyst 9800 means the WLC is now a switch hardware right?” The skeptical network admin Company XYZ BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 11 C9800 WLC hardware : 2 architectures FED/Doppler architecture • 9800-SW on 9300/9500 switches uses Doppler chipset • Doppler (UADP) hardware chipset. FED (Forwarding Engine Driver) architecture for packet forwarding • Software & control plane stays the same but dataplane is completely different • We will not cover this dataplane type. More details in BRKARC-2035 BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 12 C9800 WLC hardware : 2 architectures CPP architecture • Cloud and appliances : same CPP architecture ! • Cloud just virtualizes the dataplane • Appliance have the QFP (Quantum Flow Processor) hardware for the CPP. • No differences apart from the performance (and Manufacture Installed Certificate) • More details in BRKARC-3147 (ASR1k architecture and troubleshooting) BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 13 C9800 WLC hardware Tips about the self-signed certificate on the C9800-CL • While appliances have a Manufacture-Installed Certificate, virtual controllers can only rely on Self-Signed Certificate. • It is generated by the day-0 wizard once you enabled the wireless management interface and configured the country. • In case of issues, it can be re-generated with this CLI : #wireless config vwlc-ssc key-size 2048 signature-algo sha256 password <yourpassword> • As of 16.10, all SSC have the same Serial Number which can cause browser issues when you have multiple C9800-CL. You can install another certificate for UI management to work around the problem. BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 14 C9800 WLC hardware Cisco “Quantum Flow Processor” • Packet Processing Engine (QFP-PPE) – 64 Packet Processors with 4 threads per core – 1.5GHz Tensilica ISA processors + DRAM packet memory – Single 80M TCAM4 I/F – C-language for feature development; extensive development support tools Multi-Core (64) Packet Processor – HW assist for flow-locks, look-ups, stats, WRED, policers, range lookup, crypto, CRC • Buffer/queue subsystem (QFP-BQS) – HW hierarchical 3-parameter (min, max & excess) scheduler – Fully configurable # of layers based on HQF – Priority propagation through the multiple layers Traffic Manager (BQS) BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 15 C9800 WLC hardware Example layout of a 9800-80 • 9800-80 is actually capable of 100Gbps QFP 1 QFP 0 • 2 load-balanced QFPs (1 in 9800-40) eUSB • 1 crypto chip TCAM • 12-core CPU in 9800-80 PCH • 8-core CPU in 9800-40 CPU SSD BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 16 C9800 WLC Hardware Appliances High Level Block Diagram Block X86 CPU Block Crypto chip C complex D C9800-40: Single QFP QFP Astro C9800-80: Dual QFP Block B Buffering Ingress ASIC (NP5c/Ezchip) buffering 10G PHY Block A SFP+ x4 or x8 BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 17 Catalyst 9800 general architecture • Hardware • Software • Life of a Packet BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 18 Software architecture IOS-XE concept • The underlying operating system is binOS (unix) • IOS(d) is a process like another. IOSd takes care of routing, CLI, multicast and interfaces IOS-XE Polaris 16.10 IOSd IOSd Hosted subsystemIOSd IOSd subsystemIOSd Apps Blob subsystem WNCd WNCd HA WNCd Management interface RRMd Module drivers Rogued Kernel Wireshark BinOS BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 19 Software architecture Wireless process WNCd • SANET : client AAA policies • SISF : client IP learning SANET SISF • WNCd : “controller” process managing APs and clients WNCd Legend database Wireless process BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 20 Software architecture Wireless processes WNCmgr WNCd WNCd smd wstatsd rogued mobilityd Horizontally scaled RRM NMSPd Legend database Wireless process BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 21 Why horizontally scaling processes? • To take real profit of multi-core systems, otherwise: AP distribution across WNCd processes: • APs of the same site join the same WNCd process • Exception is on the default site where APs are load-balanced BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 22 Software architecture IOSd, FMAN-RP,FMAN/FP IOSd FMAN RP WNCd Wireless WNCd WNCd processes FMAN ASIC FP driver Legend database Wireless Programmable interface Polaris infra process Punt path BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 23 Software architecture Dataplane IOSd FMAN WNCd RP WNCd WNCd Wireless FMAN processes ASIC FP Linux LSMPI LFTS TCP/IP Dataplane (CPP/ Doppler) Legend database Wireless Linux Polaris infra Programmable interface process kernel Punt path BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 24 Software architecture rd Cisco Web DMI 3 Prime Management party DNA-C UI procs REPM IOSd infra DBM CLI ODM FMAN agent RP WNCd WNCd Wireless WNCd FMAN processes ASIC FP Linux LSMPI LFTS TCP/IP Dataplane (CPP/ Doppler) Legend database Wireless Linux Management Programmable interface Polaris infra process kernel access Crimson access Punt path BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 25 Catalyst 9800 general architecture • Hardware • Software • Life of a Packet BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public 26 Life of a Packet : Dataplane wireless client traffic X86 CPU complex Crypto chip Traffic Forwarding Path: Astro Faceplate Ports EZChip QFP NP5c Astro Yoda Complex Buffering Ingress ASIC buffering Block A/B/C/D troubleshooting 10G PHY commands in following hidden slides SFP+ x4 or x8 BRKEWN-3013 © 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public Life of a Packet : Control