Cisco Express Forwarding
Nakia Stringfield, CCIE No. 13451 Russ White, CCIE No. 2635 Stacia McKee
Cisco Press 800 East 96th Street Indianapolis, IN 46240 USA
ii
Cisco Express Forwarding Nakia Stringfield, Russ White, Stacia McKee Copyright© 2007 Cisco Systems, Inc. Published by: Cisco Press 800 East 96th Street Indianapolis, IN 46240 USA All rights reserved. No part of this book may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system, without written permission from the publisher, except for the inclusion of brief quotations in a review. Printed in the United States of America 2 3 4 5 6 7 8 9 0 Second Printing August 2013 ISBN-10: 1-58705-852-9 ISBN-13: 978-1-58705-852-3
Warning and Disclaimer This book is designed to provide information about Cisco Express Forwarding (CEF). Every effort has been made to make this book as complete and as accurate as possible, but no warranty or fitness is implied. The information is provided on an “as is” basis. The authors, Cisco Press, and Cisco Systems, Inc., shall have nei- ther liability nor responsibility to any person or entity with respect to any loss or damages arising from the informa- tion contained in this book or from the use of the discs or programs that may accompany it. The opinions expressed in this book belong to the authors and are not necessarily those of Cisco Systems, Inc.
Trademark Acknowledgments All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capital- ized. Cisco Press or Cisco Systems, Inc., cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.
Feedback Information At Cisco Press, our goal is to create in-depth technical books of the highest quality and value. Each book is crafted with care and precision, undergoing rigorous development that involves the unique expertise of members from the professional technical community. Readers’ feedback is a natural continuation of this process. If you have any comments regarding how we could improve the quality of this book, or otherwise alter it to better suit your needs, you can contact us through email at [email protected]. Please make sure to include the book title and ISBN in your message. We greatly appreciate your assistance. iii
Corporate and Government Sales Cisco Press offers excellent discounts on this book when ordered in quantity for bulk purchases or special sales. For more information please contact: U.S. Corporate and Government Sales 1-800-382-3419 corpsales@pearson- techgroup.com. For sales outside the U.S. please contact: International Sales [email protected].
Publisher Paul Boger Cisco Representative Anthony Wolfenden Cisco Press Program Manager Jeff Brady Associate Publisher David Dusthimer Executive Editor Brett Bartow Managing Editor Patrick Kanouse Development Editor Dayna Isley Senior Project Editor San Dee Phillips Copy Editor Written Elegance, Inc. Technical Editors Neil Jarvis, LJ Wobker Team Coordinator Vanessa Evans Book and Cover Designer Louis Adair Composition Mark Shirar Indexer Tim Wright Proofreader Molly Proue iv
About the Authors Nakia Stringfield, CCIE No. 13451, is a network consulting engineer for Advanced Services at Cisco in Research Triangle Park, North Carolina, supporting top financial customers with network design and applying best practices. She was formerly a senior customer support engineer for the Routing Protocols Technical Assistance Center (TAC) team, troubleshooting issues related to CEF and routing protocols. Nakia has been with Cisco for more than six years, previously serving as a technical leader for the Architecture TAC team. She has given training courses on CEF operation and troubleshooting for inter- nal employees. Nakia also worked for a year with IBM Global Services LAN Support in Research Tri- angle Park, North Carolina. Nakia attended North Carolina State University and completed her bachelor of science degree in electrical engineering in 1996. She also earned a master of science in computer net- working and computer engineering from North Carolina State University in 2000. Russ White, CCIE No. 2635, is a member of the Routing Protocol Design and Architecture Team at Cisco, Research Triangle Park, North Carolina. He is a member of the Internet Engineering Task Force (IETF) Routing Area Directorate, a cochair of the Routing Protocols Security Working Group in the IETF, a regular speaker at Networkers, a member of the Cisco Certified Internetwork Expert (CCIE) Content Advisory Group, a member of the core team developing the new Cisco Design certification, a regular contributor to the Internet Protocol Journal, and the coauthor of six other books about routing and routing protocols, including Optimal Routing Design, from Cisco Press. Russ primarily works in the development of new features and design architectures for routing protocols. Stacia McKee is a customer support engineer and technical leader of the Routing Protocols (RP) Tech- nical Assistance Center (TAC) team at Cisco in Research Triangle Park, North Carolina. This team focuses on providing postsales support of IP routing protocols, Multiprotocol Label Switching (MPLS), quality of service (QoS), IP multicast, and many other Layer 3 technologies. Stacia has been with Cisco for more than six years, previously serving as a technical leader of the Architecture TAC team and mem- ber of the WAN/Access TAC team. She has created and presented training on packet switching, router architecture, and troubleshooting for internal employees. Stacia has also been a technical editor and reviewer of Cisco.com technical documentation, mainly in router and IOS architecture and IP routing protocols technologies. She works closely with the IP Routing and IP Services groups within the Cisco Network Software and Systems Technology Group (NSSTG) on customer problems and early field tri- als. In 2000, Stacia completed her bachelor of science degree in computer information systems, bache- lor of science degree in business administration, and bachelor of arts degree in computer science at the College of Charleston in Charleston, South Carolina. v
About the Technical Reviewers Neil Jarvis has been a software engineer in the networking industry since 1990. He is currently employed by Cisco Systems as a distinguished engineer, responsible for the architecture and develop- ment of switching control and data plane software, including Cisco Express Forwarding (CEF). He was a technical contributor and editor of a number of IEEE 802 standards, including 802.1 (bridging) and 802.5 (token ring). He was IEEE 802.1 vice-chair for a number of years. Neil graduated with a master’s degree in microelectronic systems engineering from UMIST (Manchester, England) in 1989 and now lives with his wife in Edinburgh, Scotland. LJ Wobker, CCIE No. 5020, holds a bachelor of science degree in computer science from North Caro- lina State University in Raleigh, North Carolina. He started his networking career running cables as a college intern in the Cisco Research Triangle Park TAC lab and has worked in TAC, Advanced Services, and software development. For the last five years, LJ has been a technical marketing engineer, support- ing the Cisco 12000 and CRS-1 series routers. vi
Dedications Nakia Stringfield: I would like to dedicate this book to my wonderful, supportive husband, Kwame Stringfield, and to our beautiful daughter, Kyra. Most of all, thanks go to God for favor and challenging opportunities. Thanks to my parents, Robert and Annette; my family; my pastors; Dr. Frank and JoeNell Summerfield; and my friends for their many prayers and for believing in me. Russ White: I would like to dedicate this book to my two daughters, Bekah and Hannah, as well as to my beautiful wife, Lori. I would like to thank God for the opportunities and skills to work on routers, routing, and books. Stacia McKee: I would like to dedicate this book in memory of my former colleague and dearest friend, Parag Avinash Kamat (July 19, 1977–August 19, 2004). May his memory live on forever. I would like to thank my wonderful husband, Michael McKee, and my parents, Richard and Sidney Froom, for their love, patience, and support while completing this project. I also thank God for all His blessings in my life. vii
Acknowledgments This book would not have been possible without the help of many people whose various comments and suggestions helped to formulate this project. First, we would like to give special recognition to Richard Froom for providing crucial direction and valuable feedback for this book. We also want to thank the technical reviewers for this book, Neil Jarvis and LJ Wobker. Finally, we want to thank Brett Bartow, Chris Cleveland, and Dayna Isley, as well as the other people at Cisco Press, for working with us, keeping us on track, and getting this book published. viii
This Book Is Safari Enabled
The Safari® Enabled icon on the cover of your favorite technology book means the book is available through Safari Bookshelf. When you buy this book, you get free access to the online edition for 45 days. Safari Bookshelf is an electronic reference library that lets you easily search thousands of technical books, find code samples, download chapters, and access technical information whenever and wherever you need it. To gain 45-day Safari Enabled access to this book: • Go to http://www.ciscopress.com/safarienabled • Complete the brief registration form • Enter the coupon code R7CH-25PD-7T4V-4VDV-RYMJ If you have difficulty registering on Safari Bookshelf or accessing the online edition, please e-mail [email protected]. ix
Contents at a Glance Introduction xvi Part I Understanding, Configuring, and Troubleshooting CEF 3 Chapter 1 Introduction to Packet-Switching Architectures 5 Chapter 2 Understanding Cisco Express Forwarding 51 Chapter 3 CEF Enhanced Scalability 81 Chapter 4 Basic IP Connectivity and CEF Troubleshooting 103 Part II CEF Case Studies 135 Chapter 5 Understanding Packet Switching on the Cisco Catalyst 6500 Supervisor 720 137 Chapter 6 Load Sharing with CEF 163 Chapter 7 Understanding CEF in an MPLS VPN Environment 217 Part III Appendix 257 Appendix A Scalability 259 Index 264 x
Contents Introduction xvi Part I Understanding, Configuring, and Troubleshooting CEF 3 Chapter 1 Introduction to Packet-Switching Architectures 5 Routing and Switching 5 Understanding Broadcast and Collision Domains 5 Broadcast and Collision Domains 6 Broadcast and Collision Domains in Routing 7 Layer 3 Switching 8 Understanding Router Pieces and Parts 9 Interface Processors 10 Central Processing Unit 11 Memory 11 Backplanes and Switching Fabrics 11 Shared Memory 11 Crossbar Switching Fabric 13 Bus Backplanes 14 Cisco IOS Software: The Brains 17 Memory Management 17 Memory Pools 17 Memory Regions 18 Packet Buffers 20 Interaction with Interface Processors 28 Processes and Scheduling 28 Process Memory 28 Process Scheduling 29 Understanding the Scheduler 29 Process Life Cycle 29 Process Priorities 32 Scheduling Processes 32 Process Watchdog 34 Special Processes 35 Putting the Pieces Together: Switching a Packet 35 Getting the Packet off the Network Media 35 Inbound Packets on Shared Media Platforms 36 Inbound Packets on Centralized Switching Routers with Line Cards 37 Inbound Packet Handling on Distributed Switching Platforms 38 Switching the Packet 39 Switching the Packet Quickly During the Receive Interrupt 39 Process-Switching the Packet 41 Transmitting the Packet 44 xi
Hardware and Software show Commands 45 Summary 48 Chapter 2 Understanding Cisco Express Forwarding 51 Evolving Packet-Switching Methods 51 Process Switching 51 Fast Switching 52 What Is CEF? 53 CEF Tables 54 Forwarding Information Base (FIB) 54 The Adjacency Table 60 Relating the CEF Tables 61 CEF Table Entries 62 FIB Entries 62 Attached FIB Entry 63 Connected FIB Entry 63 Receive FIB Entry 63 Recursive FIB Entry 64 Default Route Handler FIB Entry 66 ADJFIB FIB Entry 66 Learned from IGPs 67 Generic FIB Entries 67 Interface-Specific FIB Entries 68 FIB Entries Built for a Multiaccess Network Interface 68 FIB Entries Built on a Point-to-Point Network Interface 69 FIB Entries Built on a 31-Bit Prefix Network Interface 69 Special Adjacencies 69 Auto Adjacencies 70 Punt Adjacency 70 Glean Adjacency 71 Drop Adjacency 72 Discard Adjacency 73 Null Adjacency 73 No Route Adjacencies 74 Cached and Uncached Adjacencies 74 Unresolved Adjacency 75 Switching a Packet with CEF 75 The CEF Epoch 77 Configuring CEF/dCEF 77 Summary 78 References 79 xii
Chapter 3 CEF Enhanced Scalability 81 Fundamental Changes to CEF for CSSR 82 Data Structures 82 Switching Path Changes 84 Changes to show Commands 86 show ip cef 86 show ip cef interface 86 show ip cef summary 87 show cef state capabilities 88 New show ip cef Commands 89 show ip cef tree 89 show ip cef internal 90 show ip cef switching statistics 91 New show cef Commands 91 CEF Event Logger 94 CEF Consistency Checker 97 Passive Checkers 97 Active Checkers 97 Consistency-Checking Process 98 New CEF Processes 100 FIB Manager 100 Adjacency Manager 100 Update Manager 100 Summary 101 Chapter 4 Basic IP Connectivity and CEF Troubleshooting 103 Troubleshooting IP Connectivity 103 Accurately Describe the Problem 104 Scoping the Network Topology 105 Reviewing the OSI Model for Troubleshooting 106 Troubleshooting Physical Connectivity 106 Troubleshooting Layer 2 Issues 107 Verifying the ARP Table 108 Verifying the Routing Table 111 Using IOS Ping with the Record Option to Rule Out CEF 115 Troubleshooting the CEF FIB Table 116 Verifying the CEF Configuration 117 Confirming the IP CEF Switching Path 119 Using CEF Accounting Counters to Confirm the Switching Path 123 Verifying the CEF Switching Details 123 xiii
Verifying the Adjacency Table 126 Hardware-Specific Troubleshooting 128 Troubleshooting Punt Adjacencies 129 Understanding CEF Error Messages 131 Troubleshooting Commands Reference 131 Summary 133 References 133 Part II CEF Case Studies 135 Chapter 5 Understanding Packet Switching on the Cisco Catalyst 6500 Supervisor 720 137 CEF Switching Architecture on the Cisco Catalyst 6500 137 Understanding Software-Based CEF and Hardware-Based CEF 137 Centralized and Distributed Switching 138 Troubleshooting CEF on the Catalyst 6500 SUP720 Platforms 139 Simple Checking of Connectivity 139 Systematic Checking of Connectivity 141 Troubleshooting Load Sharing 148 Summary 149 References 149 Chapter 6 Load Sharing with CEF 163 Benefits of Load Sharing 163 Load Sharing with Process Switching and Fast Switching 164 Comparing CEF Per-Packet and CEF Per-Destination Load Sharing 168 Understanding Per-Destination Load Sharing 168 Understanding Per-Packet Load Sharing 169 Minimizing Out-of-Order Packets 169 Configuring CEF Per-Packet Load Sharing 170 CEF Architecture and Load Sharing 171 CEF Load Sharing Across Parallel Paths 173 CEF Per-Destination Example 173 CEF Per-Packet Example 180 Per-Packet Load Sharing on Hardware-Based Platforms 184 CEF Per-Packet Load Sharing on the Cisco GSR Platform 185 CEF Load-Sharing Troubleshooting Examples 186 xiv
CEF Per-Destination Load Sharing Overloading One Link 186 CEF Per-Packet Load Sharing Causing Performance Issues 198 Troubleshooting a Single-Path Failure with CEF Load Sharing 200 CEF Traffic-Share Allocation 202 CEF Polarization and Load-Sharing Algorithms 210 Original Algorithm 212 Universal Algorithm 212 Tunnel Algorithm 213 Hardware Platform Implementations 213 Summary 214 References 215 Chapter 7 Understanding CEF in an MPLS VPN Environment 217 An Internet Service Provider’s Simple MPLS VPN Design 217 Understanding the CEF and MPLS VPN Relationship 219 Case 1: Label Disposition 221 Case 2: Label Imposition 222 Case 3: Label Swapping 224 Troubleshooting an MPLS VPN 224 CEF Considerations When Troubleshooting MPLS VPN Across Various Platforms 225 Cisco 7200 Router with an NPE-G2 226 Cisco 7500 Router 226 Cisco Catalyst 6500 with a Supervisor 2 227 Catalyst 6500 with a Supervisor 720 3BXL 228 Cisco 12000 Series Router 231 Cisco 10000 Series Router 236 CEF and MPLS VPN Load-Sharing Considerations 237 PE-CE Load Sharing: CE Multihomed to Same PE 237 PE-CE Load Sharing: Site Multihomed to Different PEs 243 Load Sharing Between P and P Devices 252 CEF and MPLS VPN Load-Sharing Platform Dependencies 253 Summary 253 References 254 Part III Appendix 257 Appendix A Scalability 259 Index 264 xv
Icons Used in This Book
PC Multilayer Terminal Catalyst Router Switch Switch
Network Cloud Line: Ethernet Line: Serial File Server
Command Syntax Conventions The conventions used to present command syntax in this book are the same conventions used in the IOS Command Reference. The Command Reference describes these conventions as follows: • Boldface indicates commands and keywords that are entered literally as shown. In actual con- figuration examples and output (not general command syntax), boldface indicates commands that are manually input by the user (such as a show command). • Italics indicate arguments for which you supply actual values. • Vertical bars (|) separate alternative, mutually exclusive elements. • Square brackets [ ] indicate optional elements. • Braces { } indicate a required choice. • Braces within brackets [{ }] indicate a required choice within an optional element. xvi
Introduction How does a router switch a packet? What is the difference between routing a packet and switching a packet? What is this CEF feature that is referred to in Cisco documentation and commonly found in Cisco IOS commands? This book answers these questions through comprehensive discussions of Cisco Express Forwarding (CEF). CEF is a term used to describe one of the mechanisms used by Cisco IOS routers and Cisco Catalyst switches to forward packets. Other packet-switching mechanisms include process switching and fast switching. CEF is found in almost all Cisco IOS routers and Catalyst switches. However, documentation of the topic is scarce. From a technical support perspective, CEF is a widely misunderstood topic whose implementation varies significantly on multiple Cisco platforms. Cisco engineers, Cisco partners, and customers need material on CEF to properly deploy, maintain, and troubleshoot their networks. CEF offers the following benefits: • Improved performance—CEF is less CPU-intensive than fast-switching route caching. More CPU processing power can be dedicated to Layer 3 services such as quality of service (QoS) and encryption. • Scalability—CEF offers full switching capacity at each line card when distributed CEF (dCEF) mode is active. • Resilience—CEF offers unprecedented levels of switching consistency and stability in large dynamic networks. CEF can switch traffic more efficiently than typical demand-caching schemes.
Goals and Methods This book addresses common misconceptions about CEF and packet switching across various plat- forms. The goal is to help end users understand CEF and know how to troubleshoot, regardless of whether a CEF or another problem is occurring in the network. Little information collectively addresses these concerns because CEF is proprietary. This book helps you understand CEF better by using the fol- lowing methods: • Explaining CEF basics • Supplying troubleshooting scenarios that enhance your ability to recognize common mistakes • Providing best practices for configuration
Who Should Read This Book The focus audience of this book is networking professionals who require an understanding of Cisco packet-forwarding architecture and who are tasked with troubleshooting routing and switching issues in a Cisco network environment. This book is an invaluable guide for those seeking to gain an understand- ing of how CEF works and how to troubleshoot CEF issues on various hardware platforms. xvii
How This Book Is Organized Although this book could be read from cover to cover, it is designed to be flexible and allows you to eas- ily move between chapters and sections of chapters to cover just the material that you need to trouble- shoot an immediate problem or to understand a concept. Cisco Express Forwarding is divided into two parts. The first part of the book provides an overview of packet-switching architectures and CEF operation and advanced features. It also covers the enhanced CEF structure and general troubleshooting. The second part of the book focuses on particular case stud- ies. Because CEF is a widely misunderstood technology, the case studies focus on a list of the common topics that have been problematic for customers and those supporting Cisco networks. The case studies review and expand on material from the previous parts of the book and provide more in-depth analysis of real networking topologies and troubleshooting steps. Part I, “Understanding, Configuring, and Troubleshooting CEF” includes the following chapters: • Chapter 1, “Introduction to Packet-Switching Architectures”—This chapter explains packet-switching architecture and terminology. It also explains utilization of memory and buffers. • Chapter 2, “Understanding Cisco Express Forwarding”—This chapter deals with the basics of CEF architecture and operation. It defines CEF terminology and history. • Chapter 3, “CEF Enhanced Scalability”—This chapter discusses the enhanced CEF struc- ture and its purpose. • Chapter 4, “Basic IP Connectivity and CEF Troubleshooting”—This chapter deals with general troubleshooting in a software-switching environment. Software switching has typically been used on routers. Part II, “CEF Case Studies,” deals with special CEF case studies covering the following common scenarios: • Chapter 5, “Understanding Packet Switching on the Cisco Catalyst 6500 Supervisor 720”—This chapter helps you understand the impact of CEF and learn how packet switching works on a Cisco Catalyst 6500 SUP720. • Chapter 6, “Load Sharing with CEF”—This chapter discusses load sharing with CEF. It covers the purpose, configuration, and troubleshooting of common problems. • Chapter 7, “Understanding CEF in an MPLS VPN Environment”—This chapter explains the impact of CEF in an MPLS VPN environment. The book concludes with Appendix A, “Scalability,” which discusses CEF design considerations that could impact network scalability. xviii
The Future of CEF and Packet Switching Although this book provides solid information for software handling and hardware handling, it does not provide a detailed description of implementation on all Cisco platforms and related technologies. Hard- ware design changes rapidly, and packet handling on one platform could easily consume the entire book. This book does not address Parallel Express Forwarding (PXF), which is used on devices such as Cisco 10000 series routers, Cisco 7600 series Optical Service Modules (OSMs), and Cisco 7300 series rout- ers. PXF leverages a combination of parallel processing and pipelining techniques to the CEF algo- rithms for faster throughput and optimal flexibility through ASIC technology. Because PXF is highly dependent on the platform and specific ASIC technology, it is not covered in this book. Hardware switching will continue to be optimized for performance advantages. Introduction of distrib- uted CEF (dCEF) on Cisco 7500 series routers was a start down this path years ago to offload packet switching from the central processor to the Versatile Interface Processor (VIP) line card. Then progres- sion occurred to hardware-based localized switching on Cisco 6500s with Distributed Forwarding Cards (DFCs), FlexWans, and OSMs. Cisco recently introduced IOS Software Modularity, which provides subsystem In-service Software Upgrades and Process Fault Containment to the Cisco Catalyst 6500 series switches. As you continue to learn more about Cisco Express Forwarding, you may find the following resources helpful: • Bollapragada, V., R. White, and C. Murphy, Inside Cisco IOS Software Architecture, Indianapolis, Indiana: Cisco Press; 2000. Provides a detailed treatment of Cisco 7500 routers and Cisco 7200 routers. • Cisco, Parallel Express Forwarding on the Cisco 10000 Series, www.cisco.com/en/US/ products/hw/routers/ps133/products_white_paper09186a008008902a.shtml. • Cisco, Cisco 7600 Series Router Q & A, www.cisco.com/en/US/products/hw/routers/ps368/ products_qanda_item09186a008017a32b.shtml. • Cisco, PXF Information for Cisco 7304 Routers, www.cisco.com/en/US/products/hw/routers/ ps352/prod_maintenance_guide09186a008057410a.html. • Cisco, Cisco Catalyst 6500 Series Switches with IOS Software Modularity Make IT Managers More Productive and Further Improve Network Reliability, http://newsroom.cisco.com/dlls/ 2005/prod_082905.html. • Cisco, Cisco Catalyst 6500 with Cisco IOS Software Modularity, www.cisco.com/en/US/ products/hw/switches/ps708/products_promotion0900aecd80312844.html. This page intentionally left blank
This chapter covers the following topics: • Fundamental changes to CEF for CCSR • Changes to show commands • New show ip cef commands • New show cef commands • CEF event logger • CEF consistency checker • New CEF processes C HAPTER 3
CEF Enhanced Scalability
Cisco Express Forwarding (CEF) is the most widely used forwarding mechanism on IP networks, so why change what works? There are several reasons: • To improve CEF scaling and convergence times, with the end goal being the ability to handle up to 1 million prefixes in the forwarding table (so that you can sleep well if your network has reached the 900,000-route mark, and you were worried about the next phase of network growth) • To make the interfaces between CEF, the routing table, access lists, Multiprotocol Label Switching (MPLS), and the various hardware forwarding engines more consistent and more defined • To improve memory utilization • To provide a more consistent mechanism to add new features to the switching path • To provide for CEF Management Information Base (MIB) support • To improve the performance of MPLS traffic engineering (TE) switching • To merge the IP version 4 (IPv4) CEF tables and the IPv6 CEF tables, and their associated infrastructure and control interfaces Modifications to CEF that occurred in Cisco IOS Release 12.2S largely involve internal changes that aren't obvious through output at the console, but improve the rate at which new features can be introduced, decrease the amount of work required to make CEF work with new hardware, and increase the quality of the code in the switching path. This chapter starts by discussing fundamental changes to CEF to implement CEF Enhanced Scalability (also called CSSR), including new data structures that are slightly more complex than the ones we describe in Chapter 2, “Cisco Express Forwarding.” The output from show commands has changed somewhat in this newer version of CEF; that's the next item on the menu, followed by a very useful new feature, the CEF event logger, and finally, a high-level overview of some new CEF processes. 82 Chapter 3: CEF Enhanced Scalability
Fundamental Changes to CEF for CSSR Two primary changes were made to fundamental CEF operation for CSSR: • The data structures making up the CEF tables • The switching path packets take when switched by CEF The following sections cover these two areas of fundamental change.
Data Structures Before CSSR, CEF held data using three different types of data structures: the Forwarding Information Base (FIB), the loadinfo, and the adjacency. These data structures contained some amount of overlapping information, which was copied between them when some types of CEF entry were created or modified. To save memory when storing forwarding information, CSSR added a new, fourth type of data structure, which contains a single copy of the replicated information from the older data structures. Figure 3-1 illustrates CEF before CSSR.
Figure 3-1 CEF Data Structures
fibtype loadinfo hash destination: 10.1.1.0/24 hash bucket 1 next hop hash bucket 2 path information 1 hash bucket 3 path information 2 hash bucket X path information X
information
Duplic
ated The route to 10.1.2.0/24 depends fibtype on the route to destination: 10.1.2.0/24 10.1.1.0/24 next hop: 10.1.1.1 adjacency table path information 1 forwarding info path information 2
path information X
In this case, the route to 10.1.2.0/24 uses 10.1.1.1 as its next hop. The 10.1.1.1 network is not, itself, directly connected but is reachable through some other next hop. This is called a routing recursion, a common occurrence in many networks. As Figure 3-1 shows, in this case, the path information for 10.1.1.0/24 and 10.1.2.0/24 are identical, because both routes Fundamental Changes to CEF for CSSR 83
are reachable through the same adjacency table entries. Figure 3-2 shows how path recursion arises in a network.
Figure 3-2 Route Recursion in a Network
OSPF
eBGP iBGP
10.1.1.0/24 10.1.3.0/24 A B C
10.1.2.0/24
router bgp 65000 router bgp 65001 network 10.1.2.0 mask 255.255.255.0 neighbor 10.1.3.1 remote-as 65001 neighbor 10.1.1.2 remote-as 65001 ...... router ospf 100 network 0.0.0.0 0.0.0.0 area 0
router bgp 65001 neighbor 10.1.1.1 remote-as 65000 neighbor 10.1.3.2 remote-as 65001 .... router ospf 100 network 0.0.0.0 0.0.0.0 area 0
In Figure 3-2, Routers B and C are running Open Shortest Path First (OSPF) on all their interfaces. This means that Router C has a route, learned through OSPF, for the 10.1.1.0/24 network, which connects Routers A and B. Router A is also configured to advertise 10.1.2.0/24 to Router B through Border Gateway Protocol (BGP) across an external BGP (eBGP) peering session. Router B is readvertising 10.1.2.0/24 through BGP, across an internal BGP (iBGP) peering session, to Router C. When Router B is readvertising 10.1.2.0/24 through BGP to Router C, it leaves the next hop toward the destination set to the IP address of the BGP it learned the route from—in this case, Router A—or an address on the 10.1.1.0/24 network. For Router C to forward traffic to a destination on 10.1.2.0/24, it looks up the destination and finds it is reachable through a next hop on the 10.1.1.0/24 network. This is the route recursion on the network that Figure 3-1 shows in the CEF table. Router C uses the path to 10.1.1.0/24 to reach destinations on 10.1.2.0/24. Because BGP normally does not reset the next hop on routes received from eBGP peers, route recursion is normal in large-scale networks using BGP on top of some other routing protocol, such as most service provider networks. To reduce the amount of memory CEF uses in networks of this type, CSSR separates the path information out from the remainder of the FIB information, including the destination 84 Chapter 3: CEF Enhanced Scalability
and prefix information. This allows recursive routes to share the same path information, reducing memory requirements, as shown in Figure 3-3.
Figure 3-3 CEF Data Structures After CSSR
fibtype loadinfo hash destination: 10.1.1.0/24 hash bucket 1 next hop hash bucket 2 Pointer to Path Information hash bucket 3
hash bucket X path information 1 path information 2
path information X adjacency table forwarding info The route to 10.1.2.0/24 depends on the route to 10.1.1.0/24 fibtype destination: 10.1.2.0/24 next hop: 10.1.1.1 Pointer to Path Information
As you can see from Figure 3-3, only one copy of the path information is stored in this new structure design. This reduces duplicated information in the CEF table, especially in networks with the largest number of routes, which use BGP and hence have a lot of routes reachable through recursion.
Switching Path Changes CSSR also made one change to the CEF switching path. In pre-CSSR CEF, the switching path was a monolithic unit; each feature was checked as a packet was switched through the router in software, as Figure 3-4 illustrates. Fundamental Changes to CEF for CSSR 85
Figure 3-4 CEF Switching Path
Packet No No No processed by Is A enabled? Is B enabled? Is .... enabled? device driver Yes Yes Yes
A Processing B Processing .... Processing
Packet No No No Switch Packet Is C enabled? Is .... enabled? processed by device driver Yes Yes Yes
.... Processing C Processing .... Processing
In pre-CSSR CEF, each feature in the CEF switching path, including Network Address Translation (NAT), packet filtering, and Web Cache Communications Protocol (WCCP), is checked as the packet is switched, regardless of whether the feature is enabled. If the feature is not enabled, the check is simple—just one or two lines of code—but for each feature added to the CEF switching path, the CEF code itself must be changed and maintained. CSSR changed this process, creating an insertion point in the switching path, as illustrated in Figure 3-5.
Figure 3-5 Feature Insertion in CSSR
Packet Packet processed by Switch Packet processed by device driver device driver
Packet Packet processed by Switch Packet processed by device driver device driver
With feature A inserted
Feature A 86 Chapter 3: CEF Enhanced Scalability
Normally, packets are switched along the path from the inbound device driver, through the CEF switching code, to the outbound device driver. No checks are made to determine whether additional features require processing on the packet. If a feature is required, a new node is inserted in the switching path. As the packet is switched, control is passed to this additional node in the switching chain and then back to the CEF process. Multiple nodes can be inserted either before or after the CEF switch itself; they are chained, so each one occurs before or after some other feature in the switching path. This change in the forwarding path not only eliminates the requirement to check for each feature in the path being enabled, but it also allows new features to be added, or old features to be removed, without modifying the CEF code responsible for switching packets.
Changes to show Commands A number of commands have been added or changed in CSSR. The following sections describe each change or added show command. show ip cef Example 3-1 shows the output of the show ip cef command in CEF and CSSR. The output is much shorter than in pre-CSSR CEF, because some information has been taken out of the command. The version, epoch, packet count, and byte count have all been removed, because this information is available elsewhere. The wording of the output has been shortened as well. Example 3-1 Output of the show ip cef Command
CEF-router#show ip cef 10.1.1.0 detail 10.1.1.0/24, epoch 0 via 10.1.2.1, 0 dependencies, recursive next hop 10.1.2.1, Ethernet0/0 via 10.1.2.0/24 valid glean adjacency
CSSR-router#show ip cef 10.1.1.0 detail 10.1.1.0/24, epoch 0 recursive via 10.1.2.1 attached to FastEthernet0/0 show ip cef interface Most of the information in the header of the show ip cef [interface] detail command has been removed, because it is available elsewhere; this includes the number of routes, number of leaves, the type of load sharing configured, the number of resets, and the timer configuration. The display of the CEF table entries in the second part of the output is similar Changes to show Commands 87
to the output format for show ip cef, with the same changes as noted previously. You can see these changes in Example 3-2. Example 3-2 Output of the show ip cef detail Command
CEF-router#show ip cef [interface] detail IP CEF with switching (Table Version 66), flags=0x0 46 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 1 46 leaves, 23 nodes, 30360 bytes, 85 inserts, 39 invalidations 0 load sharing elements, 0 bytes, 0 references universal per-destination load sharing algorithm, id 51DB3C24 3(0) CEF resets, 0 revisions of existing leaves Resolution Timer: Exponential (currently 1s, peak 1s) 0 in-place/0 aborted modifications refcounts: 3167 leaf, 3112 node
Table epoch: 0 (46 entries at this epoch)
Adjacency Table has 5 adjacencies 10.1.2.0/24, version 43, epoch 0, attached, connected 0 packets, 0 bytes via Ethernet0/0, 0 dependencies valid glean adjacency ....
CSSR-router#show ip cef [interface] detail IPv4 CEF is enabled and running VRF Default: 31 prefixes (31/0 fwd/non-fwd) Table id 0, version 31, 1 resets Database epoch: 0 (31 entries at this epoch)
10.1.2.0/24, epoch 0, flags attached, connected attached to FastEthernet0/0 10.1.2.1/32, epoch 0, flags adjfib NetFlow: Origin AS 0, Mask Bits 0 attached to FastEthernet0/0 10.1.1.0/24, epoch 0 recursive via 10.1.2.1 attached to FastEthernet0/0 show ip cef summary A number of fields have been removed from the show ip cef summary output as well, because the information is available in the output of other show commands. The output of show ip cef summary is identical to the header section of show ip cef [interface] detail, described in the preceding section, and the changes are identical. Example 3-3 provides a sample output for show ip cef summary. 88 Chapter 3: CEF Enhanced Scalability
Example 3-3 Output of the show ip cef summary Command
CEF-router#show ip cef summary IP CEF with switching (Table Version 66), flags=0x0 46 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 1 46 leaves, 23 nodes, 30360 bytes, 85 inserts, 39 invalidations 0 load sharing elements, 0 bytes, 0 references universal per-destination load sharing algorithm, id 51DB3C24 3(0) CEF resets, 0 revisions of existing leaves Resolution Timer: Exponential (currently 1s, peak 1s) 0 in-place/0 aborted modifications refcounts: 3167 leaf, 3112 node
Table epoch: 0 (46 entries at this epoch)
CSSR-router#show ip cef summary IPv4 CEF is enabled and running VRF Default: 31 prefixes (31/0 fwd/non-fwd) Table id 0, version 31, 1 resets Database epoch: 0 (31 entries at this epoch) show cef state capabilities The output of the show cef state command has changed dramatically in CSSR, as shown in Example 3-4. The new output shows three sections: one for the route processor (RP instance), one for IPv4 (IPv4 CEF Status), and one for IPv6 (IPv6 CEF Status). Much of the information included in the older version of the output has been removed, because it is available in the output of other show commands. Example 3-4 Output of the show ip cef state capabilities Command
CEF-router#show cef state capabilities CEF Status [RP] CEF enabled/running dCEF disabled/not running CEF switching enabled/running CEF default capabilities: Always CEF switching: no Always dCEF switching: no Default CEF switching: yes Default dCEF switching: no Drop multicast packets: no OK to punt packets: yes NVGEN CEF state: no fastsend() used: yes CEF NSF capable: no RPR+/SSO standby capable: no IPC delayed func on SSO: no FIB auto repair supported: yes LCs not running at init time: no Hardware forwarding supported: no New show ip cef Commands 89
Example 3-4 Output of the show ip cef state capabilities Command (Continued)
Hardware forwarding in use: no Load-sharing pr. packet supported: yes
CSSR-router#show cef state CEF Status: RP instance common CEF enabled IPv4 CEF Status: CEF enabled/running dCEF disabled/not running CEF switching enabled/running universal per-destination load sharing algorithm, id 5E850505 IPv6 CEF Status: CEF disabled/not running dCEF disabled/not running original per-destination load sharing algorithm
New show ip cef Commands Three new commands have been added under show ip cef, including show ip cef tree, show ip cef internal, and show ip cef switching statistics. These three commands are covered in the following sections. show ip cef tree Example 3-5 shows the output of the show ip cef tree command. Example 3-5 Output of the show ip cef tree Command
CSSR-router#show ip cef tree VRF Default tree information: MTRIE/RTREE storing IPv4 addresses 31 entries (31/0 fwd/non-fwd) Forwarding tree: Forwarding lookup routine: IPv4 mtrie 8-8-8-8 optimized 36 inserts, 5 deletes 8-8-8-8 stride pattern short mask protection enabled for <= 4 bits without process suspension 31 leaves (868 bytes), 20 nodes (20800 bytes) 23236 total bytes leaf ops: 36 inserts, 5 deletes leaf ops with short mask protection: 3 inserts, 2 deletes per-prefix length stats: lookup off, insert off, delete off refcounts: 2150 leaf, 2085 node node pools: pool[C/8 bits]: 20 allocated (0 failed), 20800 bytes Non-Forwarding tree: 29 inserts, 29 deletes 0 nodes using 0 bytes 90 Chapter 3: CEF Enhanced Scalability
The output of show ip cef tree includes the following: • A section per the virtual routing and forwarding (VRF) table. In the output in this example, there is only one VRF, the Default, so only one section is shown. If more than one VRF existed on this router, there would be a section for each VRF configured. • The type of addresses, IPv4 or IPv6, being stored in the tree. • The number of entries in the tree. • The type of tree used in this VRF, which indicates the stride. In Example 3-5, the stride is 8-8-8-8. • The number of nodes and leaves in the tree, as well as the number of tree operations, such as inserts and deletes. • Information about the amount of memory used by each tree. show ip cef internal Another new command added is show ip cef internal, as shown in Example 3-6. If you run the show ip cef internal command, you might see some entries prefixed by the tilde symbol (~). These are nonforwarding entries, which are stored in a separate tree from the forwarding entries. Nonforwarding entries result from CEF receiving information about specific destinations from multiple sources and only using one of the provided forwarding entries. Example 3-6 Output of the show ip cef internal Command
CSSR-router #show ip cef internal IPv4 CEF is enabled and running VRF Default: 31 prefixes (31/0 fwd/non-fwd) Table id 0, version 31, 1 resets Database epoch: 0 (31 entries at this epoch)
0.0.0.0/32, version 0, epoch 0, flags receive, RIB, refcount 4 sources: CEF path 638F83F0, path list 638F3350, share 1, flags receive ifnums: (none) path_list contains no resolved destination(s). HW IPv4 notified. receive output chain: receive (10) 10.1.2.0/24, version 4, epoch 0, flags attached, connected, RIB, refcount 4 sources: RIB feature space: IPRM: 0x0004800C path 638F82A0, path list 638F3230, share 1, flags attached ifnums: (none) path_list contains at least one resolved destination(s). HW IPv4 notified. attached to FastEthernet0/0, adjacency glean output chain: glean New show cef Commands 91
The show ip cef internal command essentially shows each CEF table entry, with all the available information about the entry. Useful fields include the source of the CEF table entry or the process that installed the entry, the type of entry, interfaces, and features installed on the switching path for each entry. This is a lot of information, and it probably won't be useful in many troubleshooting situations. show ip cef switching statistics CSSR adds a new command, show [ip|ipv6] switching statistics [feature]. This new show command provides information on each packet that CEF switched. For any packets punted to the process-switching path, the output of show ip cef switching will tell you why they were punted. Example 3-7 provides sample output for this new command. Example 3-7 Output of the show ip cef switching statistics Command
CSSR-Router# show ip cef switching statistics
Path Reason Drop Punt Punt2Host RP RIB Packet destined for us 0 253 0 RP RIB Total 0 253 0
RP LES Packet destined for us 0 253 0 RP LES Total 0 253 0
RP PAS Packet destined for us 0 506 0 RP PAS TTL expired 0 0 160 RP PAS Total 0 506 160
All Total 0 1012 160
New show cef Commands Another series of commands under show cef have also been added as part of CSSR. These commands contain generic information about CEF, rather than information related to IP switching of CEF. As shown in Example 3-8, the output of the show cef fib and show cef loadinfo commands provides information about the number of entries allocated in each table and memory failures encountered when allocating new entries. This information is useful when troubleshooting a router with low memory or a memory leak. Example 3-8 Output of the show cef fib and show cef loadinfo Commands
CSSR-router#show cef fib 31 allocated IPv4 entries, 0 failed allocations 0 allocated IPv6 entries, 0 failed allocations
CSSR-router#show cef loadinfo 0 allocated loadinfos, 0 failed allocations 92 Chapter 3: CEF Enhanced Scalability
In Example 3-9, the output of show cef memory shows each type of memory CEF uses and how much of it is in use. This is useful information for troubleshooting memory leaks or a router with a memory allocation problem. Example 3-9 Output of the show cef memory Command
CSSR-router#show cef memory Memory in use/allocated Count ------ADJ: DROP adjacency : 368/424 ( 86%) [1] ADJ: Discard adjacency : 368/424 ( 86%) [1] .... CEF: FIBHWIDB : 7592/8320 ( 91%) [13] CEF: FIBIDB : 2600/3328 ( 78%) [13] CEF: FIBSWSB control : 576/1024 ( 56%) [8] ....
The new command show cef table provides a summary of each CEF table configured on the router. Example 3-10 shows two CEF tables configured: a single table for IPv4 forwarding information and a single table for IPv6 forwarding information. The number of prefixes and the table version number are given. Any VRF within each table is listed in the table. Example 3-10 Output of the show cef table Command
CSSR-router#show cef table 1 active IPv4 table out of a maximum of 10000 VRF Prefixes Version Memory Flags Default 31 31 25584
1 active IPv6 table out of a maximum of 1 VRF Prefixes Version Memory Flags Default 0 0 72
Example 3-11 shows the output of show cef timers, which provides a somewhat graphical display of the timers used to maintain the CEF tables. Timers in Cisco IOS Software are related by a parent/child relationship; when the parent timer expires (wakes up), all the child timers are marked as expired as well. This allows multiple overlapping events to be controlled independently but resynchronized, or easily restarted at the same time, when certain events occur. Example 3-11 Output of the show cef timers Command
CSSR-router#show cef timers CEF background process Expiration Type | 18.196 (parent) | 18.196 FIB checkers: IPv4 scan-rib-ios scanner | 18.196 FIB checkers: IPv4 scan-ios-rib scanner | 18.196 FIB checkers: IPv6 scan-ios-rib scanner | 18.468 FIB checkers: IPv4 scan-hw-sw scanner New show cef Commands 93
Example 3-11 Output of the show cef timers Command (Continued)
| 18.468 FIB checkers: IPv4 scan-sw-hw scanner
Platform counter polling is not enabled IPv4 CEF background process Expiration Type | 0.160 (parent) | 0.160 adjacency update hwidb | 0.196 ARP throttle | 3.192 fibidb queue
Most of the timers relate to checking the CEF tables for consistency periodically or with throttling certain types of responses, such as Address Resolution Protocol (ARP) requests, so that they don't occur too often. Finally, an entire chain of new commands allow you to examine the path information that has been added, including show cef path, show cef path list, and show cef path list walk. Each one shows the same information, with increasing amounts of detail, as Example 3-12 shows. Example 3-12 Output of the show cef path Command
CSSR-router#show cef path 39 allocated IPv4 paths, 0 failed allocations 0 allocated IPv6 paths, 0 failed allocations
39 Total Paths, 1 Recursive Paths, 0 Unresolved Paths
CSSR-router#show cef path list 38 path lists (11 in shared path list hash table, 27 in special list) 0 failed allocations
hash table: [ 2] path list 638F2870, 1 path, 1 output chain, 1 lock [ 4] path list 638F3230, 1 path, 1 output chain, 1 lock [ 8] path list 638F30B0, 1 path, 1 output chain, 1 lock [ 9] path list 638F2F30, 1 path, 1 output chain, 1 lock [ 10] path list 638F2DB0, 1 path, 1 output chain, 1 lock [ 11] path list 638F2C30, 1 path, 1 output chain, 1 lock [ 12] path list 638F28D0, 1 path, 1 output chain, 1 lock [ 12] path list 638F2930, 2 paths, 1 output chain, 1 lock [ 12] path list 638F2AB0, 1 path, 1 output chain, 1 lock [ 31] path list 638F2510, 1 path, 1 output chain, 3 locks [ 46] path list 638F2690, 1 path, 1 output chain, 3 locks
CSSR-router#show cef path list walk CSSR-router##show cef path list walk 38 path lists (11 in shared path list hash table, 27 in special list) 0 failed allocations
hash table: [ 2] path list 638F2870, 1 path, 1 output chain, 1 lock continues 94 Chapter 3: CEF Enhanced Scalability
Example 3-12 Output of the show cef path Command (Continued)
[ 4] path list 638F3230, 1 path, 1 output chain, 1 lock [ 8] path list 638F30B0, 1 path, 1 output chain, 1 lock [ 9] path list 638F2F30, 1 path, 1 output chain, 1 lock [ 10] path list 638F2DB0, 1 path, 1 output chain, 1 lock [ 11] path list 638F2C30, 1 path, 1 output chain, 1 lock [ 12] path list 638F28D0, 1 path, 1 output chain, 1 lock [ 12] path list 638F2930, 2 paths, 1 output chain, 1 lock [ 12] path list 638F2AB0, 1 path, 1 output chain, 1 lock [ 31] path list 638F2510, 1 path, 1 output chain, 3 locks [ 46] path list 638F2690, 1 path, 1 output chain, 3 locks
hash table path lists: path list 638F2870, flags 21, 2 locks ifnums: (none) path_list contains no resolved destination(s). HW IPv4 notified. 1 path path 638F76D0, path list 638F2870, share 1, flags attached ifnums: (none) path_list contains no resolved destination(s). HW IPv4 notified. attached to Null0, adjacency Null0 1 output chain chain[0]: Null0 path list 638F3230, flags 29, 2 locks ifnums: (none) path_list contains at least one resolved destination(s). HW IPv4 notified. ....
CEF Event Logger Cisco IOS Software components include event loggers. An event logger is a process that runs constantly, collecting much of the same information that various types of debug output provide, but without the overhead and without having to be explicitly enabled. The event logger allows you to gather the information required to troubleshoot a problem regardless of whether you explicitly enabled debugging when the problem occurred, as long as you catch the log soon after the problem occurs. Event logs generally have a fixed size, which means that they will only hold a specific number of events before discarding the oldest event to replace it with the most recent one. The size of the IP CEF event log is set using the ip cef table event-log command; the default size is 16,000 events. The CEF event logger records events in the CEF table, such as the insertion and deletion of CEF entries, as shown in Example 3-13. Example 3-13 Output of the show ip cef event Command
CSSR-router#show ip cef event % Command accepted but obsolete, unreleased or unsupported; see documentation.
00:00:09.380: [Default] *.*.*.*/*'00 New FIB table [OK] CEF Event Logger 95
Example 3-13 Output of the show ip cef event Command (Continued)
00:00:11.112: [Default] 0.0.0.0/32'00 FIB insert [OK] 00:00:11.112: [Default] 255.255.255.255/32'00 FIB insert [OK] 00:00:11.112: [Default] 224.0.0.0/24'00 FIB insert [OK] 00:00:11.112: [Default] 224.0.0.0/4'00 FIB insert [OK] 00:00:11.332: [Default] 224.0.0.0/4'00 FIB remove (flagged) [OK] 00:00:11.332: [Default] 224.0.0.0/4'00 FIB remove (deleted) [OK] 00:00:11.332: [Default] 224.0.0.0/4'00 FIB insert [OK] 00:00:11.584: [Default] 0.0.0.0/32'00 FIB remove (flagged) [OK] 00:00:11.584: [Default] 0.0.0.0/32'00 FIB remove (deleted) [OK] 00:00:11.584: [Default] 224.0.0.0/24'00 FIB remove (flagged) [OK] 00:00:11.584: [Default] 224.0.0.0/24'00 FIB remove (deleted) [OK] 00:00:11.584: [Default] 224.0.0.0/4'00 FIB remove (flagged) [OK] 00:00:11.584: [Default] 224.0.0.0/4'00 FIB remove (deleted) [OK] 00:00:11.584: [Default] 255.255.255.255/32'00 FIB remove (flagged) [OK] 00:00:11.584: [Default] 255.255.255.255/32'00 FIB remove (deleted) [OK] 00:00:11.584: [Default] *.*.*.*/*'00 Flush FIB table (4/0ms) [OK] 00:00:11.584: [Default] 0.0.0.0/32'00 FIB insert [OK] 00:00:11.584: [Default] 255.255.255.255/32'00 FIB insert [OK] 00:00:11.584: [Default] 224.0.0.0/24'00 FIB insert [OK] 00:00:11.584: [Default] 224.0.0.0/4'00 FIB insert [OK] 00:00:11.588: [Default] 10.1.2.0/24'00 FIB insert [OK] 00:00:11.588: [Default] 10.1.2.21/32'00 FIB insert [OK] 00:00:11.588: [Default] 10.1.2.0/32'00 FIB insert [OK] 00:00:11.588: [Default] 10.1.2.255/32'00 FIB insert [OK] 00:00:11.588: [Default] 10.1.2.0/24 NBD up [OK] ....
The first line indicates that this is an unsupported command, which means you will not find much documentation about this command on Cisco.com or in any manuals. This is because this command will eventually be obsolete, replaced by commands under the monitor event-log chain, as shown in the following example. The columns in this output contain the following information: • Timestamp—This is the time at which the event occurred. • VRF—The name of the VRF in which the event occurred is contained in brackets. • Prefix and Prefix Length—The prefix that was inserted or removed, or that some other action was taken on is contained in this column. • Action—The action taken is contained in this column. • Result—The result of the action is contained in this column. The action column can contain a large number of values, including the following: • Events concerning the enabling or running of the CEF process, such as FIB enabled, FIB running, distributed FIB (dFIB) enabled, and dFIB running • Error-handling events, such as “Handling malloc failed” 96 Chapter 3: CEF Enhanced Scalability
• CEF process events, such as “Scanner process created” and “Scanner event loop enter” • IPv4 FIB table entry events, such as FIB insert and FIB delete • IPv6 FIB table entry events, such as FIB insert and FIB delete • Line card events driven by the route processor • Adjacency table events, such as interface up or down, protocol up or down, and others Another CEF event trace log is also accessible through the show monitor event-trace cef command, as shown in Example 3-14. Example 3-14 Output of the show monitor event-trace cef Command
CSSR-router#show monitor event-trace cef ? all Show all the traces in current buffer back Show trace from this far back in the past clock Show trace from a specific clock time/date events CEF Events from-boot Show trace from this many seconds after booting interface CEF Interface Events ipv4 CEF IPv4 Events ipv6 CEF IPv6 Events latest Show latest trace events since last display merged Show entries in all event traces sorted by time
CSSR-router##show monitor event-trace cef all
cef_events:
00:00:03.172: Inst unknown -> RP 00:00:03.172: SubSys fib_ios_chain init 00:00:09.372: SubSys fib init 00:00:09.376: SubSys ipv4fib init 00:00:09.384: SubSys ipv4fib_ios init 00:00:09.432: SubSys fib_ios init .... cef_interface:
00:00:09.440:
As you can see from the output shown in Example 3-13, this information is more useful for code-level debugging than CEF-level debugging. CEF Consistency Checker 97
CEF Consistency Checker The CEF consistency checker verifies that the Update Manager (discussed in the section “New CEF Processes,” later in this chapter) is maintaining the local FIB tables on each line card correctly. Two consistency checkers are included with CSSR: active and passive.
Passive Checkers Passive checkers run constantly, in the background, unless you disable them using the [no] cef table consistency-check
Active Checkers An active consistency check is initiated at the console, using the test cef enable command, followed by test cef table consistency [detail], as shown in Example 3-15. Example 3-15 Output of the test cef enable and test cef table consistency Commands
CSSR-router#test cef enable The use of TEST CEF commands will severely impact network performance and stability and should be used with extreme caution. For safety, execute the “test cef disable” command to disable this capability when it is no longer required.
CSSR-router#test cef table consistency detail full-scan-rib-ios: Checking IPv4 RIB to FIB consistency full-scan-rib-ios: FIB checked 8 prefixes, and found 0 missing. full-scan-ios-rib: Checking IPv4 FIB to RIB consistency full-scan-ios-rib: Checked 8 FIB prefixes in 1 pass, and found 0 extra. Error: Failed to run IPv6 full-scan-rib-ios checker continues 98 Chapter 3: CEF Enhanced Scalability
Example 3-15 Output of the test cef enable and test cef table consistency Commands (Continued)
Error: Failed to run IPv6 full-scan-ios-rib checker No IPv4 inconsistencies found, check took 00:00:00.004 No IPv6 inconsistencies found, check took 00:00:00.000
An active check on a table of 150,000 prefixes can take between 5 and 60 seconds.
Consistency-Checking Process The consistency-checking process contains two phases: • Building, transmitting, and comparing the FIB table information • Handling a detected inconsistency The following list and Figure 3-6 illustrate the first part of this process: 1 The CEF consistency checker on the route processor builds a consistency check message by walking the local CEF table. For each entry, a description of the entry, including a checksum, is inserted into an IPC message. 2 When the IPC message is full, it is transmitted to all the line cards. 3 The CEF consistency checker compares the information received in the consistency check IPC packet with the same entries in the local CEF tables, including comparing the checksum computed locally with the checksum computed on the route processors. 4 If any entry does not match, the line card creates a new IPC message containing the local information about this entry and transmits it to the route processor.
Figure 3-6 CEF Consistency-Checking Process RP CEF Table LC CEF Table
1 3 10.1.1.0/24 10.1.1.0/24 4 10.1.2.0/242 10.1.2.0/24 10.1.3.0/24 10.1.3.0/24 10.1.4.0/24 10.1.4.0/24 10.1.5.0/24 10.1.5.0/24 10.1.6.0/24 10.1.6.0/24 CEF Consistency Checker 99
When the route processor receives the IPC message from the line card, it reexamines the local CEF tables, looking for a mismatch. If the data is still mismatched, the route processor will build a new IPC packet with the correct information and transmit it to the line card. If, after three checks (which allows time for any pending updates to be completed), the line card and route processor tables continue to be inconsistent, the line card is marked inconsistent on the route processor. After a line card is marked inconsistent, CEF can auto-repair the CEF tables, if cef table consistency-check
CSSR-router#show cef table consistency-check Consistency checker master control: enabled
IPv4: Table consistency checker state: scan-rib-ios: disabled 0/0/0/0 queries sent/ignored/checked/iterated scan-ios-rib: disabled 0/0/0/0 queries sent/ignored/checked/iterated full-scan-rib-ios: enabled [1000 prefixes checked every 60s] 0/0/0/0 queries sent/ignored/checked/iterated full-scan-ios-rib: enabled [1000 prefixes checked every 60s] 0/0/0/0 queries sent/ignored/checked/iterated Checksum data checking disabled Inconsistency error messages are disabled Inconsistency auto-repair is enabled (10s delay, 300s holddown) Inconsistency auto-repair runs: 0 Inconsistency statistics: 0 confirmed, 0/16 recorded
IPv6: Table consistency checker state: scan-ios-rib: disabled 0/0/0/0 queries sent/ignored/checked/iterated full-scan-rib-ios: enabled [1000 prefixes checked every 60s] 0/0/0/0 queries sent/ignored/checked/iterated full-scan-ios-rib: enabled [1000 prefixes checked every 60s] 0/0/0/0 queries sent/ignored/checked/iterated Checksum data checking disabled Inconsistency error messages are disabled Inconsistency auto-repair is enabled (10s delay, 300s holddown) Inconsistency auto-repair runs: 0 Inconsistency statistics: 0 confirmed, 0/16 recorded 100 Chapter 3: CEF Enhanced Scalability
New CEF Processes Beyond the data structure and switching path changes, which impact CEF switching directly, CSSR also added the following new processes to CEF to better manage the CEF data structures and CEF operation: • FIB Manager • Adjacency Manager • Update Manager The following sections describe each manager.
FIB Manager The FIB Manager manages the prefix path lists and loadinfo maps, including managing multilevel load sharing (through recursions). The FIB Manager also dynamically allocates new memory chunks as needed and handles statistics and feature data.
Adjacency Manager The Adjacency Manager, as its name implies, manages the adjacency tables. This includes managing the interface state, enabling and disabling protocols, and maintaining of a per- interface tree.
Update Manager The Update Manager keeps track of which entries in the FIB tables need to be updated. Updates are pulled from the line cards to the route processor through the Update Manager, which allows the line cards to regulate the rate at which new FIB information is transferred. Summary 101
Summary This chapter introduced you to CEF Enhanced Scalability (CSSR). Cisco IOS Release 12.2S updated the data structures that make up the CEF tables and the switching path packets taken when switched by CEF. This chapter also described changes to the show commands and new show and show ip cef commands in CSSR. CSSR also includes an event logger and consistency checker. Information gathered by the event logger aids in troubleshooting. The CEF consistency checker verifies that the Update Manager is maintaining the local FIB tables on each line card correctly. The consistency checkers included with CSSR can be either active or passive. CSSR also added a FIB Manager, an Adjacency Manager, and an Update Manager to better manage the CEF data structures and CEF operation.
I NDEX
Numerics C
31-bit prefix network interface, FIB entries, 69 cached adjacencies, 74–75 CE (customer edge) routers, 207 CEF A accounting, enabling, 123 configuring, 77–78 active consistency checkers, 97–98 hardware implementations, 203–204 Adjacency Manager, 100 load distribution table, 162 adjacency table, 60 load sharing, troubleshooting, 176–188 auto adjacencies, 70 per-destination load sharing, 158 cached adjacencies, 74–75 example of, 163–170 discard adjacencies, 73 per-packet load sharing, 159 drop adjacencies, 72 configuring, 160–161 epoch, 77 example of, 170–173 glean adjacencies, 71 out-of-order packets, minimizing, 159 not route adjacencies, 74 performance issues, troubleshooting, null adjacencies, 73 188–189 punt adjacencies, 70–71 traffic-share allocation, 192–199 relationship to other address-mapping tables, CEF EXEC error messages, 131 60–61 CEF polarization, 210–212 uncached adjacencies, 74–75 cef table consistency-check command, 97 unresolved adjacencies, 75 CEF table, verifying, 116–125 adjacency table, verifying, 126–128 CEF/MFI (CEF/MPLS Forwarding ADJFIB FIB entries, 66 Infrastructure), 51 aliases, 19 centralized switching, 138 ARP table, verifying, 108–111 Cisco 10000 series routers articulating IP connectivity problems, 104–105 MPLS VPN, troubleshooting, 226 attached FIB entries, 63 Cisco 12000 series routers auto adjacencies, 70 MPLS VPN, troubleshooting, 221–226 Cisco 6500 router, troubleshooting MPLS VPN on router with Supervisor 2, 217–218 B on router with Supervisor 720, 218–221 Cisco 7200 with NPE-G2, troubleshooting MPLS benefits of load sharing, 153–154 VPN, 216 blackholing, 262 Cisco 7500 router, troubleshooting MPLS VPN, broadcast domains, 5–6 216–217 in routing, 7–8 Cisco Catalyst 6500 switches buffer pools centralized switching, 138 managing, 21–27 distributed switching, 138 types of, 20 SUP720, 137 bus backplane, 14–16 CEF, troubleshooting, 139–141 connectivity, verifying, 141–148 load sharing, troubleshooting, 148–149 PFC3, 138 256 Cisco GSR (Gigabit Switch Router)
Cisco GSR (Gigabit Switch Router), per-packet show ip cef interface, 86–87 load sharing, 175–176 show ip cef internal, 90–91 Cisco IOS Software, 17 show ip cef summary, 87–88 input queues, 42 show ip cef switching statistics, 91 interaction with interface processors, 28 show ip cef tree, 89–90 memory management, 17 show ip route, 112, 132, 143, 157, 181 memory pools, 17–18 show ip route summary, 250 memory regions, 18 show memory, 18, 45 packet buffers, 20–27 show mls cef adjacency entry, 145 output queues, 42 show mls statistics, 149 processes, 28 show monitor event-trace cef, 96 life cycle of, 29–32 show processes, 31, 48 priorities, 32 show processes memory, 29, 35 scheduling, 32–34 show region, 19, 46 special types of, 35 test cef enable, 97–98 watchdog timer, 34 test cef table, 97–98 collision domains, 5–6 configuring in routing, 7–8 CEF, 77–78 commands per-packet load sharing, 160–161 cef table consistency-check, 97 dCEF, 77–78 debug arp, 110 connected FIB entries, 63 external overload-signalling, 252 connectivity, 103 ip cef table loadinfo force, 252 troubleshooting mls ip cef load-sharing full, 148 adjacency table, verifying, 126–128 ping, 115–116, 191 ARP table, verifying, 108–111 remote login module, 146 CEF table, verifying, 116–125 show adjacency, 127 issues, articulating, 104–105 show adjacency detail, 182 Layer 2, 107–108 show arp, 109, 132 network topology, scoping, 105 show buffers, 20–21, 47 on SUP720, 139–141 show cef drop, 124 physical connectivity, 106–107 show cef fib, 91 routing table, verifying, 111–115 show cef interface, 160, 185–186 verifying on SUP720, 141–148 show cef loadinfo, 91 consistency checking process, 98–99 show cef memory, 92 active, 97–98 show cef not-cef-switched, 130 passive, 97 show cef path, 94 convergence, 262–263 show cef state capabilities, 88–89 CPU (central processing unit), 11 show cef table, 92 crossbar switching fabric, 13–14 show cef table consistency-check, 99 CSSR show cef timers, 92 CEF operation, changes in, 82 show interfaces, 106, 167–172, 177–190 data structures, 82, 84 show interfaces stat, 119 switching path, 84, 86 show ip cache, 120 processes added to CEF, 100 show ip cef, 86, 117, 123, 131–132, 161, 164–166, 169, 173, 178–179 show ip cef event, 94–95 inbound packets 257
for multiaccess network interface, 69 D for point-to-point network interface, 69 generic entries, 67–68 data structures, CEF modifications for CSSR, interface-specific entries, 68 82–84 longest-match lookups, 58 dCEF (Distributed CEF ), 53 mtries, 54 configuring, 77–78 performance, 56 dead processes, 35 stride patterns, 54, 57 debug arp command, 110 structure of, 57 debug commands, troubleshooting MPLS VPN, FIB Manager, 100 215 flags (FIB), 62–66 default route handler FIB entries, 66 entries learned from IGPs, 67 demand-based switching caches, 52 for 31-bit prefix network interface, 69 design considerations for switching tables, for multiaccess network interface, 69 handling overlapping prefixes, 59–60 for point-to-point network interface, 69 direct memory access, 10 generic enties, 67–68 discard adjacencies, 73 interface-specific entries, 68 distributed switching, 138 flat memory model, 29 inbound packets, 38 DRAM (dynamic random-access memory, 11 drop adjacencies, 72 dynamic buffer pools, 20 G-H
generic FIB entries, 67–68 E glean adjacencies, 71 hardware, required packet switching components eiBGP multipath feature, configuring, 233–234 bus backplane, 14–16 EIGRP (Enhanced IGRP), CEF traffic-share CPU, 11 allocation, 192–199 interface processors, 10 enabling CEF accounting, 123 memory, 11 epoch, 77 switching fabric, 11–14 Error Message Decoder, 131 hardware abstraction, 18 event logging, 94–96 hardware-based CEF, 137–138 examples hardware-based platforms, per-packet load of CEF per-destination load sharing, 163–170 sharing, 174 of CEF per-packet load sharing, 170–173 hardware-switching adjacency table, external overload-signalling command, 252 troubleshooting, 129
F I fast switching, 52, 154–158 idle state (processes), 31 FIB (Forwarding Information Base), 54 IMA (inverse multiplexing over ATM), 154 epoch, 77 inbound packets flags, 62–66 on centralized switching routers with entries learned from IGPs, 67 line cards, 37 for 31-bit prefix network interface, 69 258 inbound packets
on distributed switching platforms, 38 lookups, longest match, 58 on shared media platforms, 36 LSP (Label Switch Path)tunnels, 209 Init process, 35 input queues, 42 interface buffers, 20 M interface processors, 10 interaction with Cisco IOS Software, 28 managing buffer pools, 21–24, 26–27 interface-specific FIB entries, 68 memory, 11 ip cef table loadinfo force command, 262 constraints, 249–251 IP connectivity, troubleshooting, 103 on line cards, verifying, 251 adjacency table, verifying, 126–128 process memory, 28 ARP table, verifying, 108, 110–111 memory management (Cisco IOS Software), 17 CEF table, verifying, 116–117, 119, 121, memory pools, 17–18 123–125 memory regions, 18 issues, articulating, 104–105 packet buffers, 20–27 Layer 2 issues, 107–108 memory pools on Cisco IOS Software, 17–18 network topology, scoping, 105 buffer pools, managing, 21–27 physical connectivity, 106–107 memory regions, 18 routing table, verifying, 111–115 memory subregions, 19 ISP networks, 207 packet buffers, 20 minimizing out-of-order packets, 159 mls ip cef load-sharing full command, 148 L MPLS VPNs ISP networks, 207 label disposition, 210–212 label disposition, 210–212 label imposition, 209, 212–213 label imposition, 209, 212–213 label swapping, 210, 214 label swapping, 210, 214 Layer 2 connectivity, troubleshooting, 107–108 LFIB, 209 Layer 2 header rewrite string, 53 LSP tunnels, 209 Layer 3 switching, 8 PE-CE load sharing LFIB (Label Forwarding Information Base), 209 between P and P devices, 242 life cycle of processes, 29–32 CE multihomed to same PE, 227–233 line cards platform dependencies, 243 consistency checking process, 98–99 site multihomed to different PEs, 233–241 memory, verifying, 251 troubleshooting, 214 load distribution table, 162 debug commands, 215 load sharing on Cisco 10000 series routers, 226 benefits of, 153–154 on Cisco 12000 series routers, 221–226 CEF polarization, 200–202 on Cisco 6500 routers with Supervisor 2, in MPLS VPN environment 217–218 between P and P devices, 242 on Cisco 6500 routers with Supervisor 720, CE multihomed to same PE, 237–233 218–221 platform dependencies, 243 on Cisco 7200 router with NPE-G2, 216 site multihomed to different PEs, 233–241 on Cisco 7500 router, 216–217 troubleshooting, 148–149 show commands, 215 VRF tables, 208 ring buffers 259
mtries (multiway tries), 54 per-packet load sharing, 159 performance, 56 configuring, 160–161 stride patterns, 54, 57 on Cisco GSR platform, 175–176 multiaccess network interface, FIB entries, 69 on hardware-based platforms, 174 PFC3 (Policy Feature Card 3), 138 physical connectivity, troubleshooting, 106–107 N-O ping command, 115–116, 191 platform dependencies for MPLS VPN load new state (processes), 30 sharing, 243 not route adjacencies, 74 point-to-point network interface, FIB entries, 69 null adjacencies, 73 preventing out-of-order packets, 40–41 priorities assigned to Cisco IOS Software, 32 out-of-order packets private buffer pools, 20 minimizing, 159 process memory, 28 preventing, 40–41 process scheduling, 29 output interface, 53 process life cycle, 29–32 output queues, 42 process priorities, 32 overlapping prefixes, switching table design process switching, 51–52, 154–158 considerations, 59–60 processes overloaded links, troubleshooting CEF scheduling, 32–34 per-destination load-sharing, 176–188 special types of, 35 watchdog timer, 34 processor hogs, 34 P process-switching packets, 41–44 public buffer pools, 20 packet buffers, 20 punt adjacencies, 70–71 packet switching, 51 troubleshooting, 129–131 during receive interrupt, 39–40 inbound packets on centralized switching routers with line R cards, 37 on distributed switching platforms, 38 ready state (processes), 31 on shared media platforms, 36 receive FIB entries, 63–64 packets, transmitting, 44 receive interrupt, 38 process switching, 41–44 receive rings, 10 required components, 9 recursive FIB entries, 64–65 bus backplane, 14–16 recursive lookups, 64–65 CPU, 11 regions, aliases, 19 interface processors, 10 remote login module command, 146 memory, 11 required components for packet switching, 9 switching fabric, 11–14 bus backplane, 14–16 passive consistency checkers, 97 CPU, 11 PE (provider edge)routers, 207 interface processors, 10 per-destination load sharing, 158 memory, 11 switching fabric, 11–14 ring buffers, 10 260 routing
routing show ip cef interface command, 86–87 broadcast domains, 7–8 show ip cef internal command, 90–91 collision domains, 7–8 show ip cef summary command, 87–88 packet switching show ip cef switching statistics command, 91 bus backplane, 14–16 show ip cef tree command, 89–90 CPU, 11 show ip route command, 112, 132, 143, 157, 181 memory, 11 show ip route summary command, 260 required components, 9–10 show memory command, 18, 45 switching fabric, 11–14 show mls cef adjacency entry command, 145 routing recursions, 82–84 show mls statistics command, 149 routing tables, verifying, 111–115 show monitor event-trace cef command, 96 show processes command, 31, 48 show processes memory command, 29, 35 S show region command, 19, 46 single-path failures, troubleshooting, 190–192 scalability, memory constraints, 249–251 software-based CEF, 138 Sched process, 35 static buffer pools, 20 scheduler, 29 stride patterns, 54, 57 scheduling processes, 32–34 structure of FIB, 57 SDRAM (synchronous dynamic random-access subregions, 19 memory), 11 SUP720 (Supervisor 720), 137 shared media platforms, inbound packets, 36 CEF, troubleshooting, 139 shared memory architectures, 12–13 connectivity problems, 139–141 show adjacency command, 127 connectivity, verifying, 141–148 show adjacency detail command, 131, 192 load sharing, troubleshooting, 148–149 show arp command, 109, 132 PFC3, 138 show buffers command, 20–21, 27, 47 switching, required components, 9 show cef drop command, 124 bus backplane, 14–16 show cef fib command, 91 CPU, 11 show cef interface command, 160, 185–186 interface processors, 10 show cef loadinfo command, 91 memory, 11 show cef memory command, 92 switching fabric, 11–14 show cef not-cef-switched command, 130 switching fabric, 11 show cef path command, 94 bus backplane, 14–16 show cef state capabilities command, 88–89 crossbar switching fabric, 13–14 show cef table command, 92 shared memory architectures, 12–13 show cef table consistency-check command, 99 switching path, CEF modifications for CSSR, show cef timers command, 92 84–86 show commands, troubleshooting switching tables, handling overlapping prefixes, MPLS VPN, 225 59–60 show interfaces command, 106, 167–172, 177–180 syntax for aliases, 19 show interfaces stat command, 119 show ip cache command, 120 show ip cef command, 86, 117, 123, 131–132, 161, 164–166, 169–170, 173, 178–179 show ip cef event command, 94–95 watchdog timer 261
T U-V
test cef enable command, 97–98 uncached adjacencies, 74–75 test cef table command, 97–98 unresolved adjacencies, 75 traffic-share allocation, 192–199 Update Manager, 100 transmit interrupt, 44 consistency checkers transmit rings, 10 active, 97–98 transmitting packets during packet switching passive, 97 process, 44–45 troubleshooting CEF, 103 verifying IP connectivity adjacency table, 126–128 adjacency table, verifying, 126–128 ARP table, 108–111 ARP table, verifying, 108–111 CEF table, 116–125 CEF table, verifying, 116–125 connectivity on SUP720, 141–148 issues, articulating, 104–105 memory on line cards, 261 Layer 2 issues, 107–108 routing table, 111–115 network topology, scoping, 105 VIPs (Versatile Interface Processors), 216 physical connectivity, 106–107 VRF (virtual routing and forwarding) tables, 208 routing table, verifying, 111–115 load-sharing on SUP720, 148–149 W-X-Y-Z overloaded links, 176–188 MPLS VPN, 214 watchdog timer, 34 debug commands, 215 on Cisco 10000 series routers, 226 on Cisco 12000 series routers, 221–226 on Cisco 6500 routers with Supervisor 2, 217–218 on Cisco 6500 routers with Supervisor 720, 218–221 on Cisco 7200 routers with NPE-G2, 216 on Cisco 7500 routers, 216–217 show commands, 215 on Catalyst 6500 SUP720 platform, 139 connectivity, 139–141 per-packet load sharing, performance issues, 188–189 punt adjacencies, 129–131 single-path failures, 190–192 TTIB (Toaster Tag FIB), 217 tunnel algorithm, 213