Anatomy of Network Elements Anatomy of Core Network Elements

AnatomyAnatomy of Coreof Network Network Elements Elements from 1Gbps to 10Tbps

Josef Ungerman CSE, CCIE#6167

1. Basic Terms

2. Router Architectures

3. Switch Architectures

4. Hybrid Architectures

5. Network Processors

6. Switch Fabrics

CPU DRAM

Flash, NVRAM, CON, AUX,... Packet

interfaces Interconnect interfaces

Store & Forward Switching – using packet buffers and QoS, handles WAN interfaces (very variable interface speeds)

CPU DRAM

Flash, NVRAM, CON, AUX,... process level Packet

interrupt level

interfaces Interconnect interfaces

Process Switching – IOS Process handles the forwarding decision and other operations with the packet

CPU DRAM

Flash, NVRAM, CON, AUX,... process level process region Control Packet interrupt level I/O region Data Packet

interfaces Interconnect interfaces

Data Plane – transit packets (aka. fast path) Control Plane – packets for the router (routing, management, exceptions) • routing/control plane = routing and vital functions (OSPF, BGP, LDP, NTP, keepalives,...) • management plane = access to the router (telnet, ssh, SNMP, ...)

CPU Route DRAM Flash, NVRAM, Control Packet CON, AUX,... IOS Data Packet

NP (Network u-code Packet DRAM Processor) Data Packet

interfaces Interconnect interfaces

NP (Network Processor) – NP handles the data plane, not IOS (platform-dependent) CPU – runs IOS – handles only the control plane (platform-independent)

Slow Path – IOS on the CPU can still forward some packets that NP cannot handle (eg. exceptions, non-IP protocols routing, unsupported features) Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 7 Real-Time Packet Processing NP (Network Processor) – S/W vs. H/W router

Routing & Forwarding Engine CPU Route DRAM Flash, NVRAM, Control Packet CON, AUX,... IOS Data Packet

NP (Network NPU header BQS Packet DRAM Processor) Data Packet

interfaces Interconnect interfaces

BQS (Buffering, Queuing, Scheduling) or TM (Traffic Manager) ASIC – handles the memory access and QoS (packet body)

NPU (Network Processing Unit) – handles only packet forwarding and operations (packet header) Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 8 Summary – what is inside the router? BBB – basic building blocks

• Processor • control-plane – OS processor • data-plane – network processor

• Memory • DRAM for OS memory and packet buffers • SRAM for caches • TCAM for fast lookups

• Interconnects • bus • serial link • switch fabric

We do not care about what is visible on the router • chassis, fans, power supplies • control ports – CON, AUX, BITS, Alarms, Disks • data ports – LAN and WAN interfaces

(corollary). Good, Fast, Cheap:

Pick any two (you can’t have all three).”

RFC 1925 “The Twelve Networking Truths”

CPU (Central Processing Unit) • multi-purpose processors (CISC, RISC) • high s/w flexibility [weeks] • low performance stability [cca 1Mpps today] • usage example: access routers (ISR’s)

ASIC (Application Specific Integrated Circuit) • mono-purpose hard-wired functionality • low engineering flexibility [2 years] • high performance stability [over 200 Mpps today]

• usage example: switches (Catalysts), core routers

Input Demux

Feedback NP (Network Processor) = “something in between”

IM IM IM IM

0 4 8 12

Mem.

Column • performance + programmability

IM IM IM IM

1 5 9 13

Mem.

Column

IM IM IM

IM • moderate s/w flexibility [months]

2 6 10 14

Mem.

Column

IM IM IM IM

3 7 11 15

7 • moderate and stable performance [4Mpps – 40 Mpps+]

Mem.

Column Mux

Output • can be expensive, power-hungry, can have low code memory • usage: fast feature-rich edge and aggregation

Two basic memory technologies are in use today: • Static RAM (SRAM, SSRAM) • Dynamic RAM (DRAM, EDO DRAM, SDRAM, DDR)

SRAM DRAM High Power Low Power High Speed Low Speed [10-20ns] [40-60ns] Low Density High Density [eg. 16M per chip] [eg. 1G per chip]

• Bus • half-duplex, shared medium • for example PCI [800Mbps to 25Gbps+ today] • simple and cheap

• Serial Lane (Point-to-Point Link Set) • dedicated, unidirectional or full-duplex line • for example SPI4.2 [11.2Gbps+ today]

• Switching Fabric (cross-bar, exchange) • non-blocking, full-duplex, any-to-any • for example GSR, CRS [40Gbps to 9.6Tbps+ today]

TCAM (Ternary Content Addressable Memory)

SRAM with a comparator at each cell ROOT

1 step – very fast, but very expensive 10.0.0.0 192.0.0.0

54.0.0.0 parallel, order independent lookups 10.1.0.0 10.10.0.0 192.5.0.0 192.8.0.0 (ACL, QoS, Netflow, even FIB) 54.10.0.0 10.1.1.0 10.10.5.0 192.8.2.0

Content and Mask Address 10.1.1.1 54.10.1.0 54.10.4.0 192.8.2.0 192.8.2.128

. . . load share 192.168.100.xxx 801 punt Tree or Serial Lookup host-route 192.168.200.xxx 802 cache drop 192.168.300.xxx 803 8-8-8-8 used by generic IOS glean incomplete . . . 16-8-8 used by the C12000, 11-8-5-8 used by C10K memory vs. speed tradeoff! - could be 8-1-1-1-1-1-1-1-1-1-1-1... (low SRAM) 192.168.200.111 802 - could be used also for ACL, uRPF, accounting Query Result NEXTHOP Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 14 Router Anatomy Chapter 2

Simplex serial link set Module, card Duplex serial link set

I/O module (hardware module with I/O Active and backup backplane interfaces connection (serial duplex link set) Bus Switch Fabric (any-to-any full-duplex switching element)

Mux/demux, fabric interface (typically including a tiny buffer) Forwarding ASIC (a complex of hardware F elements and SRAM’s handling data plane)

Queuing ASIC (BQS – Buffering/Queuing/Scheduling, Q TM – Traffic Manager, etc.) NP, Network Processor (programmable hardware NP element handling data plane)

buff. Packet buffering, packet memory, QoS point

Control Plane element : CPU + DRAM + Flash + NVRAM and IOS control interfaces. CPU (Central Processing Unit) is a general

Anatomy © 2009 Ciscopurpose Systems, Inc. microAll rights reserved.-processor running the OS (Operating System) 16 Cisco 7200 (1990’s) Software Router • data plane = IOS interrupt level architecture • control plane = IOS processes

NPE-200

buff. IOS I/O controller bridge CON/AUX Flash FE

PA PCI Bus PA PA 600 Mbps PA PCI Bus PA 600 Mbps

Bus L Bus R PA 1, 4 or 6 PA slots

NPE-G2 I/O controller on-board PCI Bus 4x GE buff. IOS crypto 600 Mbps CON/AUX/Flash bridge

PA PCI Bus PA PA 600 Mbps PA PCI Bus PA 600 Mbps

Bus L Bus R PA 1, 4 or 6 PA slots

1.6G H/H PRE (active)

H/H IOS Q buff. NP SIP-600 (2-slot) 11G SPA

SPA PRE (standby) NP Q buff. Full IOS Height Linecard

8 full-height slots (ESR 10008)

Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 19 Cisco ASR1000 Split data and control plane • RP = control-plane only architecture • ESP = data-plane (QFP chip) 20Gbps, 16Mpps, C-programmable

ASR1006 SIP SPA 11.2G ESP (active) RP (active) Encryption SPA Coprocessor NP IOS buff. SIP SPA

SPA ESP (standby) RP (standby)

buff. SIP NP IOS Encryption SPA Coprocessor

SPA

1-3 SIP slots

RFC 1925 “The Twelve Networking Truths”

architecture Single-Processor • one CPU for everything

NPE-G2 I/O controller on-board PCI Bus 4x GE buff. IOS PA 600 Mbps VSA CON/AUX/Flash bridge

PA PCI Bus PA PA 600 Mbps PA PCI Bus PA 600 Mbps

Bus L Bus R PA 1, 4 or 6 PA slots

architecture Multi-Processor • distributed, parallel CPU’s

RSP (active) RSP (standby)

buff. IOS buff. IOS memd memd

VIP VIP

PA PA buff. buff. IOS IOS PA PCI Bus PA 600 Mbps VIP VIP Cy Bus Cy Bus 1Gbps 1Gbps PA PA buff. buff. IOS IOS PA Bus L Bus R PA

3, 5 or 7 VIP slots

architectural evolution Distributed Forwarding Architecture • up to 600Gbps today

RP (active) RP (standby)

IOS IOS

Switch Fabric Cards Engine 0 arb. Engine 5 buff. CSC redundant 10G 622 Q NP SPA buff. M IOS arb. IOS Q CSC NP SPA buff.

Engine 2 Engine 6 buff. SFC buff. Q 3G 40G Q F F SFC IOS Q F Q buff. IOS buff.

8, 12 or 16 Linecard/RP slots SFC

architecture Distributed Forwarding Architecture • up to 1.28Tbps today • 8 multi-chassis: 10.24Tbps today

RP (active) RP (standby)

IOS IOS IOS IOS

SIP-800 MSC FP PLIM 56G49G rx Switch Fabric Cards buff. buff. 112G98G tx tx (all 8 active) NP Q Q NP SPA IOS IOS Q NP NP Q buff. buff.

SPA midplane midplane

buff. buff. NP Q Q NP SPA IOS IOS Q NP NP Q buff. buff. SPA 4, 8 or 16 Linecard slots

evolution to 100GE Distributed Forwarding Architecture • up to 3.84Tbps single node in 2010

RP (active) RP (standby)

IOS IOS IOS IOS

SIP-900 MSC FP PLIM 140G rx Switch Fabric Cards buff. buff. 200G tx (all 8 active) NP Q Q NP SPA IOS IOS 100GE Q NP NP Q buff. buff.

SPA midplane midplane

buff. buff. NP Q Q NP SPA IOS IOS Q NP NP Q buff. buff. SPA 4, 8 or 16 Linecard slots

Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 27 Switch vs. Router • Definition 1 (old): switch is L2, router is L3 UNTIL PEOPLE INVENTED L3 SWITCH... • Definition 2: • switch is optimized for LAN – FE, GE, 10GE • router is optimized for WAN – many speeds 64Kbps – 40Gbps UNTIL ETHERNET CAME TO WAN/CORE TOO... • Definition 3: • switch is optimized for speed (fixed functionality) • router is optimized for functionality (programmable, modular)

SWITCH ROUTER

Programmable NO YES Forwarding Engine

deep, hundreds of Packet Buffers shallow, max. ~16MB MB

thousands per Queues and Shapers 4-8 per port port

Hardware Modularity 1-level 3-level+

buff. CatOS OS

F Bus buff. L2 linecard buff.

Sup (standby)

buff. Bus buff. CatOS linecard buff. OS

F L2

Bus buff. linecard buff. Cybus 1Gbps

buff. CatOS OS NFFC F F (Netflow Feature Card) Bus buff. L2 L3/4 linecard buff. IOS RSM IOS (Router/Switch Module)

SupIII (standby)

buff. Bus buff. CatOS linecard buff. OS NFFC F F (Netflow Feature Card) L2 L3/4

IOS RSM IOS Bus buff. (Router/Switch Module) linecard buff. 3x Cybus 1Gbps

buff. CatOS IOS RSFC OS IOS (Route/Switch Feature Card) NFFC F F (Netflow Feature Card) Bus buff. L2 L3/4 linecard buff.

SupIII (standby)

buff. IOS Bus buff. CatOS RSFC linecard buff. OS IOS (Route/Switch Feature Card) NFFC F F (Netflow Feature Card) L2 L3/4

Bus buff. linecard buff. 3x Cybus 1Gbps

buff. CatOS IOS MSFC OS IOS (Multilayer Switching Feature Card) PFC F (Policy Feature Card) Bus buff. L2/3/4 linecard buff.

Sup1A (standby)

buff. IOS Bus buff. CatOS MSFC linecard buff. OS IOS (Multilayer Switching Feature Card) PFC F (Policy Feature Card) L2/3/4

Bus buff. linecard buff. D-Bus 16Gbsp R-Bus 4 Gbps EOBC bus 10Mbps 3, 5, 7, or 13 Linecard/Sup slots Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 32 Real-Time Packet Processing Fast-Path Forwarding (aka. Switching)

CPU CPU

Routing Table Routing Table Process Process

flow setup FIB setup

Fast Path Fast Path

Flow Switching (aka. Fast Switching) FIB Switching (aka. CEF) • the 1st packet of a flow goes to slow-path • RIB (Routing Information Base) • the flow paths sets up the fast cache entry • FIB (Forwarding Information Base) • next packets of the flow take the fast-path • RIB is mirrored to fast-path FIB first, before any packets are switched • traffic doesn’t collide with control plane

RFC 1925 “The Twelve Networking Truths”

buff. SP RP MSFC2 IOS IOS (Multilayer Switching Feature Card) PFC2 F (Policy Feature Card) Bus-only buff. linecard buff. SFM – active (Switch Fabric Module, 256Gbps)

Sup2 (standby)

buff. Fabric-enabled buff. SP RP MSFC2 linecard buff. IOS IOS (Multilayer Switching Feature Card) PFC2 F (Policy Feature Card)

DFC 8G SFM – backup Fabric-enabled buff. (Switch Fabric Module, 256Gbps) F linecard + DFC buff. D-Bus 16Gbsp IOS R-Bus 4 Gbps EOBC bus 10Mbps 4, 6, 9, or 13 Linecard/Sup slots Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 35 Catalyst 6500 Sup720 Sup720 (active)

buff. SP RP MSFC3 IOS IOS (Multilayer Switching Feature Card) PFC3 A,B,C F (Policy Feature Card) Bus-only buff. linecard buff. Integrated SF – 720 Gbps

Sup720 (standby) buff. buff. Fabric-enabled buff. SP RP buff. F Fabric-only linecard buff. IOS IOS buff. linecard buff. IOS F

20G buff. Fabric-enabled buff. buff. Fabric-only F linecard + DFC buff. F buff. linecard D-Bus 16Gbsp buff. IOS R-Bus 4 Gbps IOS EOBC bus 10Mbps 4, 6, 9, or 13 Linecard/Sup slots Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 36 Nexus 7000 architecture

RP (active) RP (standby) RP CM RP CM IOS arb. IOSP IOS arb. IOSP

Switch Fabric Cards

buff. buff. buff. 230 buff. G 48xGE IOS F F IOS buff. buff. buff. buff.

buff. buff. buff. buff. buff. buff. buff. buff. 32xTGE IOS F F IOS buff. buff. buff. buff. buff. buff. buff. buff. 4, 8, or 16 Linecard slots X

RP (active) RP (standby)

IOS IOS IOS IOS

SIP-800 MSC FP40 PLIM Switch Fabric Cards buff. buff. (all 8 active) NP Q Q NP SPA IOS IOS Q NP NP Q buff. buff.

SPA midplane midplane

buff. buff. NP Q Q NP SPA IOS IOS Q NP NP Q buff. buff. SPA 4, 8 or 16 Linecard slots

Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 39 Cisco 7600 RSP (Route/Switch Processor) extremely compact architecture • CPU + Switch-fabric • active/standby SF RSP (active) • integrated ports + PFC

buff. SP RP IOS IOS

buff. buff. NP buff. buff. NP buff. buff. X6708 ES+40 F F buff. NP buff. buff. buff. buff. NP RSP (standby) buff. IOS IOS buff. SP RP buff. buff. NP IOS IOS buff. ES20 F F X6704 F buff. buff. NP buff. IOS IOS

Anatomy 3, ©4, 2009 6, Cisco9 or Systems, 13 Linecard/RSP Inc. All rights reserved.slots 40 Cisco 7600 switch/router architecture RSP (active) buff. buff. buff. NP buff. SPA buff. SP RP buff. X6700 buff. SIP-600 buff. IOS IOS FWSM buff. ACE buff. F IOS SPA F

buff. buff. SPA NP IOS SPA SIP-400 SIP-200 NP Bus (headers only) IOS SPA buff. buff. SPA

Anatomy 3, ©4, 2009 6, Cisco9 or Systems, 13 Linecard/RSP Inc. All rights reserved.slots 41 “It is always possible to agglutinate multiple separate problems into a single complex interdependent solution.

In most cases this is a bad idea.”

RFC 1925 “The Twelve Networking Truths”

buff. SP RP IOS IOS

F 20G buff. NP NP buff.

buff. NP NP buff. ES+ F F NP ES+ buff. NP 20G buff.

buff. NP RSP (standby) NP buff. IOS IOS buff. SP RP buff. NP NP buff. IOS IOS buff. NP NP buff. ES+ F F ES+ buff. NP F NP buff.

buff. NP NP buff. IOS IOS

RSP (active)

Transport LC – 40G IOS IOS

buff. NP NP buff. 45+45G buff. NP NP buff. IOS IOS buff. NP NP buff.

buff. NP NP buff.

Transport LC – 80G RSP (fab. active)

buff. NP buff. NP NP buff. IOS IOS buff. NP NP buff. NP buff. IOS IOS buff. NP NP buff. NP buff. buff. NP buff. NP X NP buff. 4 or 8 Linecard slots

RSP (active)

100G LC IOS IOS 200G LC 200+200G NP buff. buff. NP IOS HGE HGE IOS NP buff.

Existing LC – 80G RSP (fab. active) Existing LC – 40G

buff. NP buff. NP NP buff. IOS IOS buff. NP NP buff. NP buff. IOS IOS buff. NP NP buff. NP buff. buff. NP buff. NP X NP buff. 4, 8 or 12 Linecard slots

Pipelining L2/L3 Switch ASIC Massive SMP u-program. NP • example – Catalyst 4500 • example – Cisco CRS-1 • 250Mpps, 320Gbps • 80Mpps, 40Gbps • 185M transistors Parallel L2/L3 Switch ASIC • core/edge/eth. features • example – Catalyst 6500 • 48Mpps, 80+Gbps Massive SMP C-program. NP Pipelining ASIC with u-program. stages • example – Cisco ASR1000 • example – Cisco 12000 • 16Mpps, 20Gbps • 16Mpps, 10Gbps • 1.3B transistors • complex features – cRTP, ATM, MVPN... • fully programmable in C • >40 patents Pipelining u-program. NP • 100 engineers, 5 years • example – Cisco 10000, 7600, 7300 • sees full packet bodies! • 9Mpps, 18Gbps (firewall, IPSec, DPI, ISG,...) • complex features – PPPoE, ISG... Pipelining SMP (symmetric multiprocessing) u-program. NP • example – Cisco 7600, ASR9000 • 20Mpps, 20Gbps • very complex features – VPLS, PPPoE, ISG, Vidmon... Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 47 Non-programmable L2/L3 Switching ASIC Catalyst 45/4900 Forwarding [Mpps]: 250 Throughput [Gbps]: 320 Programmability: None Forwarding L3 fwd classify Engine police police ASIC

classify map Netflow TCAMs L2 fwd queue

TCAM map statistics SRAMs

headers only

BQS ASIC - 2K queues DRAM - SRR (1L shaping)

Linecards (8x 3Gbps fdx per LC)

RFC 1925 “The Twelve Networking Truths”

Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 49 Pipelining Parallel Programmable ASIC GSR – Engine5 (SIP-x01) Forwarding [Mpps]: 16 Throughput [Gbps]: 10 Programmability: u-code DRAM DRAM TCAM SRAM

ACL u-code u-code L3 fwd CAM L2 fwd u-code QOS u-code NF

ACL BQS ASIC RX or TX u-code u-code L3 fwd CAM L2 fwd u-code QOS u-code - 8K queues - 2L shaping headers only NF Fetch Gather ACL u-code u-code L3 fwd CAM L2 fwd u-code QOS u-code DRAM NF

ACL u-code u-code L3 fwd CAM L2 fwd u-code QOS u-code NF

SRAM

Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 50 u-Programmable SMP NPU CRS – SPP (Silicon Packet Processor) Forwarding [Mpps]: 80 Throughput [Gbps]: 40 Programmability: u-code SRAM DRAM TCAM 185M transistors Resources Interconnect & Memory

Processing Pool

188 Engines DRAM

BQS ASIC headers only - 16K queues - MDRR Distribute/Gather

Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 51 C-Programmable SMP NPU Forwarding [Mpps]: 16 Throughput [Gbps]: 20 with complete packet processing Programmability: C code ASR – QFP (Quantum Flow Processor) 1.3B transistors (NPU = cca 500M) TI 90nm, 8L 0.51W per thread

Processing Pool memory access

160 Engines (40 PPEs x 4 threads) on-chip DRAM 7 resources SRAM TCAM DRAM 0 complete packets Resources Interconnect & Memory complete packets Distribute/Gather BQS ASIC - 200K queues - 5L shaping HQF >100 engineers >5 years of development DRAM >40 patents

Ingress Egress Linecards Linecards 10G 10G Port-to-Port traffic TX RX •zero packet loss 10G TX 10G RX

10G 10G Any-to-Any traffic TX RX • Voice/Video/Data 10G • Unicast/Multicast TX 10G RX • non-zero packet loss!

Ingress RX Linecards Good: Egress Replication TX RX • Cisco CRS-1, 12000 • Cisco ASR9K, 7600 TX RX

10Gbps of multicast RX eats 10Gbps fabric bw!

Good-enough: Binary RX RX Ingress RX Ingress Replication RX Linecards • dumb switch fabric TX RX TX RX TX RX TX RX • non-Cisco

TX RX TX RX TX RX 10Gbps of multicast TX RX eats 80Gbps fabric bw!!

RX RX RX RX

Ingress Egress Good Fabric QoS Linecards fabric arbiter Linecards example 10G 10G TX RX • GSR, ASR9000, Nexus • per-destination queues 10G 10G TX RX • strict priority for Voice/TV • thousands of queues

Good-enough Fabric QoS linecard example arbiter • non-Cisco 10G 18.8G TX RX • traffic unaware arbiter (overspeed is needed) 10G 18.8G TX RX • only 2 queues, no priority • drops voice/video if loaded • loss at >66% load

Ingress Linecard Egress Linecard

Good: 40G 100G 40 Gbps per slot! 40G Network Processor 40G Network Processor non-blocking!!! Cisco CRS-1

16 slots: 16x16 matrix

Good-enough: 10G 18.8G 40 Gbps per slot! 10G 18.8G 10G 18.8G non-blocking!!! 10G 18.8G 10G Network Processor 10G Network Processor non-Cisco 12 slots: 48x48 matrix

Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 57 Marketing Math 100 ≠ 100 Ingress Linecard Egress Linecard Switch Fabric 3.84 Tbps 100GE 140G 200G Cisco CRS-1 120G 120G Network Processor Network Processor

7+1 planes

Ingress Linecard Egress Linecard Switch Fabric 1.6 Tbps load- - Fabric slowdown! balancer 50G 70G - Head-of-Line Blocking 100GE?! - additional latency 50G 70G - packet out of sequence No 100G processor available, internal load- 50G 50G non-Cisco balancing across two old Network Processor 4:1 planes Network Processor 50G processors Ingress Linecard Egress Linecard Active SF 50G 50G 100GE??!

No independent Switching Active SF 100G Network Processor 100G Network Processor Fabric available, if one CMP is removed, only 50Gbps 50G 50G non-Cisco throughput (no redundancy) X Anatomy © 2009 Cisco Systems, Inc. All rights reserved. 58 Ferrari = red car with horse

The art of engineering optimization • The Fast, the Good, and the Cheap

Router and Switch Architectures •Centralized, Distributed, Hybrid

Network Processors • Programmable? Body visibility? Tunneling?

Switch Fabrics • Is 40G really 40G? Multicast Replication? Redundancy? QoS? 100GE?