Network Processors
Douglas Comer
Computer Science Department Purdue University 250 N. University Street West Lafayette, IN 47907-2066
http://www.cs.purdue.edu/people/comer
Copyright 2003. All rights reserved. This document may not be reproduced by any means without written consent of the author. Topic And Scope
The emerging ®eld of network processors: devices that form the basis of the modern packet processing systems used in computer networks and the Internet. We will examine the motivation, architectures and technologies.
Network Processors 1 2003 List Of Topics
Generations of network systems
Emergence of network processors
Network processor architectures
Network Processors 2 2003 List Of Topics
Generations of network systems
Emergence of network processors
Network processor architectures
Network Processors 3 2003 Our Goal
Construct a network system ± Individual hardware component that processes packets ± May contain processor(s) and software ± Operates at one or more layers of the protocol stack
Accommodate ± Change of speci®cation during construction ± Changes for next-generation product
Network Processors 3 2003 Example Network Systems
Layer 2 ± Bridge, VLAN switch, DSL modem
Layer 3 ± IP router
Layer 4 ± NAT box, TCP splicer
Layer 5 ± Firewall, web load balancer, softswitch
Network Processors 4 2003 Challenges
Operate at wire speed without packet loss
Manage state information ef®ciently (e.g., TCP ¯ow)
Support traf®c in both directions
Handle reordered packets ± IP fragments especially dif®cult ± May require storing packets
Network Processors 5 2003 First Generation Network Systems
Traditional software-based router ± Packet processing in software
Use conventional (minicomputer) hardware ± Single CPU ± Single shared memory ± I/O over a bus ± Network interface cards function like other I/O devices
Network Processors 6 2003 Protocol Processing In First Generation Network Systems
NIC1 Standard CPU NIC2
framing & all other framing & address address recognition processing recognition
General-purpose processor handles most tasks
NIC handles layer 1 and basic layer 2 tasks
Suf®cient for low-speed networks
Network Processors 7 2003 Statement Of Hope (1990 version)
If there is hope, it lies in faster CPUs.
Network Processors 8 2003 How Fast Is A Fast Network?
De®nition of fast data rate keeps changing ± 1960: 10 Kbps ± 1970: 1 Mbps ± 1980: 10 Mbps ± 1990: 100 Mbps ± 2000: 1000 Mbps (1 Gbps) ± 2003: 2400 Mbps (2.4 Gbps)
Network Processors 9 2003 How Fast Is A Fast Network?
De®nition of fast data rate keeps changing ± 1960: 10 Kbps ± 1970: 1 Mbps ± 1980: 10 Mbps ± 1990: 100 Mbps ± 2000: 1000 Mbps (1 Gbps) ± 2003: 2400 Mbps (2.4 Gbps) ± Soon: 10 Gbps???
Network Processors 10 2003 The Importance Of Packet Rates
105 Kpps 77760.0
19440.0 104 Kpps 4860.0 1953.1 1214.8 103 Kpps 303.8 195.3 102 Kpps
19.5 101 Kpps
100 Kpps
10Base-T 100Base-T OC-3 OC-12 1000Base-T OC-48 OC-192 OC-768
Gray areas show rates for large packets
Network Processors 10 2003 Fundamental Question About Software-Based Network Systems
Which is growing faster? ± Processing power ± Network data rates
Network Processors 11 2003 Growth Of Technologies
10 Gbps OC-192 2.4 Gbps 10,000 ..... OC-48 ...... 622 Mbps ...... OC-12 ...... Pent.-3GHz 1,000 ...... 100 Mbps ...... Pent.-400 FDDI ...... 100 ...... Pent.-166 ...... 486-33 ...... 486-66 ...... 10 .. 10 Mbps Ethernet
1990 1992 1994 1996 1998 2000 2002
Network Processors 12 2003 Second Generation Network Systems
Concurrent with ATM development (early 1990s)
Purpose: scale to speeds faster than single CPU capacity
Decentralized architecture with multiple NICs to of¯oad CPU
Classi®cation rather than demultiplexing
High-speed interconnect (switching fabric)
General-purpose processor only handles exceptions
Network Processors 13 2003 Protocol Processing In Second Generation Network Systems
Interface1 Standard CPU Interface2
Control And Exceptions Layer 1 & 2 Class- Forward- Forward- Class- Layer 1 & 2 (framing) ification ing fast data path ing ification (framing)
NIC handles most of layers 1 - 3
Fast data path (switching fabric) avoids CPU completely
Network Processors 14 2003 Packet Classi®cation
Alternative to demultiplexing
Designed for higher speed
Considers all layers at the same time
Linear in number of ®elds
Network Processors 15 2003 Illustration Of Classi®cation
0 4 8 10 16 19 24 31 ETHERNET DEST. (0-1) ETHERNET DESTINATION (2-5) ETHERNET SOURCE (0-3) ETHERNET SOURCE (4-5) ETHERNET TYPE
VERS HLEN SERVICE IP TOTAL LENGTH IP IDENT FLAGS FRAG. OFFSET IP TTL IP TYPE IP HDR. CHECKSUM IP SOURCE ADDRESS IP DESTINATION ADDRESS
TCP SOURCE PORT TCP DESTINATION PORT TCP SEQUENCE TCP ACKNOWLEDGEMENT HLEN NOT USED CODE BITS TCP WINDOW TCP CHECKSUM TCP URGENT PTR Start Of TCP Data . . .
Highlighted ®elds are used for classi®cation of Web server traf®c
Network Processors 16 2003 Switching Fabric
Used inside a single network system
Central interconnects for I/O ports (and possibly CPU)
Can transfer unicast, multicast, and broadcast packets
Typical architecture: synchronous bus
Network Processors 17 2003 Third Generation Design
NIC contains ± ASIC hardware ± Embedded processor plus code in ROM
NIC handles ± Packet forwarding ± Traf®c policing ± Monitoring and statistics
Network Processors 18 2003 Protocol Processing In Third Generation Systems
Interface1 Standard CPU Interface2
Layer 4 Other processing Layer 4 Embedded Embedded processor Traffic Mgmt. (ASIC) Processor Layers 1 & 2 Layers 1 & 2 Layer 3 & class. fast data path Layer 3 & class. ASIC ASIC
Special-purpose ASICs handle lower layer functions
Embedded (RISC) processor handles layer 4
CPU only handles low-demand processing
Network Processors 19 2003 Statement Of Hope (1995 version)
If there is hope, it lies in ASIC designers.
Network Processors 20 2003 List Of Topics
Generations of network systems
Emergence of network processors
Network processor architectures
Network Processors 21 2003 Problems With Third Generation Systems
High cost
Long time to market
Dif®cult to simulate/test
Require in-house expertise (ASIC designers)
Expensive and time-consuming to change ± 18-20 months for silicon respin
Little reuse across products
Limited reuse across versions
Network Processors 21 2003 Statement Of Hope (1999 version)
???
If there is hope, it lies in ASIC designers.
Network Processors 22 2003 A Fourth Generation
Goal: combine best features of ®rst generation and third generation systems ± Flexibility of programmable processor ± High speed of ASICs
Technology called network processors
Network Processors 23 2003 De®nition Of A Network Processor
A network processor is a special-purpose, programmable hardware device that combines the low cost and flexibility of a RISC processor with the speed and scalability of custom silicon (i.e., ASIC chips). Network processors are building blocks used to construct network systems.
Network Processors 24 2003 Network Processors: Potential Advantages
Relatively low cost
Straightforward hardware interface
Facilities to access ± Memory ± Network interface devices
Programmable
Ability to scale to higher ± Data rates ± Packet rates
Network Processors 25 2003 Network Processors: Potential Advantages
Relatively low cost
Straightforward hardware interface
Facilities to access ± Memory ± Network interface devices
Programmable
Ability to scale to higher ± Data rates ± Packet rates
Network Processors 26 2003 Statement Of Hope (2003 version)
programmers!
If there is hope, it lies in ASIC designers.
Network Processors 26 2003 Costs And Bene®ts Of Network Processors
? ASIC Designs
? Network Increasing Processor Performance Designs
Software On Conventional Processor
Increasing cost
Currently more expensive than conventional processors
Currently slower than ASICs
Future trends still unclear
Network Processors 27 2003 List Of Topics
Generations of network systems
Emergence of network processors
Network processor architectures
Network Processors 28 2003 Network Processor Design
What is known ± Must partition packet processing into separate functions ± To achieve highest speed, must handle each function with separate hardware
Still being researched ± Which functions to choose ± Which hardware building blocks to use ± How building blocks should be interconnected
Network Processors 28 2003 The Range Of Architecture Styles
Embedded processor plus ®xed coprocessors
Embedded processor plus programmable I/O processors
Parallel (number of processors scales to handle load)
Pipeline processors
Network Processors 29 2003 Embedded Processor Architecture
f(); g(); h()
Single processor ± Handles all functions ± Passes packet on
Known as run-to-completion
Network Processors 30 2003 Parallel Architecture
coordination f(); g(); h() mechanism
f(); g(); h() . .
f(); g(); h()
Each processor handles 1/N of total load
Network Processors 31 2003 Pipeline Architecture
f () g () h ()
Each processor handles one function
Packet moves through ``pipeline''
Network Processors 32 2003 Clock Rates
Embedded processor runs at > wire speed
Parallel processor runs at < wire speed
Pipeline processor runs at wire speed
Network Processors 33 2003 Commercial Network Processors
Emerge in late 1990s
Become popular in early 2000s
Exceed thirty vendors by 2003
Network Processors 34 2003 Examples
Chosen to ± Illustrate concepts ± Show broad categories ± Expose the variety
Not necessarily ``best''
Not meant as an endorsement of speci®c vendors
Show a snapshot as of 2003
Network Processors 35 2003 Multi-Chip Pipeline (Agere)
Brand name PayloadPlus
Three individual chips ± Fast Pattern Processor (FPP) for classi®cation ± Routing Switch Processor (RSP) for forwarding ± Agere System Interface (ASI) for traf®c management and exceptions
Note: second generation will use a single chip
Network Processors 36 2003 Multi-Chip Pipeline (Agere) (continued)
Fast Routing Pattern Switch Processor Processor ( FPP ) ( RSP )
packets packets sent arrive to fabric configuration bus
Agere System Interface ( ASI )
Network Processors 37 2003 Architecture Of Agere's FPP Chip
input data buffer output framer controller interface
block buffers and context memory
program pattern checksum / memory engine CRC engine
control queue ALU memory engine
conf. bus functional bus interface interface
Network Processors 38 2003 Augmented RISC (Alchemy)
Based on MIPS-32 CPU ± Five-stage pipeline
Augmented for packet processing ± Instructions (e.g. multiply-and-accumulate) ± Memory cache ± I/O interfaces
Network Processors 39 2003 Alchemy Architecture
to SDRAM SDRAM controller fast IrDA
EJTAG instruct. cache MIPS-32 DMA controller embed. proc. bus unit SRAM Ethernet MAC bus data cache MAC LCD controller
SRAM controller USB-Host contr.
USB-Device contr. RTC (2) interrupt controller power management GPIO
SSI (2) I2S
AC '97 controller Serial line UART (2)
Network Processors 40 2003 Parallel Embedded Processors Plus Coprocessors (AMCC)
One to six nP core processors
Various engines ± Packet metering ± Packet transform ± Packet policy
Network Processors 41 2003 AMCC Architecture
external search external memory host interface interface interface
policy metering engine engine
memory access unit
six nP cores onboard memory
input packet transform engine output
control iface debug port inter mod. test iface
Network Processors 42 2003 Pipeline Of Homogeneous Processors (Cisco)
Parallel eXpress Forwarding (PXF)
Arranged in parallel pipelines
Packet ¯ows through one pipeline
Each processor in pipeline dedicated to one task
Network Processors 43 2003 Cisco Architecture
input
MAC classify
Accounting & ICMP
FIB & Netflow
MPLS classify
Access Control
CAR
MLPPP
WRED
output
Network Processors 44 2003 Con®gurable Instruction Set Processors (Cognigine)
Up to sixteen parallel processors
Connected in a pipeline
Processor called Reconfigurable Communication Unit (RCU)
Interconnected by Routing Switch Fabric (RSF)
Instruction set determined by loadable dictionary
Network Processors 45 2003 Cognigine Architecture
routing switch fabric connector
packet buffers pointer diction- data instr. file ary memory cache registers & scratch memory
source source source source route route route route dict. pipe- addr. de- line calc. code ctl. execut. execut. execut. execut. unit unit unit unit
Network Processors 46 2003 Pipeline Of Parallel Heterogeneous Processors (EZchip)
Four processor types
Each type optimized for speci®c task
Network Processors 47 2003 EZchip Architecture
TOPparse TOPsearch TOPresolve TOPmodify
...... memory memory memory memory
Network Processors 48 2003 EZchip Processor Types
Processor Type Optimized For TOPparse Header field extraction and classification TOPsearch Table lookup TOPresolve Queue management and forwarding TOPmodify Packet header and content modification
Network Processors 49 2003 Extensive And Diverse Processors (IBM)
Multiple processor types
Extensive use of parallelism
Separate ingress and egress processing paths
Multiple onboard data stores
Model is NP4GS3
Network Processors 50 2003 IBM NP4GS3 Architecture
to switching PCI external DRAM from switching fabric bus and SRAM fabric
ingress ingress internal egress egress data switch SRAM switch data store interface interface store
Embedded Processor Complex SRAM (EPC) traffic for manag. ingress and data sched.
ingress egress physical physical MAC MAC multiplexor multiplexor
packets from packets to egress physical devices physical devices data store
Network Processors 51 2003 IBM's Embedded Processor Complex
to onboard memory to external memory
H0 H1 H2 H3 H4 S D0 D1 D2 D3 D4 D5 D6
control memory arbiter
ingress egress queue completion unit queue
interrupts .... debug & inter. hardware regs. inter. bus control . exceptions . . . . . embed. . PCI . PowerPC . bus programmable . protocol processors . . ingress ingress (16 picoengines) . data data . . store iface egress . egress data . data . iface . store . . . . . internal instr. memory classifier assist bus arbiter bus
ingress egress data frame dispatch data store store
Network Processors 52 2003 Flexible RISC Plus Coprocessors (Motorola C-Port)
Onboard processors can be ± Dedicated ± Parallel clusters ± Pipeline
Network Processors 53 2003 C-Port Architecture
switching fabric
network network network processor processor . . . processor 1 2 N
physical physical physical interface 1 interface 2 interface N
Network Processors 54 2003 C-Port Con®gured As Parallel Clusters
switching SRAM fabric SRAM PCI bus serial PROM DRAM
queue table buffer mgmt. fabric lookup pci ser. prom mgmt. unit proc. unit unit Exec. Processor
multiple onboard buses
clusters
CP-0 CP-1 CP-2 CP-3 . . . CP-12 CP-13 CP-14 CP-15
connections to physical interfaces
Network Processors 55 2003 Internal Structure Of A C-Port Channel Processor
To external DRAM
memory bus
RISC Processor
extract merge space space
Serial Data Serial Data Processor Processor (in) (out)
packets arrive packets leave
Actually a processor complex
Network Processors 56 2003 General Purpose Processor (Intel)
First generation model is IXP1200 ± One embedded processor (StrongARM) ± Six programmable packet processors (microengines) ± Four external bus interconnections ± Other functional units
Network Processors 57 2003 IXP1200 External Connections
optional host connection PCI bus
SRAM bus
serial SRAM line
FLASH
IXP1200 Memory chip Mapped I/O
SDRAM
SDRAM bus
High-speed I/O bus IX bus
Network Processors 58 2003 IXP1200 Internal Architecture
optional host connection PCI bus
SRAM bus IXP1200 chip
serial SRAM SRAM PCI access Embedded access RISC line processor (StrongARM)
FLASH Microengine 1 multiple, scratch independent Memory memory internal Microengine 2 Mapped buses I/O Microengine 3
Microengine 4
Microengine 5 SDRAM SDRAM IX access access Microengine 6
SDRAM bus
High-speed I/O bus IX bus
Network Processors 59 2003 Example Of Complexity: SRAM Access Unit
SRAM access unit
service priority clock SRAM pin arbitration inter- signals face AMBA SRAM address bus AMBA memory inter- data & FIFO face from data StrongARM
buffer Flash (Boot AMBA addr. ROM) command queues decoder addr & addr. generator Memory microengine addr. & command queues Mapped Microengine I/O commands
microengine data
Network Processors 60 2003 A Second Generation Of Network Processors
Announced by several vendors
Due to emerge in late 2003
Trends ± Higher data rates (10 Gbps) ± More functionality
Network Processors 61 2003 A Second Generation Of Network Processors (continued)
Examples ± Agere Systems APP 550 and APP 750 ± IBM enhanced PowerPC ± Intel IXP 2400 and IXP 2800
Network Processors 62 2003 Open Questions
Will network bandwidth or CPU speed increase fastest?
Can network processors become as inexpensive as general- purpose processors?
Can network processors become as fast as ASICs?
What hardware architecture should be used for network processors?
What software architecture should be used for network processors?
Network Processors 63 2003 Summary
Goal of network processors ± Flexibility of software-based systems ± High speed of ASICs
Many architectural experiments
No consensus on ± How to provide functionality ± How to attain highest speed ± How to architect a switching fabric ± Which programming language(s) to use
Many open problems
Network Processors 64 2003 Questions? STOP