Modern Service Provider Routers

MODERN SERVICE PROVIDER ROUTERS

Guy C Fedorkow Distinguished Engineer, Juniper Networks April 2013

THE INTERNET EXPLOSION

# Web Sites 130EB/yr

Internet Capacity 162M # Connected Devices 1B

Total Digitized Information 420EB # Google Searches/Month 100M 31B/mo

12EB/yr 40M

110EB 4PB/yr 60PB/yr 9.5M 160M 25M 33K 1 1.7M 2.7B/mo

1988 1993 1998 2003 2008

Exponential growth, no matter how you measure it!

The clearest indication of value delivered to end-users

From RFC-1812 “An IP router can be distinguished from other sorts of packet switching devices in that a router examines the IP protocol header as part of the switching process. It generally removes the Link Layer header a message was received with, modifies the IP header, and replaces the Link Layer header for retransmission.”

Routers: § Strip the L2 Header § Decrement the TTL § Look up the Next Hop § Add New L2 header § Transmit the Packet Router

And also Routing topology, Routing policy, Access Control, Filters, Rate ExternalInterfaces

Policing, Shaping, Traffic ExternalInterfaces Prioritization, Buffering, Tunnels, MPLS, v4/v6 interworking, NAT, 3 Subscriber Authentication, etc, etc. Copyright © 2010 Juniper Networks, Inc. NATIONAL ISP NETWORK ARCHITECTURE

Service Provider Domain

CPE (Customer Edge Core GigE, GPON, Premise Routers) Routers Routers 10GigE DSL, SDH, and PDH Access 100GigE Large Networks show links Evolution in Action! 4 Copyright © 2010 Juniper Networks, Inc. WHAT’S A SERVICE PROVIDER ROUTER

Core Routers § Highest Bandwidth Common Features Capacity – Nx100GigE § Common Element § Emphasis on Route Redundancy Scaling – Millions of § Hot-Swap & ISSU Routes § BGP Control § Modest Forwarding Plane Feature Set § Demanding Environmental Edge Routers (e.g. 55C) § Highest Interface Scaling § Redundant DC(!) – Hundreds of GigE and Power 10GigE interfaces § Complex Forwarding Feature Set § Emphasis on Subscriber Scaling 5 Copyright © 2010 Juniper Networks, Inc. SAMPLE EDGE ROUTER PORTFOLIO 80Tbps 34Tbps

40Tbps Edge Routers come 17Tbps in small, medium, large and larger… 8.8Tbps 4.8Tbps 5.3Tbps 2.8Tbps 2.6Tbps 1.6Tbps 1.4Tbps

80Gbps

MX 80 MX 240 MX 480 MX 960 MX 2010 MX 2020

10GE Ports* 48 136 256 260 520

High-Scale Routers comprise several key elements:

• Control Plane § Responsible for managing routing tables, Control Plane authenticating subscribers, configuring interfaces Fabric •Packet Forwarding Engine(s) (PFE) PFE PFE § Responsible for forwarding each packet (i.e. address lookup, queues, access lists, etc) • Fabric § Responsible for moving packets from one line card to another inside the router

Route Engine Zero Line Card N

General Purpose Ethernet PFE CPU Switch Backplane Backplane

Embedded Control Processor

General Purpose Ethernet Switch CPU PFE

Route Engine One

Fabric Planes

Most high-scale routers are Line Cards Fabric Based

• Multiple Line Cards, each containing PFEs

• A chassis-wide Interconnect Fabric transfers traffic from ingress to egress line cards PFE complex

Lookup chip Queuing chip

PIC PIC PIC PIC WAN WAN WAN WAN FPCPorts FPCPorts FPCPorts FPCPorts TL TL TL TL TL TL TL TL Each line card connects to

TQPFET Q TQPFET Q TQPFET Q TQPFET Q all Fabric cards.

CCG CThereB are a lot of wires on the backplane! RE B B B B B B B B B I I I I I I I I I S S S S S S S S S Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric Fabric

CB CCG

TQ TQ TQ TQ TQ TQ TQ TQ PFE PFE PFE PFE TL TL TL TL TL TL TL TL

FPCWAN FPCWAN FPCWAN FPCWAN PortsPIC PortsPIC PortsPIC PortsPIC

19” W Cra Panel

Upper 10 I/O Slots

45 RU H 8 Fabric’s

2 RE’s 920mm D

Lower 10 I/O Slots

What to do when one chassis isn’t big enough? …So if that’s not enough, go to multiple XM Line XM Card Fab Line fabric stages using WAN Card WAN Inter- XM Line XM Inter- Benes or Butterfly faces Card Fab Line faces Card networks … … XM Line XM Card Fab Line Card Fab Cards Fabric Router Chassis Router Chassis #0 Router Chassis #0 Chassis

XM Line XM Card Fab Fab Fab Line WAN Card WAN Single Stage Inter- XM Line XM Inter- faces Card Line fabric only Fab Fab Fab faces Card scales so far… … … XM Line XM Card Fab Fab Fab Line Card XM Line XM Issues to Solve: Card Fab Fab Line • Expensive optical cables Fab Card Router Chassis #N Router Chassis #N • SW Scaling & Distributed Control • Proto costs(!) Wrap this mesh around a cylinder so these points meet… 13 Copyright © 2010 Juniper Networks, Inc. PACKET FORWARDING ENGINE (PFE)

PFE’s do the work to move packets from Ingress to Egress

Key Functions: § L2 & L3 Analysis & Features Figure out who’s packet it is, what should happen to it, and where it should go. § Packet Buffering Store the Packet in DRAM until there’s room to transmit it § Queuing & Scheduling Decide which packets should go in what order to achieve fairness and real-time delivery guarantees.

PFEs may be micro-programmable, table-driven or hard-coded [It’s the old Cost/Performance/Flexibility Tradeoff Matrix…]

SRAM RLDRAM SRAM RLDRAM QX LU Queues RLDRAM Optional

Classification HQoS RLDRAM Policing RLDRAM Re-write Link RAM

10GE

IX DDR3 DDR3 TCAM Optional

Pre-classification 10GE Fabric queuing MQ Fabric Port based 10GE queuing IX Optional

10GE DDR3 DDR3 DDR3 DDR3 RLDRAM

Pre-classification DDR3 DDR3 DDR3 DDR3 Link RAM Packet data

BA Classiﬁcaon (LU) MF Classiﬁcaon (LU)

Policing MQ : Fabric Interface Forwarding Class & Can overwrite Fabric Queuing Packet Loss Priority FC & PLP

Switch Fabric

LU: QX Scheduling: MQ: Fabric Interface Egress Filters/Policers HQoS Queuing No Schedulers Rewrite Headers

Buffer INTERFACE-SET PIR PIR VLAN VLAN

CIR CIR Traffic INTERFACE-SET PIR PIR VLAN VLAN

PIR Port CIR CIR INTERFACE-SET

Shaper EgressInterface PIR

VLAN VLAN PIR

CIR CIR PQ-DWRR Queues Scheduler

§ Power Management § Keeping power dissipation down § Getting the heat out § Signal Integrity § Crosstalk from a zillion wires on the backplane § Features § QoS § Multicast § Tunnels § Link Aggregation & ECMP Load Balancing -- and -- “Feature Velocity” § Scaling the Control Plane -- FIB and Subscribers § Service Delivery (NAT&Firewall, Content Delivery, DPI, & who knows what else)

General-Purpose Microprocessor (G) Services Engine (S) Edge Engine Core (E) Fabric Engine (F) (C)

1000,000 X over 20 years (2x /year) 224

222 Post-ASIC era: 2.2x /year TX T1600

220 T640 218 Pre-ASIC era: 1.6x /year M160 216 M40 214

212 Interface CAGR: 1.7x /year 210

Megabitsper second 2

22 20 ‘88 ‘89 ‘90 ‘91 ‘92 ‘93 ‘94 ‘95 ‘96 ‘97 ‘98 ‘99 ‘00 ‘01 ‘02 ‘03 ‘04 ‘05 ‘06 ‘07 ‘08

Packet buffering • Need high throughput, high density • Long bursts ok • SDRAM or RLDRAM (Reduced Latency DRAM)

Queuing/Link memory • Need high throughput, low latency • Shorter bursts • SRAM, RLDRAM, or SDRAM

Control memory • Need high throughput, low latency • Even smaller access quantum • SRAM, TCAM, or RLDRAM

– Chips get very expensive once Chip yield & they cross a certain die size PCB layers $ dominate – Economics of silicon is all about 1 chip fabrication yield Pins, packaging, power, PCB area § Goals dominate 2 chips – Balance size of each chip within packet forwarding engine – Minimize pin-count on each chip – Minimize overall component cost X depends on technology – Flexibility of support different configs with the same chipset Total # of transistors

Trio/NISP 65nm 4 Chips 1.2Bn Transistors IP3 I-Chip 180nm (90nm) 90nm 604 Gbps IO IP1, 2 10 Chips 1 Chip RLDRAM/ 250nm 446m Trans 160m Transistors DDR3 SDRAM 4 Chips 412 Gbps IO 219 Gbps IO 18m Trans RLDRAM/ 47 Gbps IO SRAM/ DDR2 SDRAM SRAM/ RDRAM SDRAM (RLDRAM)

1998 2002 2006 2009

For further information on router functionality: Juniper MX Series, Douglas Richard Hanks Jr and Harry Reynolds, published by O’Reilly MICRO ARCHITECTURE

Xchip

Take each subsystem, divide into blocks, divide each block into sub- blocks, design down to the basic X_in X_out logic elements

Document both functionality and X_in_a X_in_b architecture § Rigorous peer reviews of all documents X_in_b_cntl X_in_b_dp

Control logic Datapath

Translate micro architecture for all blocks to “Register Transfer Level” code.

always @ (sel or a or b) a begin out if (sel == 1) out = a; b else out = b; sel end OR assign out = sel ? a : b;

§ A large chip will have hundreds of thousands of lines of RTL code § Must always keep in mind physical placement and timing during the micro architecture phase – You pay now or you pay more later

Synthesis is the exercise of mapping RTL to GATES in the technology of choice INPUT – RTL code – Specification of clocks and cycle-time (frequency) – Input and output constraints for module being synthesized – Wire-load models as basis to model interconnect effects on gates – Recent trends: physical synthesis

Goal: First-time-right silicon § Avoid expensive ASIC respins § Simulations are far easier to debug than real chips

Recipe: At least as many verification engineers as design engineers per chip TOOLS

Performed at multiple levels Test-bench tool § Block level SystemVerilog § Chip level C/C++, Verilog Coverage tools § Sub-system level Equivalency checkers § System level Simulators Waveform viewers § Software/hardware co-simulation

Power and clock planning

Perform high-level floor-planning

Place I/O, SRAMs, & Register Arrays

Random logic placements

Perform congestion analysis

Wire up all the logic and IOs

Run timing with physical placement

Many iterations of all of the above

1) Memory placement 2) Logic placement & clocks 3) M1 routing 4) M2 routing 5) M3 routing 6) M4 routing 7) M5 routing 8) M6 routing 9) M2/M4/M6 routing 10) M1/M3/M5 routing

Criteria for ASIC Tapeout § All functionality complete § All verification complete § Performance simulations meet goals § Chip is error free from a testability perspective § Chip meets timing under all process, temperature and voltage conditions § Design and verification database is archived

After the ASIC is taped out § Masks are generated for photolithography § ASICs are then built layer-by-layer on a silicon substrate wafer

Once the ASIC wafer is complete § Each die is tested in wafer test § Only good die are laser cut for packaging

Once cut die are available § They are put in a package § The packaged devices are then tested again

Tested packaged parts are put on system boards § Test with other hardware and software