Comparison of High-Speed Interconnects: Ethernet, PCI Express® and Rapidio® Technology

July 2009 Comparison of High-Speed Interconnects: Ethernet, PCI Express® and RapidIO® Technology

Greg Shippen System Architect, Network Systems Division Networking & Multimedia Group TM Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. Agenda

►Interconnect Trends

►Technical Overview

►Comparison

►Summary and Conclusion

Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 2 QorIQ™ P4 Series P4080 Block Diagram QorIQ™ P4080 MULTICORE 1024 KB 64-bit PROCESSOR Power Architecture™ Frontside DDR-2 / 3 128 KB e500-mc Core L3 Cache Memory Controller Backside 1024 KB 64-bit L2 Cache 32 KB 32 KB DDR-2 / 3 D-Cache I-Cache Frontside L3 Cache Memory Controller

eOpenPIC CoreNet™ PreBoot Loader Coherency Fabric Peripheral PAMU Security Monitor PAMU PAMU PAMU PAMU Access Mgmt Unit Internal BootROM

Power Mgmt Frame Manager Frame Manager Real Time Debug RapidIO® Watchpoint SD/MMC Security Queue eLBC Message 2x DMA Cross 4.0 Mgr. Parse, Classify, Parse, Classify, Trigger SPI Distribute Distribute Unit (RMU)

2x DUART Pattern Buffer Buffer Perf CoreNet Test Monitor Trace Match Buffer 4x I 2C Port/ Engine Mgr. 1GE 1GE 1GE 1GE PCIe PCIe PCIe sRIO 10GE SAP 2.0 10GE SRIO 2x USB 2.0/ULPI 1GE 1GE 1GE 1GE Aurora Clocks/Reset GPIO 18-Lane 5 GHz SerDes CCSR

I want more CPU cycles Quit spending them moving data

Support living standards I don’t want to change next time with a living ecosystem

I want to meet my technical Implement QoS, scalable BW, requirements multicore ready, high availability…

I don’t want to rewrite my Use common usage models software and software APIs

I want it cheap Use multi-vendor standards

Bandwidth CPU CPU CPU CPU

GB/s Accel I/O I/O

Cost Connected

$$$ NRE Devices Protocols

SP $$$ OPEX X I4 CSI TDM .2

P C I E x P pr /I es $$$ CAPEX P s ® TC ATM

Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 5 Interconnect Trends ► 2nd Generation Point-to-Point • Packet switched Device Device • PHY: SERDES differential st ► 1 Generation Point-to-Point • Lowest pin count Switch • Packet switched Fabric Device • PHY: Source-sync differential • Lower pin count Device Device Device Device ≥ 10 GHz Example: HT/P-RapidIO® ≤ 3 GHz Device Ex: PCIe, ► Hierarchical Bus S-RapidIO, Device Device • Bridged Hierarchy SATA, • Broadcast Device SAS Bridge Device Device • PHY: Single-ended Example: PCI/PCI-X/SCSI ≤ 133MHz ce ► Shared Bus n • Single segment Device Device Device Device Device orma • Broadcast f er • PHY: Single-ended Device Device Device Device Device P • Highest pin count Example: VME ≤ 66MHz

►Chip-to-chip ►Board-to-Device ►Board-to-board ►Chassis-to-chassis Chassis-to-chassis

Device

Chip-to-chip Board-to-Board

Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 8 Ethernet Overview ► WAN scale interconnect • Box-to-box, board-to-board, backplane • Connect thousands to millions of endpoints • Physical layer defined for LAN-scale interconnection Closet to computer Backplane Ethernet Endpoint • Optical, twisted pair and backplane copper media CPU ► Target market • GigE WAN to workstations, PCs and laptops DRAM • 10GE now used in aggregation settings High performance switches, routers and LAN backbones MAC/PHY ► Specification history • First spec (10Mbps) ~1975 by Xerox Port 2 • 100Mbps spec in 1995 • 1Gbps spec in 1998 End Port Switch/ Port End • 10Gbps spec in 2002 1 3 10G Copper (10GBase-T) in 2006 point Router point • Recent relevant additions Backplane Ethernet (802.3ap-2007) Port Data Center Bridging (DCB) 0 ► Gigabit Ethernet ubiquitous now • 10G Copper PHYs shipping End ► Extensible layered specification point ► Point-to-point packetized architecture • High header overhead • Variable packet size • 46-1500 byte packet L2 PDU • Up to 9000 byte jumbo frames

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Preamble 4

Preamble SFD 8

Destination Address 12

Destination Address Source Address 16

Source Address 20

Type/Length Packet PDU 24

Packet PDU 278

FCS 282

Inter-Frame Gap 294 Bytes Layer 2 Packet Type: 1500 Byte Max Packet PDU Total = 294 Bytes Interframe overhead L2 header/trailer Payload (256 Byte PDU)

Preamble/SFD L2 Header IP Header 8 Bytes 14 Bytes 20 Bytes

TCP Header User PDU FCS IFG 334 Bytes 20 Bytes 256 Bytes 4 Bytes 12 Bytes

TCP/IP Packet Type: 1460 Byte Max User PDU

Total = 334 Bytes (256 Byte User PDU)

► Chassis-scale interconnect • Chip-to-chip, Board-to-board CPU • Required legacy PCI compatibility • Physical layer defined for board + connector • Copper-on-board and cable media ► Successor to PCI 2.3/PCI-X 2.0 Host/ • Fully SW/firmware backward compatible to PCI End Port Port End Root ► Target market point 0 2 point • PC and Servers space Complex • Embedded where suitable Port ► Specification history 1 • Rev 1.0 (Gen1) completed in 2002 Upstream Switch Port External cable spec released Feb 2007 • Rev 2.0 (Gen2) completed in 2006 Switch P2P • Rev 3.0 (Gen3) expected “late 2009” 8 GTransfers/s • Recent relevant additions P2P P2P P2P Multiroot/single-root IO Virtualization Cable Spec Down Down Down ► stream stream stream PCIe Gen2 now widely deployed Port Port Port • First Gen2 Intel Silicon (X38 chipset) Sep 2007 ► Extensible layered specification ► Point-to-point packetized architecture End End End • Relatively low overhead point point point • Variable size packets • 128-4096 byte PDU

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Rsvd TLP Sequence Number R FMT Type R TC Rsvd 4

T E D P Attr R Length Requester ID 8 Last DW First DW Tag Address[31:16] 12 BE BE Address[15:2] R Packet PDU 16

Packet PDU 20

Packet PDU Optional TLP Digest (ECRC) 272

Optional TLP Digest (ECRC) Cont LCRC 276

LCRC Cont Next Packet/DLLP 280 Bytes

Memory Write: 4096 Byte Max Packet PDU Total = 278 Bytes Link Layer Transaction Layer Payload (256 Byte PDU)

► Chassis scale interconnect • Chip-to-chip, Board-to-board, backplane • Initially a processor interconnect as Motorola/Mercury collaboration • Physical layer defined for board + connectors • Copper-on-board and cable media ► Target market Host • Embedded systems Wireless infrastructure, media, networking, compute & defense • CPU I/O, Line-card aggregation, backplane Port • Extensive dataplane features 2 QoS, VCs, datagrams, encapsulation ► Specification History End Port Port End • Rev 1.0 completed in 1999 1 Switch 3 • Rev 1.2 completed in 2002 point point • Rev 1.3 completed in 2005 • Rev 2.0 completed in 2007 Port 5-6G PHY, 2, 8 and 16x lanes + Virtual Channels 0 • Recent relevant additions Data streaming, encapsulation, traffic management ► Extensible layered specification End ► Point-to-point packetized architecture point • Low overhead • Variable packet size • Maximum 256 byte PDU • SAR support for 4 Kbyte messages

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 CRF AckID Prio tt FTYPE Target ID Source ID 4 0 0 0 0 (0 1 1 0)

Address 0 XAdd 8

Packet PDU 80

Early CRC Packet PDU 84

Packet PDU CRC 268 Bytes

SWRITE Packet Type: 256 Byte Max Packet PDU

Total = 268 Bytes Physical Layer Transport Layer Logical Layer Payload (256 Byte PDU)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 CRF AckID Prio tt FTYPE Target ID Source ID 4 0 0 0 0 (1 0 1 1)

Msg Source Msg Letter Mbox Packet PDU 8 length size seg

Packet PDU 80

Early CRC Packet PDU 84

2 Bytes Padding CRC 268 Bytes Type 11 Packet Type: 256 Byte Max Packet PDU, 4KB w/SAR

Total = 268 Bytes Physical Layer Transport Layer Logical Layer Payload (256 Byte PDU)

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 AckID Rsvd CRF tt FTYPE Prio Target ID Source ID 4 0 0 0 0 (1 0 0 1) xh Class-of-Service S E Rsvd O P StreamID 8

Packet PDU 80

Early CRC Packet PDU 84

2 Bytes Padding CRC 268 Bytes Type 9 Packet Type: 256 Byte Max Packet PDU, 64KB w/SAR

Total = 268 Bytes Physical Layer Transport Layer Logical Layer Payload (256 Byte PDU)

Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 18 Logical Layer Comparison Ethernet PCI Express® RapidIO® Read/Write Read/Write Memory-mapped R/W No Atomics Configuration Configuration

Write w/Response? N/A None NWRITE_R

Supported address sizes N/A 32, 64-bits 34, 50, 66-bits

Global Shared Memory Not Defined Not Defined Yes

Type 9: 64KB Payloads 1500-9000B Msg: Cntl/Int Messages Messaging/Datagram Type 10: Doorbells Payloads MsgD: User Defined Type 11: 4KB Payloads 10-100s 8 4-16M L2 Type, Traffic Class, Type 9: StreamID, CoS Channelization VLAN Tags, Virtual Channels Type 11: mbox/xmbox UDP/TCP Ports,

SR-IOV, MR-IOV Virtualization Not Defined Not Defined Specifications

Ethernet PCI Express® RapidIO®

Topologies Any Tree Any

Peer-to-peer? Yes Data only Yes

248 (L2) Max number of Large 28 (Small) 232 (IPv4) (Address-dependent) 216 (Large) endpoints 2128 (IPv6) What fields L2: None must switches TLP, Seq Num, LCRC AckID IP: TTL, MAC, FCS modify? Msg only Multicast Yes Yes (Data defined in new ECN) L2: Best Effort Rev 1.x: Guaranteed Delivery Guaranteed TCP/IP: Guaranteed Rev 2.0: +Best Effort

Backplane Ethernet 1000Base- Future 40G Future 100G XAUI CX (40GBase-KR4) (802.3ba) 10GBase-KR 10GBase-KX4

Per port data rate 1G 10G 10G 10G 40G 100G

Per lane baud rate 1.25G 3.125G 10.3125G 3.125G 10.3125G TBD

10x @ 10G or Signal pairs 1x 4x 1x 4x 4x 4x @ 25G 100 cm 100 cm 100 cm Short range Channel 25 m coax 50 cm board backplane + 2 backplane + 2 backplane + 2 copper connectors connectors connectors (50cm board?)

Encoding 8b10b 8b10b 64b66b 8b10b 64b66b TBD

NRZ PECL NRZ NRZ NRZ Signaling TBD TBD AC Coupled AC Coupled AC Coupled AC Coupled Status Shipping Shipping Emerging Emerging Spec in 2009? Spec in 2009?

Also SGMII, Intended for MAC- Pre-emphasis, DFE, 1000Base-T Optional FEC Notes PHY optional FEC Proprietary

Gen1 Gen2 Gen3

Per lane data rate 2G 4G 8G

Per lane baud rate 2.5G 5.0G ???

Signal Pairs 1x, 2x, 4x, 8x, 12x, 16x, 32x

~40-50 cm + Channel 2 connectors

Encoding 8b10b 8b10b ???

Custom Custom Custom Signaling AC Coupled AC Coupled AC Coupled

Status Shipping Emerging Final spec late 2009?

Notes Products 2010?

Rev 1.3 Rev 2.0 Future

Data rate 1.0, 2.0, 2.5 4.0, 5.0 10G

Baud Rate 1.25, 2.5, 3.125 5.0, 6.25 TBD

Signal Pairs 1x, 4x 1x, 2x, 4x, 8x, 16x TBD

~80-100 cm + ~80-100 cm + 100 cm + Channel 2 connectors 2 connectors 2 connectors

Encoding 8b10b 8b10b TBD

XAUI OIF Signaling TBD AC Coupled AC Coupled

Status Shipping 2010 2011? Notes

100% 98% 95% 98% 99% 97% 93% 96% 90% 93% 93% 86% 91% 89% 87% 80% 80% 83% 80%

77% 70% 67% 71%

66% 63% 60% RapidIO NWRITE PCI Express MWr 55% 50% 50% Ethernet L2 49% Ethernet UDP Efficiency

40% 38% 38% 33% 33% 30%

24% 20% 17% 19%

13% 8% 10% 4% 10% 3% 7% 1% 5% 2% 0% 1% 1 10 100 1000 10000 PDU Size (Bytes) NOTE: Includes header & ACK overhead

9.8 9.9 9.7 10 9.6 9.3 9.3 9.4

8.9 9.3 8.6 9.3 9.3

8.0 8.0 8 8.0 7.8 7.9 7.6 6.7 6.6 7.3

6.7 SRIO 4x 3.125G 6 5.7 PCI Express x4 5.0 10G Ethernet: UDP 4.4 4.9 Bandwidth (Gbps) Bandwidth 1G Ethernet: UDP 4 3.3 3.1 3.3 Includes 8B/10B, header and ACK 0.8 1.9 2 1.7 0.5 overhead when 0.4 1.9 0.2 1.1 0.3 0.9 0.9 1.0 1.0 1.0 1.0 present 0.0 0.8 0.1 1.0 0.7 0.5 0.5 0.0 0.3 0.1 0.2 0.0 0 1 10 100 1000 10000 PDU Size (Bytes)

►QoS depends on proper hooks across the interconnect fabric • Hierarchical Flow Control Addresses short, medium and long-term congestion events Link and end-to-end • Ability to define many streams of traffic Often defined as a logical sequence of transactions between two endpoints • Ability to differentiate classes of traffic among streams • Ability to reserve and allocate bandwidth to streams and classes

Overall Interconnect Traffic Streams Classes

►No universal QoS standard ►Many Layer 2+ switches support VLAN Priority Tagging (802.1d/q) • Eight classes ►Increasing number of routers support MPLS at L3 31 12 2 Bytes PRIO CFI VID TCI

Preamble/SFD L2 Header IP Header 8 Bytes 14 Bytes 20 Bytes

UDP Header User PDU FCS IFG 8 Bytes 256 Bytes 4 Bytes 12 Bytes

324 Bytes UDP Packet Type: 1472 byte User PDU (256 Byte User PDU)

►8 Traffic Classes (TC) • No ordering between TCs ►8 Virtual Channel (VC) • Separate buffer resources per VC • TCs are mapped onto VCs TC to VC mapping per port – No VC field in TLP ►Flexible arbitration • Arbitrary, RR, WRR ►Most implementations support a single TC/VC

►All implementations must support 3 prioritized flows • No ordering between flows • Allows shared buffer pool across flows ►Switches required to provide some improved service • Extent of improvement is implementation End dependant Point Y ►Dataplane Extensions adds carrier-grade QoS Flow 0 • Support for thousands of flows, hundreds of End End traffic classes Point Switch Point W X • End-to-end traffic management Flow 1 Flow 2 End Point Z

Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 29 Flow Control Comparison ►Ethernet PCI Express® RapidIO® ► Link-to-link flow control ► Link-to-link flow control • PAUSE frames ►Link-to-link flow control • 802.1Qbb priority-based flow ► Switch/Endpoint-to-endpoint control (new for DCB) • XON, XOFF ► L2 Bridge-to-endpoint ► Fine-grained end-to-end flow • Leverages VLAN tags • Rate limit control • 802.1Qau congestion notification • Data Streaming Logical Layer (new for DCB) End-to-End ► L3+ end-to-end flow control Traffic Mgmt • ECN, TCP windowing, others End End

point XOn-XOff point Congestion Control End End Switch Switch Switch point point

Link-level Flow Control End End point Back point Line Card Plane Line Card

► Several data use models supported by high speed interconnects • Address-based memory-mapped Read/Write • Address-less messaging and datagrams ► Memory-mapped read/write • Very efficient but scales poorly beyond a few devices • Software moves data using low-level memory- mapped read/writes Address range of target device is located – Often using a previously constructed structure produced by an initialization and system discovery routine Target buffer is allocated within the producer’s space DDR DDR – e.g. mmap in Linux Data writes Data is moved using a bcopy or DMA operation

When data transfer is complete, producer notifies Transfer complete notification the consumer – Interrupt, memory semaphore etc – How SW knows last data committed at consumer End point can be an issue Switch End point ► Write w/Response very helpful

► Several data use models are supported by high speed interconnects • Address-based memory-mapped Read/Write • Address-less messaging and datagrams ► Messaging and datagrams • Less efficient but scales well • Some abstract service types Unreliable connectionless messages – Comparable to Ethernet UDP Reliable connectionless messages Reliable connection-oriented messages Reliable connection-oriented byte streams – Comparable to Ethernet TCP • Software typically calls various underlying APIs supplied by drivers to move data Calls abstract underlying interconnect protocols and controllers – Allocate(buffer 0) – Open(AZ) connection to consumer Z on device A – Send(0) – Close(AZ) connection to consumer Z on device A Notification of arrival at device A handled locally by Data writes and notification consumer End point Switch End point

►Many interconnect services ►APIs attempt to abstract underlying interconnect protocols ►Many such APIs have been defined Applications Discovery & Initialization Sockets Proprietary

Shared TCP UDP … Memory RDMA TIPC

Low-level Hardware Device Driver Interconnect Hardware

► Ethernet • Many services and APIs are supported for many OS environments Sockets RDMA TIPC Others… ► PCI Express® • Memory-mapped read/write only Linux – /dev to locate device – mmap() to open buffer at consumer Proprietary services ► RapidIO® • Many proprietary services on Read/Write and messaging • Power Architecture™ processors rionet – Ethernet network stack using RapidIO messaging as packet transport – Work on optimizations using R/W transport • DSP FSL SmartDSP – RapidIO R/W with DMA API – Ethernet over Messaging TI DSP/BIOS – RapidIO Message Queue Transport (MQT)

Ethernet PCI Express® : Rd, Wr TCP/IP UDP Layer 2 RapidIO® : SWRITE, MSG, Streaming

Application Application Application App Client SW Client Sockets API SW Stack Driver SW SAR/Err SAR/Err DMA Stack SW TCP/IP UDP/IP Logical HW DMA DMA DMA Trans HW MAC MAC MAC Link/PHY

►High-bandwidth interconnects require low CPU overhead usage model • Hardware support for logical, transport and link layer • Low overhead DMA with QoS support

►Microsecond+ fall through latencies (~100us?) • Not just the hardware, data has to traverse the SW stack ►High CPU overhead • Rule of thumb appears to be borne out in data for TCP/IP termination SW overhead 1 Hz of CPU per bit of throughput (per direction) • Wire speed achievable with GHz class processors Some CPU will be left but how much depends on – Protocol being terminated – Offload features of GigE interfaces • Too often advanced off-load features cannot be leveraged OS & SW stack support issues ►UDP or MAC/Layer 2 solutions sometimes use proprietary higher layer protocols Can defeat the value of off-the-shelf “standards-based” solution ►Error correction at endpoint stacks introduce latency jitter and determinism issues ►Works well when application requires < 30% fabric utilization • Lack of flow control problematic for systems that can’t significantly overprovision

►Latency • Sub-microsecond switch latencies • PCI Express switches must manage address mapping ►End-to-end latency • Lower latency than Ethernet since latency does not include a SW stack ►Architecture • PCI Express switches allow limited peer-to-peer communication Multiple hosting for redundancy problematic Maintenance responses as well as wake-up beacons must move upstream Some switches support non-transparent bridging – Create two separate spaces for each host – Non-standard and implementation specific Must collect INTx messages and some power management transactions ►RapidIO switches straightforward and orthogonal in architecture • Strict peer-to-peer • Packet headers architected to reduce logic • No need to recalculate CRC

►RapidIO®, PCI Express® and Ethernet with modest TCP/IP offload have similar underlying silicon costs • PCI Express controller is slightly larger than RapidIO • Aggressive TCP/IP Offload engine larger than PCI Express and RapidIO endpoints

►Interesting fact about switches • Available established vendors for all three interconnects similar: 2-3

►Leveraging Ethernet volume economics not always a reality • L2+ Ethernet switches suitable for aggregation and backplanes are not high volume 16-24 ports, QoS features and SERDES PHYs for backplane 12-16 ports, QoS features for aggregation • Terminating TCP/IP imposes significant processor overhead Dedicate processor or reduce performance and/or application features

Attribute Ethernet PCI Express® RapidIO®

Low latency SW Stack Low CPU overhead Limited TOE QoS: Flow Control Link Only Link Only QoS: Channelization w/Flow Control VLAN, No FC 1-2 VC Avail. QoS: Jitter SW Stack Multicast (w/o New ECN) Virtualization Support High availability (hot plug, multiple hosting) Large number of endpoints Tree/Bridge Peer-to-peer for data Peer-to-peer for management (w/o MR-IOV) High port bandwidth (>10G) Commodity off-the-shelf endpoints (graphics, NICs, HBAs, etc)

Good Fit Marginal Poor Fit

► Ethernet ubiquitous as LAN-scale interconnect • GigE ubiquitous, 10G Ethernet will segment market for first time • Broad endpoint silicon and software support • Useful in low bandwidth embedded applications ► PCI Express® widely deployed in PC/Server space • Significant role in the embedded space Where there is an intersection with the PC & Server space Where PCI has been used • Backplane interconnect role in the embedded space will be limited Unwieldy when connecting large numbers of endpoints Similar switch ecosystem to RapidIO® and Ethernet • Broad switch, IP and endpoint ecosystem ► RapidIO deployed with growing ecosystem • Expanding from initial Military/Aero, DSP and line card aggregation role • Best positioned for multicore applications • Will gradually expand role onto the backplane Efficient protocol supporting both control and data plane Variety of PHY speeds • Cost competitive against 1G and 10G Ethernet • Established and diverse ecosystem