July 2009 Comparison of High-Speed Interconnects: Ethernet, PCI Express® and RapidIO® Technology Greg Shippen System Architect, Network Systems Division Networking & Multimedia Group TM Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. Agenda ►Interconnect Trends ►Technical Overview ►Comparison ►Summary and Conclusion Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 2 QorIQ™ P4 Series P4080 Block Diagram QorIQ™ P4080 MULTICORE 1024 KB 64-bit PROCESSOR Power Architecture™ Frontside DDR-2 / 3 128 KB e500-mc Core L3 Cache Memory Controller Backside 1024 KB 64-bit L2 Cache 32 KB 32 KB DDR-2 / 3 D-Cache I-Cache Frontside L3 Cache Memory Controller eOpenPIC CoreNet™ PreBoot Loader Coherency Fabric Peripheral PAMU Security Monitor PAMU PAMU PAMU PAMU Access Mgmt Unit Internal BootROM Power Mgmt Frame Manager Frame Manager Real Time Debug RapidIO® Watchpoint SD/MMC Security Queue eLBC Message 2x DMA Cross 4.0 Mgr. Parse, Classify, Parse, Classify, Trigger SPI Distribute Distribute Unit (RMU) 2x DUART Pattern Buffer Buffer Perf CoreNet Test Monitor Trace Match Buffer 4x I 2C Port/ Engine Mgr. 1GE 1GE 1GE 1GE PCIe PCIe PCIe sRIO 10GE SAP 2.0 10GE SRIO 2x USB 2.0/ULPI 1GE 1GE 1GE 1GE Aurora Clocks/Reset GPIO 18-Lane 5 GHz SerDes CCSR Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 3 Our Customer Feedback on Interconnects I want more CPU cycles Quit spending them moving data Support living standards I don’t want to change next time with a living ecosystem I want to meet my technical Implement QoS, scalable BW, requirements multicore ready, high availability… I don’t want to rewrite my Use common usage models software and software APIs I want it cheap Use multi-vendor standards Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 4 Market Trends Modularity, Reuse Multicore Devices Bandwidth CPU CPU CPU CPU GB/s Accel I/O I/O Cost Connected $$$ NRE Devices Protocols SP $$$ OPEX X I4 CSI TDM .2 P C I E x P pr /I es $$$ CAPEX P s ® TC ATM Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 5 Interconnect Trends ► 2nd Generation Point-to-Point • Packet switched Device Device • PHY: SERDES differential st ► 1 Generation Point-to-Point • Lowest pin count Switch • Packet switched Fabric Device • PHY: Source-sync differential • Lower pin count Device Device Device Device ≥ 10 GHz Example: HT/P-RapidIO® ≤ 3 GHz Device Ex: PCIe, ► Hierarchical Bus S-RapidIO, Device Device • Bridged Hierarchy SATA, • Broadcast Device SAS Bridge Device Device • PHY: Single-ended Example: PCI/PCI-X/SCSI ≤ 133MHz ce ► Shared Bus a n • Single segment Device Device Device Device Device orm • Broadcast f er • PHY: Single-ended Device Device Device Device Device P • Highest pin count Example: VME ≤ 66MHz Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 6 Interconnect Roles ►Chip-to-chip ►Board-to-Device ►Board-to-board ►Chassis-to-chassis Chassis-to-chassis Device Chip-to-chip Board-to-Board Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 7 Technical Overview Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 8 Ethernet Overview ► WAN scale interconnect • Box-to-box, board-to-board, backplane • Connect thousands to millions of endpoints • Physical layer defined for LAN-scale interconnection Closet to computer Backplane Ethernet Endpoint • Optical, twisted pair and backplane copper media CPU ► Target market • GigE WAN to workstations, PCs and laptops DRAM • 10GE now used in aggregation settings High performance switches, routers and LAN backbones MAC/PHY ► Specification history • First spec (10Mbps) ~1975 by Xerox Port 2 • 100Mbps spec in 1995 • 1Gbps spec in 1998 End Port Switch/ Port End • 10Gbps spec in 2002 1 3 10G Copper (10GBase-T) in 2006 point Router point • Recent relevant additions Backplane Ethernet (802.3ap-2007) Port Data Center Bridging (DCB) 0 ► Gigabit Ethernet ubiquitous now • 10G Copper PHYs shipping End ► Extensible layered specification point ► Point-to-point packetized architecture • High header overhead • Variable packet size • 46-1500 byte packet L2 PDU • Up to 9000 byte jumbo frames Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 9 Ethernet Layer 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Preamble 4 Preamble SFD 8 Destination Address 12 Destination Address Source Address 16 Source Address 20 Type/Length Packet PDU 24 Packet PDU 278 FCS 282 Inter-Frame Gap 294 Bytes Layer 2 Packet Type: 1500 Byte Max Packet PDU Total = 294 Bytes Interframe overhead L2 header/trailer Payload (256 Byte PDU) Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 10 Ethernet + TCP/IP Preamble/SFD L2 Header IP Header 8 Bytes 14 Bytes 20 Bytes TCP Header User PDU FCS IFG 334 Bytes 20 Bytes 256 Bytes 4 Bytes 12 Bytes TCP/IP Packet Type: 1460 Byte Max User PDU Total = 334 Bytes (256 Byte User PDU) Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 11 PCI Express® Overview ► Chassis-scale interconnect • Chip-to-chip, Board-to-board CPU • Required legacy PCI compatibility • Physical layer defined for board + connector • Copper-on-board and cable media ► Successor to PCI 2.3/PCI-X 2.0 Host/ • Fully SW/firmware backward compatible to PCI End Port Port End Root ► Target market point 0 2 point • PC and Servers space Complex • Embedded where suitable Port ► Specification history 1 • Rev 1.0 (Gen1) completed in 2002 Upstream Switch Port External cable spec released Feb 2007 • Rev 2.0 (Gen2) completed in 2006 Switch P2P • Rev 3.0 (Gen3) expected “late 2009” 8 GTransfers/s • Recent relevant additions P2P P2P P2P Multiroot/single-root IO Virtualization Cable Spec Down Down Down ► stream stream stream PCIe Gen2 now widely deployed Port Port Port • First Gen2 Intel Silicon (X38 chipset) Sep 2007 ► Extensible layered specification ► Point-to-point packetized architecture End End End • Relatively low overhead point point point • Variable size packets • 128-4096 byte PDU Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 12 PCI Express® Protocol 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Rsvd TLP Sequence Number R FMT Type R TC Rsvd 4 T E D P Attr R Length Requester ID 8 Last DW First DW Tag Address[31:16] 12 BE BE Address[15:2] R Packet PDU 16 Packet PDU 20 Packet PDU Optional TLP Digest (ECRC) 272 Optional TLP Digest (ECRC) Cont LCRC 276 LCRC Cont Next Packet/DLLP 280 Bytes Memory Write: 4096 Byte Max Packet PDU Total = 278 Bytes Link Layer Transaction Layer Payload (256 Byte PDU) Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 13 RapidIO® Overview ► Chassis scale interconnect • Chip-to-chip, Board-to-board, backplane • Initially a processor interconnect as Motorola/Mercury collaboration • Physical layer defined for board + connectors • Copper-on-board and cable media ► Target market Host • Embedded systems Wireless infrastructure, media, networking, compute & defense • CPU I/O, Line-card aggregation, backplane Port • Extensive dataplane features 2 QoS, VCs, datagrams, encapsulation ► Specification History End Port Port End • Rev 1.0 completed in 1999 1 Switch 3 • Rev 1.2 completed in 2002 point point • Rev 1.3 completed in 2005 • Rev 2.0 completed in 2007 Port 5-6G PHY, 2, 8 and 16x lanes + Virtual Channels 0 • Recent relevant additions Data streaming, encapsulation, traffic management ► Extensible layered specification End ► Point-to-point packetized architecture point • Low overhead • Variable packet size • Maximum 256 byte PDU • SAR support for 4 Kbyte messages Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc. All other product or TM service names are the property of their respective owners. © Freescale Semiconductor, Inc. 2009. 14 RapidIO® Packet Format: SWRITE 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 CRF AckID Prio tt FTYPE Target ID Source ID 4 0 0 0 0 (0 1 1 0) Address 0 XAdd 8 Packet PDU 80 Early CRC Packet PDU 84 Packet PDU CRC 268 Bytes SWRITE Packet Type: 256 Byte Max Packet PDU Total = 268 Bytes Physical Layer Transport Layer Logical Layer Payload (256 Byte PDU) Freescale™ and the Freescale logo are trademarks of Freescale Semiconductor, Inc.