FB-DIMM Spring IDF Architecture Presentation
Total Page:16
File Type:pdf, Size:1020Kb
Fully Buffered DIMM (FB-DIMM) Server Memory Architecture: Capacity, Performance, Reliability, and Longevity Pete Vogt Principal Engineer Intel Corp February 18, 2004 1 Agenda y Server Capacity Problem and Solution y Performance y Reliability y Longevity 2 Problem & Solution Intel Platform Memory Technology Roadmap DDR3 DDR3 FB-DIMM DDR2 667/800 DDR2 DDR2 400/533 DDR DDR333/400 DDR266 RDRAM* PC1066 PC800 SDRAM PC133 2004 2005 20052006 y DDR2 400/533 support in all main IA segments in 2004 – with DDR flexibility y FB-DIMM new server interconnect in 2005 y Low Power SDRAM moving to low power DDR y RDRAM still used in specific applications *Other names and brands may be claimed as the property of others 3 Problem & Solution FB-DIMM New Server Interconnect in 2005 Why is FB-DIMM Needed in the Future? 4 Problem & Solution DRAM Device Trends DRAM Bit Density; Mb/dev y DRAM bit density follows 2500 Moore’s law and 2000 increases 2X every ~2 1500 years 1000 500 0 99 00 01 02 03 04 05 06 07 Rule of thumb - Source: Pete Vogt DRAM Data Rate; Mbps y DRAM data rates driven 1200 1000 by CPU performance 800 increase 2X every 600 400 generation (3.35 years) 200 0 99 00 01 02 03 04 05 06 07 Projected historical data - Source: Pete Vogt DRAM Devices Continue to Scale 5 Problem & Solution Devices per Channel Trend Stub-bus Topology y Existing “stub-bus” DRAM DRAM architecture has DRAM DRAM impedance discontinuities Mem that effect signal integrity Ctrl Impedance Discontinuities Stub-bus Devices/Channel 160 y As the data rate goes up 140 120 the number of devices on 100 80 the channel is going down 60 40 20 0 99 00 01 02 03 04 05 06 07 Estimated trend - Source: Pete Vogt Devices per Channel is Decreasing 6 Problem & Solution Memory Capacity Trends Demand Capacity Demand; GB y Server performance drives 35 30 2X capacity increase 25 demand every ~2 years 20 15 10 5 0 99 00 01 02 03 04 05 06 07 Rule of thumb - Source: Pete Vogt Stub-bus Capacity/Ch: GB Demand 32 y Stub-bus channel capacity 28 24 (density * devices/ch) 20 Gap is 16 growing supply has hit a ceiling 12 8 Supply 4 0 99 00 01 02 03 04 05 06 07 Computed trend - Source: Pete Vogt Meeting Demand Requires a Change 7 Problem & Solution FB-DIMM Eliminates the “Stubs” DRAM DRAM DRAM DRAM y FB-DIMM buffers the DRAM Buffer Buffer data pins from the channel Buffer Buffer and uses point-to-point links to eliminate the stub Mem bus Ctrl Point-to-Point Links FB-DIMM FB-DIMM Capacity/Ch: GB y FB-DIMM capacity scales 35 30 throughout DDR2 & DDR3 25 20 generations 15 10 5 0 99 00 01 02 03 04 05 06 07 FB-DIMM Capability - Source: Pete Vogt FB-DIMM Meets the Capacity Demand 8 Problem & Solution FB-DIMM Solution Details DDR2 connector Commodity with unique key DRAMs Serial signaling similar to DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM PCI-Express DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM Up to 8 DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM 10 DIMMs Buffer Buffer Buffer ••• Buffer Memory 14 Controller DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM SMBus DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM CLK Source Common clock SMbus access to buffer source registers 9 Problem & Solution FB-DIMM Channel Pin Count Diff Signals Pins Data Path to DIMMs 10 20 Data Path from DIMMs 14 28 Total High-Speed Signals 48 Power 6 Ground 12 Shared Pins (clocks, ~ 3 calibration, PLL pwr, test) Total Pins ~ 69 y Compare with ~240 pins for DDR2 channel FB-DIMM Pin Count is 1/3rd of DDR2 10 Problem & Solution Routing Comparison Direct DDR2 Registered DIMMs: FB-DIMMs: 1 Channel, 2 Routing Layers with 3rd layer 2 Channels, 2 Routing Layers required for power Serpentine (includes power delivery) routing is complicated and uses up a lot of board area Fewer signals and no trace length matching minimizes board area FB-DIMM: Fewer Layers, Less Routing Area 11 Source: Intel Enterprise Architecture Group Problem & Solution Expansion and Visibility ~12 inches ~12 inches y Repeaters support Riser Card er flexible system M M M M M M M M Memory at e M M M Controller M M M M M packaging and DI DI DI DI DI DI DI DI Rep memory riser cards M M y Logic Analyzer Interface DI r provides visibility of e f Logic Analyzer high-speed bus activity I buf Riser for system analysis LA Memory M M M M M M M Controller M M M M M M M DI DI DI DI DI DI DI FB-DIMM is a Versatile Solution 12 Problem &Solution FB-DIMM Solves theServer Memory CapacityProblem y y y Capacity Comparison ~Lower pincount ~4x bandwidth 24x capacity – – – ~480 vs.~420 ~10GB/s vs.~40GB/s 8GB vs.192GB ~ 1 Co (on 8GB Me 0GB DD nt l y m rol w 2 R2 / o s rank r i l of t e y h r 1Gb B s W p w x e 13 r c 4 / DDR2-8 DR h an A ne DIMM DIMM M s l 00 ) DIMM DIMM FB Co Me -D nt m ~ ro IM 19 40 GB/ o l r l e y 2G M r (2 rank B w s of i t h B s 1 perDI W G b w x / DIMM DIMM DIMM DIMM DIMM DIMM 4 D M DRA D DIMM DIMM DIMM DIMM DIMM DIMM M R ) 2 DIMM DIMM DIMM DIMM DIMM DIMM -8 M 00 s DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM DIMM FB-DIMM Performance 14 Performance Entry Configuration Comparison y Equal capacity – 8GB vs. 8GB y Better throughput M M – ~6.5GB/s vs. ~8.2GB/s DI M y Lower pin count M FB-DIMM DI – ~480 vs. ~280 Memory Controller M M M M DDR2 M M DI DI DI Memory Controller M M M M M M DI DI DI 8GB with 1Gb x4 DRAMs 8GB with 1Gb x4 DRAMs ~6.5GB/s of throughput w/DDR2-800 ~8.2GB/s of throughput w/DDR2-800 (only 2 ranks per channel) (1 rank per DIMM) Better Throughput with 200 Fewer Pins 15 Performance Entry Configuration Comparison FB-DIMM provides higher sustained throughput ncy because of the increased e number of channels ad Lat e ry R o m e FB-DIMM has lower latency verage M at higher throughput A Inherent serialization delay, decreases as Source: Intel data rate goes up 2 DDR2 800 channels, 2 ranks per channel 4 FB-DIMM 800 channels, 1 single rank DIMM per channel 0 2 4 6 8 1012141618 Sustained Memory Subsystem Throughput (GB/s) 16 Performance Mid-range Config Comparison y 4x capacity – 8GB vs. 32GB y ~2.5x throughput M M M M – ~6.5GB/s vs. ~16.5GB/s DI DI M M y Lower pin count 4 DIMMs per channel M M doesn’t work FB-DIMM DI DI – ~480 vs. ~280 Memory Controller M M M M M M M M M DDR2 M M M DI DI DI DI DI DI Memory Controller M M M M M M M M M M M M DI DI DI DI DI DI 8GB with 1Gb x4 DRAMs 32GB with 1Gb x4 DRAMs ~6.5GB/s of throughput w/DDR2-800 ~16.5GB/s of throughput w/DDR2-800 (only 2 ranks per channel) (2 ranks per DIMM) FB-DIMM Provides 4x the Capacity 17 Performance Mid-range Config Comparison FB-DIMM provides very high sustained throughput due to Latency crossover simultaneous reads/writes point is earlier ead Latency ry R o m e age M r FB-DIMM has consistent latency e v A Slightly larger delay for across a wide throughput range the additional buffer Source: Intel 2 DDR2 800 channels, 2 ranks per channel 4 FB-DIMM 800 channels, 2 dual rank DIMMs per channel 0 2 4 6 8 1012141618 Sustained Memory Subsystem Throughput (GB/s) 18 Performance Features That Reduce Latency y Fast pass-through-path in the IO cells – The data is not stored and forwarded y DRAM synchronized to channel clock – No asynchronous clock boundary crossings y Simultaneous Reads and Writes – Writes do not block read accesses y Elimination of read-to-read dead time – Reads from different DIMMs have no dead time between data transfers New Channel Features Mitigate Buffer Latency 19 FB-DIMM RAS Features 20 Reliability Channel Reliability Targets y Intel’s server goal is >100 years (1142 FITs) per Silent Data Corruption – The memory channels receive a small portion of this total FIT budget y FB-DIMM is architected for exceptional Silent Data Corruption prevention – Channel data is protected by a strong CRC – Per channel segment SDC FIT rate <0.10 (1,142,000 years) to support even the highest-RAS servers 21 Reliability Improved RAS Features y CRC protection for commands and data y Optional bit widths and CRC coverage to cover wide range of applications y Transient bit error detection and retry y Bit lane fail-over “correction” y Pass-through-path for high availability y Hot Add while the channel is active y Error registers in the buffer for improved fault isolation Comprehensive RAS Features Deliver a Reliable Solution 22 FB-DIMM Longevity 23 Longevity FB-DIMM Momentum y Industry is aligned around FB-DIMM – Server OEM suppliers recognize the need for a long-term buffered DRAM memory solution – DRAM & DIMM vendors committed to delivering FB- DIMM products: Samsung, Elpida, Infineon, Micron, Hynix, Nanya, Kingston, Smart – Complete set of tools for system development y FB-DIMM is a long-term strategic direction – Smooth transition from DDR2 to DDR3 using the same connectors and topology – Industry investment in the technology leveraged over multiple generations 24 Longevity FB-DIMM Standardization y Intel working with JEDEC to establish FB-DIMM as an industry standard y Seven specifications being developed – FB-DIMM: Architecture & Protocol – FB-DIMM: High-speed Signaling – FB-DIMM: Connector (variation of DDR2) – FB-DIMM: Module – FB-DIMM: Advanced Memory Buffer (AMB) – FB-DIMM: SPD EPROM – FB-DIMM: Test & Verification 25 Longevity FB-DIMM Technology Roadmap y DDR2 is the first technology intercept – Needed part-way through the DDR2 lifetime y All DDR3 data rates supported ? FB-DIMM DDR3 DDR3 FB-DIMM DDR2 DDR2 DDR 01 02 03 04 05 06 07 08 09 26 Longevity FB-DIMM Cost Effectiveness y Cost competitive environment – Multiple buffer vendors and DIMM suppliers