Fully Buffered DIMM (FB-DIMM) Server Memory Architecture: Capacity, Performance, Reliability, and Longevity Pete Vogt Principal Engineer Corp

February 18, 2004

1 Agenda y Server Capacity Problem and Solution y Performance y Reliability y Longevity

2 Problem & Solution Intel Platform Memory Technology Roadmap

DDR3 DDR3

FB-DIMM DDR2 667/800 DDR2 DDR2 400/533 DDR DDR333/400 DDR266

RDRAM* PC1066 PC800

SDRAM PC133

2004 2005 20052006

y DDR2 400/533 support in all main IA segments in 2004 – with DDR flexibility y FB-DIMM new server interconnect in 2005 y Low Power SDRAM moving to low power DDR y RDRAM still used in specific applications

*Other names and brands may be claimed as the property of others 3 Problem & Solution

FB-DIMM New Server Interconnect in 2005

Why is FB-DIMM Needed in the Future?

4 Problem & Solution

DRAM Device Trends

DRAM Bit Density; Mb/dev y DRAM bit density follows 2500 Moore’s law and 2000 increases 2X every ~2 1500 years 1000 500

0 99 00 01 02 03 04 05 06 07 Rule of thumb - Source: Pete Vogt DRAM Data Rate; Mbps y DRAM data rates driven 1200 1000 by CPU performance 800 increase 2X every 600 400

generation (3.35 years) 200

0 99 00 01 02 03 04 05 06 07 Projected historical data - Source: Pete Vogt DRAM Devices Continue to Scale

5 Problem & Solution

Devices per Channel Trend

Stub-bus Topology y Existing “stub-bus” DRAM DRAM architecture has DRAM DRAM impedance discontinuities Mem that effect signal integrity Ctrl Impedance Discontinuities

Stub-bus Devices/Channel 160 y As the data rate goes up 140 120 the number of devices on 100 80 the channel is going down 60 40 20 0 99 00 01 02 03 04 05 06 07 Estimated trend - Source: Pete Vogt Devices per Channel is Decreasing

6 Problem & Solution

Memory Capacity Trends Demand Capacity Demand; GB y Server performance drives 35 30 2X capacity increase 25 demand every ~2 years 20 15 10 5 0 99 00 01 02 03 04 05 06 07 Rule of thumb - Source: Pete Vogt Stub-bus Capacity/Ch: GB Demand 32 y Stub-bus channel capacity 28 24 (density * devices/ch) 20 Gap is 16 growing supply has hit a ceiling 12 8 Supply 4 0 99 00 01 02 03 04 05 06 07 Computed trend - Source: Pete Vogt Meeting Demand Requires a Change

7 Problem & Solution

FB-DIMM Eliminates the “Stubs” DRAM DRAM DRAM DRAM y FB-DIMM buffers the DRAM Buffer Buffer data pins from the channel Buffer Buffer and uses point-to-point

links to eliminate the stub Mem bus Ctrl Point-to-Point Links

FB-DIMM FB-DIMM Capacity/Ch: GB y FB-DIMM capacity scales 35 30 throughout DDR2 & DDR3 25 20 generations 15 10 5 0 99 00 01 02 03 04 05 06 07 FB-DIMM Capability - Source: Pete Vogt FB-DIMM Meets the Capacity Demand

8 Problem & Solution

FB-DIMM Solution Details DDR2 connector Commodity with unique key DRAMs Serial signaling similar to DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM PCI-Express DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM Up to 8 DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM 10 Buffer Buffer Buffer ••• Buffer Memory 14 Controller DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM

SMBus DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM

CLK Source

Common clock SMbus access to buffer source registers

9 Problem & Solution

FB-DIMM Channel Pin Count

Diff Signals Pins Data Path to DIMMs 10 20 Data Path from DIMMs 14 28 Total High-Speed Signals 48 Power 6 Ground 12 Shared Pins (clocks, ~ 3 calibration, PLL pwr, test) Total Pins ~ 69

y Compare with ~240 pins for DDR2 channel

FB-DIMM Pin Count is 1/3rd of DDR2

10 Problem & Solution

Routing Comparison Direct DDR2 Registered DIMMs: FB-DIMMs: 1 Channel, 2 Routing Layers with 3rd layer 2 Channels, 2 Routing Layers required for power Serpentine (includes power delivery) routing is complicated and uses up a lot of board area

Fewer signals and no trace length matching minimizes board area

FB-DIMM: Fewer Layers, Less Routing Area

11 Source: Intel Enterprise Architecture Group Problem & Solution

Expansion and Visibility ~12 inches ~12 inches

y Repeaters support Riser Card er

flexible system M M M M M M M M Memory at e M M M Controller M M M M M packaging and DI DI DI DI DI DI DI DI Rep memory riser cards M M y Logic Analyzer Interface DI r

provides visibility of e

f Logic Analyzer

high-speed bus activity I buf Riser for system analysis LA

Memory M M M M M M M Controller M M M M M M M DI DI DI DI DI DI DI

FB-DIMM is a Versatile Solution

12 Problem & Solution M M M M M M M M M M M M M M M M

Capacity Comparison DI DI DI DI DI DI DI DI M M M M M M M M M M M M M M M M

y 24x capacity DI DI DI DI DI DI DI DI – 8GB vs. 192GB M M M M M M M M M M M M M M M M DI DI y ~4x bandwidth FB-DIMM DI DI DI DI DI DI

– ~10GB/s vs. ~40GB/s M M M M M M M M M M M M M M M M y ~Lower pin count DI DI DI DI DI DI DI DI M M M M M M M M M M M M M M M M M M – ~480 vs. ~420 M M DDR2 DI DI DI DI DI DI DI DI DI DI Memory Controller M M M M M M M M M M M M M M M M M M M M DI DI DI DI DI DI DI DI DI DI

8GB with 1Gb x4 DRAMs 192GB with 1Gb x4 DRAMs ~10GB/s of BW w/DDR2-800 ~40 GB/s of BW w/DDR2-800 (only 2 ranks per channel) (2 ranks per DIMM) FB-DIMM Solves the Server Memory Capacity Problem

13 FB-DIMM Performance

14 Performance

Entry Configuration Comparison

y Equal capacity – 8GB vs. 8GB

y Better throughput M M – ~6.5GB/s vs. ~8.2GB/s DI M

y Lower pin count M

FB-DIMM DI – ~480 vs. ~280 Memory Controller M M M M DDR2 M M DI DI DI Memory Controller M M M M M M DI DI DI

8GB with 1Gb x4 DRAMs 8GB with 1Gb x4 DRAMs ~6.5GB/s of throughput w/DDR2-800 ~8.2GB/s of throughput w/DDR2-800 (only 2 ranks per channel) (1 rank per DIMM)

Better Throughput with 200 Fewer Pins

15 Performance

Entry Configuration Comparison

FB-DIMM provides higher sustained throughput

ncy because of the increased e number of channels ad Lat e ry R o m e FB-DIMM has lower latency

verage M at higher throughput

A Inherent serialization delay, decreases as Source: Intel data rate goes up 2 DDR2 800 channels, 2 ranks per channel 4 FB-DIMM 800 channels, 1 single rank DIMM per channel

0 2 4 6 81012141618 Sustained Memory Subsystem Throughput (GB/s)

16 Performance

Mid-range Config Comparison

y 4x capacity – 8GB vs. 32GB

y ~2.5x throughput M M M M – ~6.5GB/s vs. ~16.5GB/s DI DI M M

y Lower pin count 4 DIMMs per channel M M doesn’t work FB-DIMM DI DI – ~480 vs. ~280 Memory Controller M M M M M M M M M DDR2 M M M DI DI DI DI DI DI Memory Controller M M M M M M M M M M M M DI DI DI DI DI DI

8GB with 1Gb x4 DRAMs 32GB with 1Gb x4 DRAMs ~6.5GB/s of throughput w/DDR2-800 ~16.5GB/s of throughput w/DDR2-800 (only 2 ranks per channel) (2 ranks per DIMM)

FB-DIMM Provides 4x the Capacity

17 Performance

Mid-range Config Comparison

FB-DIMM provides very high sustained throughput due to Latency crossover simultaneous reads/writes point is earlier ead Latency ry R o m e age M

r FB-DIMM has consistent latency e v

A Slightly larger delay for across a wide throughput range the additional buffer Source: Intel 2 DDR2 800 channels, 2 ranks per channel 4 FB-DIMM 800 channels, 2 dual rank DIMMs per channel

0 2 4 6 81012141618 Sustained Memory Subsystem Throughput (GB/s)

18 Performance

Features That Reduce Latency

y Fast pass-through-path in the IO cells – The data is not stored and forwarded y DRAM synchronized to channel clock – No asynchronous clock boundary crossings y Simultaneous Reads and Writes – Writes do not block read accesses y Elimination of read-to-read dead time – Reads from different DIMMs have no dead time between data transfers

New Channel Features Mitigate Buffer Latency

19 FB-DIMM RAS Features

20 Reliability

Channel Reliability Targets

y Intel’s server goal is >100 years (1142 FITs) per Silent Data Corruption – The memory channels receive a small portion of this total FIT budget y FB-DIMM is architected for exceptional Silent Data Corruption prevention – Channel data is protected by a strong CRC – Per channel segment SDC FIT rate <0.10 (1,142,000 years) to support even the highest-RAS servers

21 Reliability

Improved RAS Features

y CRC protection for commands and data y Optional bit widths and CRC coverage to cover wide range of applications y Transient bit error detection and retry y Bit lane fail-over “correction” y Pass-through-path for high availability y Hot Add while the channel is active y Error registers in the buffer for improved fault isolation

Comprehensive RAS Features Deliver a Reliable Solution

22 FB-DIMM Longevity

23 Longevity

FB-DIMM Momentum

y Industry is aligned around FB-DIMM – Server OEM suppliers recognize the need for a long-term buffered DRAM memory solution – DRAM & DIMM vendors committed to delivering FB- DIMM products: Samsung, Elpida, Infineon, Micron, Hynix, Nanya, Kingston, Smart – Complete set of tools for system development y FB-DIMM is a long-term strategic direction – Smooth transition from DDR2 to DDR3 using the same connectors and topology – Industry investment in the technology leveraged over multiple generations

24 Longevity

FB-DIMM Standardization

y Intel working with JEDEC to establish FB-DIMM as an industry standard y Seven specifications being developed – FB-DIMM: Architecture & Protocol – FB-DIMM: High-speed Signaling – FB-DIMM: Connector (variation of DDR2) – FB-DIMM: Module – FB-DIMM: Advanced Memory Buffer (AMB) – FB-DIMM: SPD EPROM – FB-DIMM: Test & Verification

25 Longevity

FB-DIMM Technology Roadmap

y DDR2 is the first technology intercept – Needed part-way through the DDR2 lifetime y All DDR3 data rates supported ? FB-DIMM DDR3 DDR3

FB-DIMM DDR2 DDR2

DDR

01 02 03 04 05 06 07 08 09

26 Longevity

FB-DIMM Cost Effectiveness

y Cost competitive environment – Multiple buffer vendors and DIMM suppliers y Eliminates memory expansion hubs – Motherboard not burdened with hubs y Reduces motherboard routing area – Only 24 pairs, no serpentine trace matching y Enables reduced PCB layer count – FB-DIMM can be routed in one signal layer

FB-DIMM is a cost effective long-term technology

27 Summary

Summary

y FB-DIMM solves the long-term server memory capacity problem y Buffer latency is managed with new channel features y Comprehensive RAS features deliver a reliable solution y FB-DIMM is a cost effective long-term technology

FB-DIMM is The Server Memory DIMM of the Future

28 Fully Buffered DIMM Server Memory Architecture

Pete Vogt

Intel Corporation

Any Questions?

PleasePlease rememberremember toto turnturn inin youryour sessionsession surveysurvey form.form.

29

Definitions y AMB: Advanced Memory Buffer – used to implement FB-DIMM y FB-DIMM: Fully Buffered DIMM y FIT: Failure in Time (failures in 1 billion hours) y JEDEC: Joint Electronic Devices Engineering Council y RAS: Reliability/Availability/Serviceability y RDIMM: Registered DIMM y SDC: Silent Data Corruption

31 Fully Buffered DIMM Architecture y Session Outline: y Server Memory Capacity Problem & Solution – Description of the capacity problem facing servers as data rates go up. Graph of DRAMs/channel vs. data rate. Description of the high-level FB-DIMM architecture that addresses this problem utilizing commodity DRAM devices. – Key point: FB-DIMM solves the Server Memory Capacity problem y FB-DIMM Performance Features – Description of architecture features to mitigate the buffer latency; synchronous operation, fast pass through path, simultaneous reads/writes, & elimination of read-to-read dead time. Loaded latency graph illustrates result. – Key point: Buffer latency is managed with new channel features y FB-DIMM Reliability/Availability/Serviceability (RAS) Features – Highlight RAS features in FB-DIMM including bit lane fail-over, reduced pin counts, high-availability pass through path, deterministic behavior, strong channel CRC protection. – Key point: Comprehensive RAS features deliver a reliable solution y Long-term Server Memory Solution – Describe the longevity of FB-DIMM from DDR2-533 to DDR3-1600, indicate multiple buffer vendor sources, working within JEDEC to establish as a standard. – Key point: FB-DIMM is a cost effective long-term technology

Pete Vogt 32 Title: Fully Buffered DIMM (FB-DIMM) Server Memory Architecture: Capacity, Performance, Reliability, and Longevity Abstract: What you’ll get from this session: – Trend data of the server memory capacity problem – Loaded latency curves illustrating FB-DIMM architecture features to mitigate inherent buffer delay – List of mechanisms that address FB-DIMM reliability concerns – Roadmap of multiple generations of DRAM support

33