Fully Buffered DIMM (FB-DIMM) Design Considerations
Howard David Memory Architect
Michael McTague Staff Engineer
Intel Corp.
Feb 18, 2004
1 Agenda
y FB-DIMM Architecture y FB-DIMM Channel Electrical Characteristics y FB-DIMM Physical Design y FB-DIMM System Design Considerations
2 Intel Platform Memory Technology Roadmap
DDR3 DDR3
FB-DIMM DDR2 667/800 DDR2 DDR2 400/533 DDR DDR333/400 DDR266
RDRAM* PC1066 PC800
SDRAM PC133
2004 2005 20052006
y DDR2 400/533 support in all main IA segments in 2004 – with DDR flexibility y FB-DIMM new server interconnect in 2005 y Low Power SDRAM moving to low power DDR y RDRAM still used in specific applications
*Other names and brands may be claimed as the property of others 3 FB-DIMM Architecture FB-DIMM Block Diagram y DRAM interface is entirely behind the buffer – DDR2 DRAM scales from 533 to 800 MT/s, with up to 8 DIMMs (288 devices) per channel
Commodity DRAMDRAM DRAMDRAM DRAMDRAM DRAMs DRAMDRAM Differential Pairs DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM Up to 8 DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM 10 DIMMs Buffer Buffer Buffer ••• Buffer Memory 14 Controller DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM SMBus DRAMDRAM DRAMDRAM DRAMDRAM DRAMDRAM
CLK Source
4 FB-DIMM Architecture FB-DIMM Module Block Diagram
Pass-through Read Data & Merging Read Data (Primary Northbound) Logic (Secondary Northbound)
Serializer AMB is the new Data Bus buffer Interface DRAM Data technology for 9 to 36 DRAM server memory De-serializer DRAM C/A Copy A & Decode Logic DRAM C/A Copy B
Reference Clock Clocking DRAM Clocks
Pass- Write Data and Command Write Data and Command through (Primary Southbound) (Secondary Southbound) Logic
Advanced SPD Memory Buffer SMBus EEPROM (AMB)
5 FB-DIMM Channel Electrical Characteristics Signaling Characteristics
y Differential point-to-point, 50 Ω termination y DC coupled, current mode driver y Derived clocking ( mesochronous clocking) y Transmitter de-emphasis to reduce ISI y 3.2 Gb/s, 4.0 Gb/s, and 4.8 Gb/s data rates
Tx Rx De- emphasis
Pre - Sync driver De-skew 50 Clock 50 Recovery
PLL 50 50 PLL
Reference CLK
6 FB-DIMM Channel Electrical Characteristics Differential Signaling Parameter Description Equation VDIFFp-p Differential Peak to 2*max[VD+ - VD-] Peak Voltage VCM-AC AC Common Mode (VD+ + VD-)/2 Voltage VCM-DC DC Common Mode AVG[(VD+ + VD-)/2] Voltage
D- VD+
VCM-AC
VD-
D+
7 FB-DIMM Channel Electrical Characteristics Tx Voltage Spec
(Medium drive strength example)
Parameter Description Min Max
VTX-DIFF P-P-Min Differential Swing, 800 mv De-emphasis off
VTX-DIFF P-P-Min, Differential swing, 505 mv 566 mv -3.5 Db De-emphasis on
VTX-DIFF P-P-Min, Differential swing, 357 mv 450 mv -6.0 Db De-emphasis on
VTX-DIFF P-P-Min, Differential swing, 239 mv 301 mv -9.0 Db De-emphasis on VTX-SE Single Ended Swing 700 mv
8 FB-DIMM Channel Electrical Characteristics Tx Timing Spec
UI min spec D ata late
Tx Total VTx diff min p-p min
Data early
Median to max Jitter Median Median Jitter Jitter
UImin- Tx min- 2Txm edian to m ax jitter= 0
Transfer UI TTx-Total- TTX-Eye-Median-Max-Jitter Rate Nom min (ps) (Gb/s) (ps) (ps) 3.2 312.0 248.5 32.0 4.0 250.0 191.0 29.5 4.8 208.0 157.0 25.68 9 FB-DIMM Channel Electrical Characteristics Rx Spec
D+,D- Crossing point D+,D- Crossing point 0.40
0.35
0.30
0.25
0.20
0.15
VRx-DIFFpp-Min 0.10
0.05 1 _ d2 n_ i _ c
e 0.00 _r f f di _ d e e_r
y -0.05 e
-0.10
-0.15
-0.20
-0.25
TRx-Total-Min -0.30
-0.35
-0.40
0.0 40.0 80.0 120.0 160.0 200.0 240.0 280.0 312.5
time, psec
•Measured •Simulated
Data UI, Nom TRX-Total- TRX-EYE- VRX-DIFFp-p-MIN (mV) Rate (ps) Min (ps) Median-Max (Gb/s) Jitter (ps) 3.2 312.0 115 98.67 170 4.0 250.0 90 80.0 170 4.8 208.0 75 66.68 170
10 FB-DIMM Channel Electrical Characteristics Clock Spec y HCSL (High-Speed Current Steering Logic) clock y 0.7 v swing y 1/4th of the DDR2 DRAM frequency ( 133, 166, and 200 MHz) y Spread Spectrum Clock (SSC) with up to –0.5% down spread y Reference clock jitter specified to allow standard clock buffers.
Z0d = 100 33 Ω Z = 100 Z = 100 < 0.2” 0d 0d
< 0.2”
33 Ω < TBD” ~2.5” 0.5” DBxxx 50 Ω Clock 50 Ω AMB Driver FB-DIMM Connector
11 FB-DIMM Channel Electrical Characteristics Interconnect Modeling
y Three (3) Topologies supported y Accurately models connection from Tx pin to Rx pin y Non-Interleaved routing with no bit to bit length matching required y Differential signaling design rules
Tx Package Southbound Interconnect Network Rx Package (Memory Writes) + _ y Package Breakout +_ y Lossy Transmission Lines y Connectors y Vias y Sockets and test points
Not Equivalent due to DIMM Routing and connector location
12 FB-DIMM Channel Electrical Characteristics FB-DIMM Topologies DIMMs on Motherboard
FB-DIMM
Memory Controller AMB
Base Board
FB-DIMM Connector
13 FB-DIMM Channel Electrical Characteristics FB-DIMM Topologies DIMMs on a Riser Card
Memory Riser Board FB-DIMM Connector
Memory Controller FB-DIMM Base Board AMB
PCI Express* or pin and socket connector
*Other names and brands may be claimed as the property of others. 14 FB-DIMM Channel Electrical Characteristics FB-DIMM Topologies DIMM to DIMM FB-DIMM FB-DIMM
AMB AMB Base Board
FB- DIMM connectors
15 FB-DIMM Channel Electrical Characteristics Interconnect Components y Package Models – Return Loss (<-10 Db) y PCB Model – Impedance, delay, and loss characteristics of traces – 3 pair, non-interleaved, cross talk model – VIA Models y Connector(s) – 3D Models validated using Measurements
MC Vendor Model AMB Vendor Model W Element W Element
+_ +_
+_ Package Network Package + Network _
+_ +_ 85 ohm Strip Line 6 x 50 Ohms
6 x 50 Ohms 85 ohm 85 ohm Strip Line FBD DDR2 Microstrip Breakout Vias Connector Model
16 FB-DIMM Channel Electrical Characteristics FB-DIMM Connector y Standard DDR2 Connector w/ FB-DIMM connector spec y Characterized up to 8 GHz y Q3D Spice Model based on FOXCONN* Connector Data
• Less than 1 Db insertion loss. • Less than 3.2% Far end Cross talk • Less than 10 Db insertion loss
*Other names and brands may be claimed as the property of others. 17 FB-DIMM Channel Electrical Characteristics Channel Design Rules y Channel impedance: 85 ohms , normal distribution y Microstrip trace geometry – w=6m, s= 4x h , tolerance = +/- 20%, etc. y Strip line trace geometry – w= 5 mil, s = 3x h, tolerance = +/-15%, etc y Asymmetric strip line geometry – w = 5 mil, s = 4x h , tolerance = +/-15%, etc, etc y Loss tangent 0.017 to 0.025, ε ~ 3.4 (FR4 materials)
ef h2 s d εr2
t
εr2 w h1
18 FB-DIMM Channel Electrical Characteristics Channel Validation Tx Package
...00110011... Tx Eye + 1. Tx Stand alone Test _ S pecification
2 x 50 ohms 2. Interconnect meets Rx Eye Spec Tx Package Worse Case Rx Eye ISI Pattern + Interconnect Network _ Specification
2 x 50 ohms 3. Rx Meets BER with Minimum Eye Tx Package Rx Package
Worse Case Pattern Check ISI Pattern + Interconnect Network + _ _ or “On-die” Measurement
FB-DIMM channel requires new memory design techniques
19 FB-DIMM Physical Design Mechanical Prototype
20 FB-DIMM Physical Design Physically Compatible Approach y FB-DIMM fits into existing system infrastructure
21 FB-DIMM Physical Design DIMM Mechanical Outline AMB on front
Retention notches reduced to 2.5 mm
Nominal height
8 DRAM on front, 10 DRAM on back
New end notches
Key moved
22 FB-DIMM Physical Design Advanced Memory Buffer (AMB) y New 0.8 mm ball pitch package – 24.5 mm x 19.5 mm x 2.15 mm – (29 columns x 23 rows) – 12 corner balls = 655 balls
1234567891011121314151617181920212223242526272829
TEST2 TEST4 A GND DQ26 DQ12 VDD DQS10 DQ13 VDD DQS01# DQ10 VDD VDD VDD VDD VDD DQ52 DQS15 VDD DQ49 DQS06#VDD DQ48 DQ38 VDD
DDRCAL TEST3 TEST5 DDRCAL B DDV DQS03 DQS03# GND DQ14 DQS10# GND DQ11 DQS01 GND VDD GND VDD GND DQS15# DQ53 GND DQS06 DQ50 GND DQS13# DQS13 GND
DDRCAL DDRCAL DDRCAL C GND DQS02 DQ18 GND DQ04 DQS09# GND DQ15 DQ09 GND DQ08 GND DQS17 DQS17# GND DQ54 GND DQ55 DQ51 GND DQS07 DQ56 GND DQ46 DQS14# VDD
D DQ19 DQS02# GND DQ16 DQ24 GND DQS09 DQ07 GND DQ03 DQS00 GND DQS8# DQS8 VDD CB6 CB7 GND DQS16 DQ63 GND DQ59 DQS07# GND DQ36 DQ44 GND DQS14 DQ47 DRAM
Mechanical E DQ21 GND DQ17 DQ29 GND DQ25 DQ06 GND DQ05 DQ01 GND DQ00 CB1 GND CB2 GND CB5 DQS16# GND DQ61 DQ57 GND DQ58 DQ39 GND DQ33 DQ45 GND DQ41 TEST0 TEST1 TEST6 TEST7 Data bus Balls F GND DQ20 DQ23 GND DQ31 DQ27 GND GND DQS00# DQ02 VDD CB0 CB3 CB4 VDD DQ62 DQ60 GND GND DQ37 DQ35 GND DQS05#DQ43 GND G DQS11# DQS11 mech mech mech GND DQS12 DQS12# mech mech mech TEST RFU3 RFU4 RFU5TEST RFU6 RFU7 mech mech mech DQS04 DQS04# GND mech mech mech DQS05 DQ40
H DQ22 GND mech mech mech DQ28 DQ30 GND mech mech mech GND VDD GND VDD GND VDD GND mech mech mech GND DQ34 DQ32 mech mech mech GND DQ42
J GND CLK2 mech mech mech BA1A GND CKE1A mech mech mech VDD GND VDD GND VDD GND VDD mech mech mechRAS#B GND RFU1 mech mech mech CLK3# GND
DRAM K CLK2# CLK0 mech mech mech GND WE#A RAS#A mech mech mech GND VCC GND VCC GND VCC GND mech mech mechODT0B CS1#B GND mech mech mech CLK1# CLK3 C/A/Clk L CLK0# GND mech mech mech A0A CKE0A GND mech mech mech VCC GND CVC GND CVC GND CVC mech mech mechGND CAS#B WE#B mech mech mech GND CLK1 DRAM M ODT0A RFU0 mech mech mech CAS#A GND BA2A mech mech mech GND VCC GND VCC GND VCC GND mech mech mech CS0#B GND BA1B mech mech mech CKE0B GND
copy “A” N CS1#A CS0#A mech mech mech GND BA0A A10A mech mech mech VCC GND CVC GND CVC GND CVC mech mech mechA0B A2B GND mech mech mech BA0B BA2B C/A/Clk P A6A GND mech mech mech A2A A1A A3A mech mech mech GND VCC GND VCC GND VCC GND mech mech mechGND A4B A1B mech mech mech GND CKE1B copy “B” R GND A8A mech mech mech A11A GND A5A mech mech mech VCC GND CVC GND CVC GND CVC mech mech mechA6B GND A10B mech mech mech A3B GND
T A4A A13A mech mech mech GND A9A A7A mech mech mech GND VCC GND VCC GND VCC GND mech mech mech A11B A9B GND mech mech mech A7B A5B
U PN0 PN0# mech mech mech A15A A14A A12A mechmechmechRFU2VCCFBD GND GND GND VCCFBD RFU8 mech mech mech A8B A15B A14B SA0 SCL SDA PS8#PS8
Northbound V PN1 PN1# GND SN0 SN0#VCCFBD GNDVCCFBD GND SNCKSNCK#VCCFBD GND GND GND VCCFBD GND VCCFBD GND VCCFBDSSCKSSCK#GNDA13B A12BSA2 SA1 PS7#PS7 Channel W PN2 PN2#GND SN1 SN1#SN3#SN4#SN5#SN13#SN12#SN6#SN7#SN8#SN9#SN10#GND SS0#SS1#SS2#SS3#SS4#SS9#SS5#SS6#SS7#SS8#GND PS6#PS6 Southbound Y PN3 PN3#GND SN2 SN2#SN3 SN4 SN5 SN13SN12SN6 SN7 SN8 SN9 SN10 GND SS0 SS1 SS2 SS3 SS4 SS9 SS5 SS6 SS7 SS8 GND PS5#PS5 AA GND PN4 PN4# GND GND GND GND GND GND GND GND GND GND GNDGND GNDGND GNDGND GNDGND GNDGND GNDGND GND PS9# PS9 GND Channel VSSAPLLVCCAPLL FBDCAL AB GND RESET# PN5#PN13#PNCK#PN12# PN6# PN7# PN8# PN9# PN10#PN11#GND SN11#GND SCK PS0#PS1#PS2#PS3#PS4#PSCK#VDDSPD GND
FBDCALTEST FBDCAL AC GND PN5 PN13PNCK PN12PN6 PN7 PN8 PN9 PN10PN11 RFU9 SN11 GNDSCK# PS0 PS1 PS2 PS3 PS4PSCKGND
23 FB-DIMM Physical Design FB-DIMM Bus Topologies y Bus topologies scale to higher frequencies than the registered DIMMs – DRAM farthest from AMB gets latest clock, C/A and data
24 FB-DIMM Physical Design Thermal Solution for x8 based DIMMs y x8 DIMMs will have an AMB-only heat spreader – Attached using a spring clip, inserted into two holes y Stacked BGA and dual die x4 DIMMs under investigation
25 FB-DIMM Physical Design Raw Cards under Design y The JEDEC DIMM and Socket Task Group is standardizing these DIMMs: (R/C=Raw Card) – R/C A SR x8 9 DRAM – R/C B DR x8 18 DRAM – R/C C SR x4 18 DRAM – R/C D DR x4 36 DRAM (Stacked) y All DIMMs will fit standard MO & are designed to meet signal integrity, timing and power delivery y FB-DIMM channels will be routed to same rules DRAM interface is self-contained on industry standard DIMMs
26 FB-DIMM System Design Considerations System Design Overview y FB-DIMM uses standard baseboard technology – With 6 layers, can route 4 FB-DIMM channels – Compared to 2 “stub bus” DDR2 channels y Will support standard pitch of 0.4” – Evaluating wider pitch for stacked BGA x4 DIMMs y Three voltage regulators for DRAM and buffer – 1.8V for DDR2 DRAM – 0.9V to terminate DRAM C/A – 1.5V for AMB
27 FB-DIMM System Design Considerations Cooling y Need to meet component requirements: – Normal refresh: DRAM Tcase ≤ 85°C (tREFI=7.8us) – 2X refresh: DRAM Tcase ≤ 95°C (tREFI=3.9us) – AMB Tcase ≤ 105°C y Recommend ducted air flow at 1.5-3.0 m/s inlet air speed – A function of inlet air temperature and application y 1st DIMM on channel is highest AMB power – typ. 3.4W y Last DIMM on channel is lowest AMB power – typ. 2.4W – Secondary channel disabled y Memory Controller will throttle activity to meet Tcase – Algorithm enhanced by AMB thermal sensor
28 FB-DIMM System Design Considerations 8 DIMM Layout y Non-interleaved DIMM layout – Better for thermals Northbound y 0.4” DIMM-to-DIMM y Length matching not required
y Shortest lead-in 1.2” Southbound – Southbound channel 0 y Longest lead-in 7.3” – Northbound channel 3
Intel will present detailed design guidelines at Fall’04 IDF 29 Summary Summary y AMB is the new buffer technology for server memory y FB-DIMM channel requires new memory design techniques y DRAM interface is self-contained on industry standard DIMMs y Intel will present detailed design guidelines at Fall’04 IDF
FB-DIMM is the next server memory technology
30 Please fill out the Session Evaluation Form.
Thank You!
31
Acronyms y AMB Advanced Memory Buffer y ISI Inter-Symbol Interference y Tx Transmitter y Rx Receiver y P-to-P Point-to-point y BER Bit Error Rate y PCB Printed Circuit Board y MCH Memory Controller Hub y MO Mechanical Outline y R/C Raw Card y JEDEC JEDEC Solid State Technology Association y C/A Command/Address
33 Birds-of-a-Feather Lunch Discussions y What: Where:– Informal Level lunch discussions3 Foyer with(Not Intel in the and main industry lunch area) technical experts on next-generation technologies, latest industryArrive standards early and – specificationsseating is limited y When: Wednesday 12PM – 1:30PM 802.11* Extensible Firmware Interface (EFI ) 802.16*/ WiMax* Latin America Mobile/Data Services Server Manageability Advanced TCA* Client Manageability Advanced Switching based on PCI Express* Architecture DMTF* Server Manageability Connecting consumers in the Working Group digital home IPMI (Intelligent Platform Management Interface) Thursday 12PM – 1:30PM Enthusiast PC Ethernet in the Datacenter Trusted Computing Group for More Advanced Switching based on Secure Computing PCI Express* Architecture Extended Battery Life
34 Round Tables for Innovation Where: Level 2 – Alcoves across from Rooms 2018-2024 Immediately following Solution Keynotes What: This is your chance to focus on relevant issues, compare notes with colleagues, and gain insight from professionals who face similar challenges. When: Tuesday 17th from 4:15 to 6:00pm Topics: • Emerging worker models for the highly connected employee • Improving PC usability • IT Benchmarking Programs • Technology and Work/Life Balance • Internal IT Consulting Model • Managing wireless networks • Productivity/collaboration/eWorkforce • Capturing IT Business Value and ROI • Mobilized Software: Addressing Technology and Business Challenges Consistently • Creating Mobility Business Value in the Enterprise • Reducing Costs and Increasing Revenue with Secure WLAN Solutions • Achieving Competitive Advantages with Real-time Mobile Business Solutions • Collaborating with Mobility • Mobile Point of Care Solutions for Healthcare • RFID technologies transforming the Retail environment
35