Evolution of High-Speed Server and Computing Interfaces
Senior Project Manager / Keysight Technologies
Francis Liu & Jacky Yu Digitizing Detection PAM4 112 PCIe6 (56Gb) 56
Thunderbolt 40G 40
Lane rate (Gb) PCIe5 (32Gb)
25 Thunderbolt 2.0 (20G) USB 4.0 (20 Gb) PCIe4 (16Gb) USB 3.1 (10Gb) 10
PCIe3 (8Gb) 8
Year 2015 2016 2017 2018 2019 2020 2021
Keysight World 2019 2 TRUST OUR EXPERTISE
Memory DisplayPort USB Computer Optical Comp. Optical WAN HDMI 400G MIPI Perry Brian Jit Rick Greg Stefan Brian Steve Roland Keller Fetz Lim Eads LeCheminant Loeffler Fetz Sekel Scherzinger
Board of Contributor Board of TSG Member Directors Board Contributor Contributor Directors Contributor UniPro Vice JEDEC Alumni, USB-IF IEEE, Contributor Contributor PCI-SIG IEEE, OIF, Chair VESA Phy OIF- CEI, OIF, ITU, IEC HDMI 64G FC MIPI Alliance Compliance Sub Group Thunderbolt T11 FC Contributor Thunderbolt Chair Compliance CCIX GenZ
DDR DP 1.3 USB PCIe G3, G4, PAM4 (Opt), PAM4 (Opt), IEEE802.3bs, D/M/C–PHY, HDMI 2.0 UFS Type C 2.0, 3.0, 3.1 PAM4 (RT) CEI 3.1 CEI 3.1 IEEE802.3by UniPro,TBT
Keysight World 2019 3 • DDR / LPDDR are moving to DDR / LPDDR5 • PCI Express (PCIe) is moving to PCIe Gen5 • Emergence of new computing standards (CCIX, GenZ, OpenCAPI, NVLink) • Possible convergence of Type-C related buses
Keysight World 2019 4 PCI Express / CCIX offer a choice to designers for CPU to accelerator communication CPU CPU PCI & CCIX
CPU CPU NVLINK replaces PCI Express for GPU to GPU communication running at >25 DDR Gbps GPU GPU GPU GPU GenZ PCI Express / GenZ battle for DDR communications between memory and GPU GPU GPU GPU CPU
DDR Accel Accel Accel Accel erator erator erator erator DDR4 is replaced by DDR5 for memory. GenZ LPDDR4 gives way to LPDDR5
DDR
OPENCAPI and other technologies push DDR NIC NIC NIC NIC for the fabric of the server
GenZ Intel announces new CXL open bus for SAS NVMe other other coherency
Keysight World 2019 5 PCI Express Gen5 – Moving to 32 Gbps
6 • High end networking • 400Gb Ethernet • Dual 200Gb/s InfiniBand
• Storage Networking • NVM Express (NVMe) • Big Data
• Increased IC I/O Speeds 400G Adoption predicted to • Co-Processors (FPGA, GPU) be 50% by mid/late 2020 • ASIC • IP • Artificial Intelligence Engines
7 • PCIe 5.0 = 32 Gb/s Raw Bit Link BW BW/Lane Total x16 Bi- Rate/Lane Directional • Required for 400Gb Ethernet Bandwidth • This equates to 50GB bidirectionally • 16 lanes gives up to 64GB/s PCIe 1.x 2.5GT/s 2Gb/s 250 MB/s 8GB/s • Total full duplex BW = 128 GB/s PCIe 2.x 5.0GT/s 4Gb/s 500 MB/s 16GB/s
• CEM connector for PCIe 5 is PCIe 3.x 8.0GT/s 8Gb/s ~1GB/s ~32GB/s planned to be backward compatible PCIe 4.x 16.0GT/s 16Gb/s ~2GB/s ~64GB/s with earlier PCIe technologies. PCIe 5.x 32.0GT/s 32 Gb/s ~4GB/s ~128 GB/s • Tentative schedule for spec release in 2019
Keysight World 2019 8 PCISIG Board of Directors
KEYSIGHT, Intel, AMD, IBM, Synopsys, Qualcomm, Dell, HP, NVIDIA
Card Electrical Work Protocol Work Serial Enabling Electromechanical Group Group Work Group Work Group
Deliverables: Electrical Spec Protocol Spec CEM Spec Test Specification Group Chairs: AMD, Intel AMD, Intel Intel & Plugfests Intel, Synopsys
PCI Express 5.0
Keysight World 2019 9 Test Chip Data
Rev 0.3 Rev 0.5 Rev 0.7 Rev 0.9
Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4
2017 2018 2019 2020 2021
•Draft 0.3 June 2017 •Draft 0.5 December 2017 •Draft 0.7 May 2017 •Draft 0.9 Stretch Goal end of 2018 •Final 1.0 “first half of 2019”
Keysight World 2019 10 SELECT THE SPECIFICATIONS THAT RELATE TO YOUR NEED
Base Specification Card Electromechanical (CEM) Spec Phy Test Specification • Contains all the • Applies to Add-In Cards and • Defines compliance system knowledge Mother Boards tests of CEM spec in • Can directly be • Mitigates card manufacturer’s detail applied to Chip need to study the base Test specification • Increases reproducibility through PCI-SIG supplied test tools CBB PCIe 5.0 BASE and CLB (compliance base and Currently in review v0.9 load board)
Keysight World 2019 11 GOALS
• BER target is 10e-12 • TX Presets P0-P10 to remain the same • Backward compatibility with previous PCIe Gen1/2/3/4 • Same approach for TX and RX testing used for Gen4 • Similar method for TX testing via de-embedding of breakout board traces • Similar method for calibrating the eye width and eye height as used with PCIe 4.0 (ISI based, fixed RJ) • Same TX Voltage and Jitter parameters as Gen4
Keysight World 2019 12 RETIMER REQUIRED WHEN LOSS EXCEEDS - 36DB OR >1 CONNECTOR
• Estimated allowable loss: ~= -36dB @ 16GHz • Root complex pkg loss allowance ~= -9dB @ 16GHz • Add-in Card pkg loss allowance ~= -4dB @ 16 GHz • Total AIC loss budget estimate = ~9dB @ 16GHz • PCIe 5.0 CEM Connector loss budget ~= 1.5dB @ 16 GHz Keysight World 2019 13 15MV EYE HEIGHT POST EQ (0.3UI EYE WIDTH) TARGET
Heavily dependent upon physical ISI channel
Constrained Impairments
Very tight eye height margin. 15
Keysight World 2019 14 TARGET TEST CHANNEL BOARD W/ NEW CEM SMT CONNECTOR
PCIe 5.0 Variable ISI PCIe 5.0 Variable ISI BASE BASE Spec Riser Card Spec Channel Board w/ MMPX connectors
PCIe 5.0 Improved CEM Edge Fingers
Keysight M8000 Series JBERT
PCIe 5.0 SMT CEM connector
Keysight 50GHz UXR Scope
Keysight World 2019 15 MASTER YOUR BEST DESIGN ▪ Supports PCIe 5.0 BASE TX Testing at 32GT/s as well as 2.5G, 5G, 8G and 16GT/s (v0.9 BASE) ▪ Supports PCIe 5.0 Reference Clock tests (2.5G, 5G, 8G, 16G) ▪ Will Support CEM tests for endpoints and root complexes (2.5G, 5G, 8G, 16G, 32G). ▪ Integrated De-embedding of break-out channel with optional InfiniiSim ▪ Automated DUT control using an 81150/60A Pulse Generator ARB. ▪ Enhanced Switch Matrix supporting arbitrary lane mapping ▪ Minimum oscilloscope BW: 50GHz
Keysight World 2019 16 1 6 5.0 BASE SPEC TESTS
PCIe 5.0 BASE Spec Tests
17 Keysight World 2019 SETUP USING M8000 SERIES 64G HIGH - PERFORMANCE BERT
New Interference M8046A with integrated CDR option 0A4 / 0A5 and Source module equalization with new option 0S2 SKP OS Filtering PCIe M8054A for DM- 8G/16G/32G and CCIX 20G/25G and option 0S1 SI & CM-SI Interactive Link Training PCIe 8G/16G/32G TP2P
New matched Substitute PCIe 5 broadband coupler pair Base Channel M8045A-803 for DM-SI Boards by new ISI & CM-SI traces M8049A-003 & M8049A-001
N9398F DC blocks TP3 used coupler outputs M8049A-001 offers traces from approximately 1.5dB to 3.5dB in 0.5dB steps.
M8049A-003 offers traces from 10dB to 22dB in 2dB steps, IL measured @ 16GHz
Keysight World 2019 18 SETUP USING M8040A + M8054A INTERFERENCE SOURCE Simulate SAS-4 test channel by new ISI traces M8049A-001, -002 and -003
M8049A-001 offers traces from approximately 1.5dB to 3.5dB in 0.5dB steps.
Matched coupler pair M8045A-803 for M8049A-002 offers traces from CM-RI approximately 3dB to 9dB in 1dB N9398F DC block steps. DM-RI New Interference Source module M8054A for CM-RI M8049A-003 offers traces from DM-RI & CM-RI approximately 10dB to 22dB in 2dB N9398F DC blocks used at interference ISI steps. source outputs trace Matched IL @ 16GHz directional coupler pair M8045A-802 for DM-RI for ICN To support SAS-4 RX testing: • M8040A system with 1 pattern generators and one error detector 2nd data lane sends 1010 as clk reference for M8040A • All jitter sources • Multi-tap de-emphasis is built-in • CDR as well as CTLE are built-in • AWG used as common mode random interference source • AWG used as differential mode random interference source • SAS RX Calibration and Test SW – N5991SA4A (June/July 2019)
DUT
Keysight World 2019 19 Request TxEQ setting
Algorithm for determining De-Emphasis equalization and Controller de-emphasis
Means for measuring signal
quality Control • Rx LinkEQ Test Control • No Jitter Tol. Test
3-Tap • Initial Tx EQ Test CDR Equalization De-Emphasis • Tx LinkEQ Test • Preset Test 2 Tap DFE 1 order CTLE Receiver Transmitter
Keysight World 2019 20 CHALLENGE IN CEM LONG TRACE The Key with Error Free: • No additional loss with integrated CDR
40mV High sensitivity
Tx signal after > 10 inches
5.00
0.00 PCIe3.0, -6dB PCIe3.0, -9dB -5.00 PCIe3.0, -12dB PCIe4.0, -6dB -10.00 PCIe4.0, -9dB PCIe4.0, -12dB -15.00 USB3.0
-20.00dB / H(s) Magnitude USB 3.1, ADC 0dB USB 3.1, ADC -3dB -25.00 USB 3.1, ADC -6dB 1.0E+06 1.0E+08 1.0E+10 Frequency / Hz Keysight World 2019 21 DDR5 - Bring on Receiver Testing for DRAM
22 Signal Specification “Light Speed” DDR6? • Complex modulation?
IQ Threshold “Hyper Speed” - No visible eye DDR5 • Pre/De-Emphasis, Impulse response
Impulse Response Threshold “Serial Speed” - Clock often embedded DDR4 • Eye Diagrams/Masks • Error rates, Jitter/Noise Jitter & Noise Threshold “High Speed” - Transmission lines DDR3 • Timing • Impedance / load curves DDR2 DDR
Signal Integrity Threshold “Low Speed” - Auto place and route • Fanout Static RAM • Capacitance
Keysight World 2019 23 RULES ARE CHANGING WITH EVERY GENERATION - DDR5 IS COMING
DDR1/2/3 DDR4 DDR5
High-Speed Serial Hyper Digital Speed Speed
Keysight World 2019 24 Logic / Protocol
Power Integrity Probing
Memory Measurement Science Test
PMIC Rx Test RAM RAM RAM RAM
RC RC Fixtures D D
D D D B B B
Modeling and Tx Test Simulation
25 Keysight World 2019 DIMM Memory Controller Package
Die Tx
RX
VGA DRAM Core TX Channel Br.O. Pckg DFE FF
Ck Tree TX Channel Br.O. Pckg Rx
Equalizer Host DIMM Pad Rx Data Package Output
Traditional DDR Specification
Traditional Measurement Indirect Measurement or Simulation
Keysight World 2019 26 • Characterization of CLK, DQS, DQ • Mismatch between DQS and DQ (tDQS2DQ) • Rj/Dj jitter separation to allow designer to identify design issues • Rj – source of noise • Dj – source of crosstalk and duty cycle distortion
• DQ stressed eye test • Eye height and eye width measurement • DFE for higher data rates • Account for instrument noise
Keysight World 2019 27 Command protocol decode with mixed signal oscilloscopes • Use command truth table to decode read and write commands • Pros: More robust • Cons: Requires additional signals. More signals -> More loading
Read or write only control • System designer has control over data transition type through SOC, FPGA. • Pros: Eliminates the need for read/write separation • Cons: Not an option for system integrators
Keysight World 2019 28 VIRTUAL PROBING METHOD AT RX EQ
Package Model Rx Model DRAM Ball Scope display
Package
Die Tx
DRAM Core Pckg VGA DFE FF
Pckg Rx Ck Tree
Keysight World 2019 29 Higher speeds require smaller, higher density probing with lower loading
Reduced loading DIMM interposer for logic analysis
Via back side probing (DDR4) BGA interposer
Detailed simulation modeling correlated to physical prototypes
Keysight World 2019 30 • Full rate clock • No Rx PLL needed
• Wide, single ended • Xtalk dominated vs. loss dominated • Increased emphasis on multi-channel Rx test impairments
• Bidirectional • Extra parasitic loading when Rx and Tx share pins • There is no clean way to terminate the bus • Reflections due to ODT changes, adjacent rd/wr bursts
• Bursty • ISI impact differs on preamble, first bits, rest of burst
• Rx must work using “slow” DRAM process • Full rate loopback impossible • “Dumber” Rx increases test complexity
Keysight World 2019 31 RX TESTING AND EQUALIZATION
“Stressed Eye” Generator / DUT (DRAM, Error Checker Precisely corrupted signal Buffer, Register) RJ- SJ Fixture source source Control Control
Victim pattern Cal. Breakout DQ, C/A, RX generator channel channel Ctl Cal. Breakout Agressor signal channel channel generator
Loopback Rx sample, Error Analyzer Fixture counter (Mux, Counter, CRC check)
• Challenge the DRAM with a “stressed” signal / known pattern New to • Compare expected vs. actual DRAM • Repeat 3e10 to 3e16 times (depending on BER goal) Keysight World 2019 32 CREATE A WELL CONTROLLED TEST SETUP
Repeatable procedures
Calibrated test fixtures Minimal noise
High quality stimulus
Keysight World 2019 33 AUTOMATED DEVICE CHARACTERIZATION AND GOLDEN CHANNEL
• Channel Test Card “CTC2” Socket for plugging in DIMM test card for calibration, DIMM module to be tested or device test board. SMP connectors to provide access to CA/CTRL, strobe, and data • Golden Channel Modeling Board (or ISI Board) Contains different length standard compliant channels CTC2
RC • Device Test Card D Plugs into CTC and can be used by DRAM/memory/register RAM manufacturers to test their chips
D B
• Fully Passive DIMM Test Card PMIC RAM RAM RAM RAM
RC RC D D D D D Plugs into CTC. Signal breakout board for clock, CA, CTRL, DIMM Test Card B B B strobe and data. Used for signal inspection and Rx stress signal Potential calibration. Signals are routed to SMP connectors. Device Test Cards
Keysight World 2019 34