Server Architecture and Kingston Memory Solutions Ingram Micro

Kingston Technology Server Architecture and Kingston Memory Solutions Ingram Micro

May 2015

Mike Mohney Senior Technology Manager, TRG

• Introduction to Kingston • Kingston Solutions and Memory Roadmap • Memory Configurations for Intel server platforms

2 Kingston Snapshot

• Founded in 1987, 27+ years • Privately Held Company • Financially Secure • For 8 years , revenue exceeding $4B • 4,000 Employees Worldwide • Worldwide Manufacturing & Testing • Focused Business: Memory Modules, Flash Products & Tech Solutions • Delivering over 1M units per day • Products distributed in 125 Countries and available in over 30,000 locations worldwide; widely accepted brand

3 Source: Kingston Technology Kingston Global Infrastructure

• Manufacturing Centers are all ISO 9001:2008 certified • 2014 worldwide capacity: Over 30 million units per month • Over 50 SMT manufacturing lines in 4 global facilities, producing both DRAM & Flash products

* * * * ** **

* 4 Kingston Technology Manufacturing Facilities Worldwide

** Panram and OSE are subcontractors to Kingston

4 Source: Kingston Technology, June 2014 Kingston Investments & Expertise

Kingston manages all six processes required to build modules, cards, and drives.

Wafer Production

Wafer Testing

Wafer Processing/Chip Production

Component Testing

Module, Card & Drive Production

Module, Card & Drive Testing

5 Strategic Alliances: Server Compatibility Check

Kingston works with many facets of technology to ensure server memory compatibility.

Industry Leading OS, PC OEM’s Motherboard Manufacturers Software, Hardware

6 Intel and Kingston

DDR3 DRAM chip validation testing on Checking early FB-DIMM prototype new Advantest tester (DDR4 capable) against specifications

Since 1998, Intel and Kingston have jointly developed memory technology validation testing processes for PC133, RDRAM, DDR, DDR2, FB-DIMM, DDR3, LRDIMM, and DDR4.

7 Kingston Performance Upgrades

• Introduction to Kingston • Kingston Solutions and Memory Roadmap • Memory Configurations for Intel server platforms

8 Reliability: The Challenges of Memory Testing

Memory Capacity Number of Cells

1GB Over 8.5 billion cells

2GB Over 17 billion cells

4GB Over 34 billion cells

8GB Over 68 billion cells

16GB Over 136 billion cells

Kingston Quality Standard: Zero bad cells 1 bad cell = defective module

9 Semiconductor Reliability

This curve of semiconductor failures over time is called a ‘bathtub’ curve

Early-Life Useful Life Of Semiconductor End-of-Life Failures Failures

A typical useful life would be 20 years or more

(in units)(in

Failures Failures

Field

Of Failure curve for lower-quality

semiconductor products Number Time

Ship out from Typically the first few Factory to user months of usage

10 Kingston Production Testing and Dynamic Server Burn-In

100% Testing • No spot testing! Kingston memory is 100% tested prior to leaving our factories, as we have always done Dynamic Server Memory Burn-In • Shrinking DRAM lithography increases sensitivity to temperature, forcing higher error rates and exposing weak memory cells • Kingston’s advanced burn-in process screens these by simulating 3 months of heavy server use at higher temperatures under full system environment and frequency stress Proprietary Production Testers • Kingston’s test engineers designed the first automated production testers in the memory industry • New patented automated testers build upon our test engineering legacy to meet the more stringent demands of evolving memory technology 35 Quality Control Checkpoints • Kingston uses AQL sampling methods to re-inspect and re-test every lot to ensure our quality standards are being met Kingston holds 29 patents related to memory testing

11 Memory Roadmap (1987 – 2017) DDR4-2400 2007- 2010- 2015 2016 2017 2004- 2014 2001- 2006 2009 2013 1997- 2005 1987- 2000 2003 1999 1990 DDR4-2666 DDR4-2933 DDR3L 1.35V DDR200 DDR2-800 DDR4-2133 PC66 DDR3-1600 FAST DDR266 FB-DIMMs DDR3L-1600 PAGE PC100 RDRAM DDR333 DDR2-400 DDR3-800 DDR3-1866 MODE PC133 PC600 DDR400 DDR2-533 DDR3-1066 LRDIMM EDO PC700 RDRAM DDR2-667 DDR3-1333 PC800 PC1066

Technology Data Rate Module Classification Peak Bandwidth 800 DDR3-800 / PC3-6400 6400 MB/s or 6.4 GB/s DDR3-1066 / PC3-8500 1066 8500 MB/s or 8.5 GB/s DDR3L-1066 / PC3L-8500 DDR3 (1.5V) DDR3-1333 / PC3-10600 1333 10600 MB/s or 10.6 GB/s DDR3L (1.35V) DDR3L-1333 / PC3L-10600 DDR3U (1.25V) DDR3-1600 / PC3-12800 1600 12800 MB/s or 12.8 GB/s DDR3L-1600 / PC3L-12800 1866 DDR3-1866 / PC3-14900 14900 MB/s or 14.9 GB/s 2133 DDR4-2133 / PC4-2133 17000 MB/s or 17 GB/s 2400 DDR4-2400 / PC4-2400 19200 MB/s or 19.2 GB/s DDR4 (1.2V) 2666 DDR4-2666 / PC4-2666 21300 MB/s or 21.3 GB/s 2933 DDR4-2933 / PC4-2933 23400 MB/s or 23.4 GB/s

1 2 New Module Key Location

Form Factors 288-pin DIMM 260-pin SODIMM

Performance 1600MT/s, 1866MT/s, 2133MT/s, 2400MT/s, 2666MT/s, 2933MT/s 4000MT/s (possible extension) 4266MT/s (possible extension)

Power 1.2V with improved refreshes, ½K page for x4 12V for NVDIMM (4-pins)

DRAM Densities/DIMM Capacities Slightly thicker for 4Gb, 8Gb, 16Gb (x4, x8, x16) improved performance Future DIMM capacities of 64GB, 128GB, 256GB

New Features 3D chip stacking support (up to 8 stacked dies) Point-to-Point Memory Channel addressing More stable than DDR3 with improved error correction code

Curved edge for better socket connection

13 “Low Voltage” (LV)

DDR3L = Low Voltage 1.35V JEDEC (industry spec) • Modules are classified as PC3L-xxx Reducing memory power consumption 10% by dropping module voltage requirement, lowering energy costs to data centers DDR3U (1.25V)

Kingston DDR3L modules are dual-voltage supporting 1.35V / 1.5V Ultra Low Voltage

Supported on current and legacy server platforms • Intel SNB/IVB/HSW (Xeon E3+ Series) JEDEC Spec for Mobile, Server • Intel WSM-EP/SNB-EP/IVB-EP/HSW-EP (Xeon 5600/E5 v1/v2 Series) • Intel WSM-EX/IVB-EX (Xeon E7/v2 Series) Low Voltage spec • AMD Magny-Cours+ (Opteron 6100/6200/6300 Series) for DDR4 deemed • AMD Lisbon+ (Opteron 4100/4200/4300 Series) not possible by Supported on mobile/desktop platforms from Intel and AMD DRAM semiconductors • Skylake (DDR3L Only), Broadwell, Haswell, & Ivybridge (Core i5 / Core i7) • AMD A-Series

14 Load-Reduced DIMMs for Servers

JEDEC Industry Standard Specification LRDIMMs overcome the impact of Ranks (Loads) upon memory speed • Parallel memory bus impacted by memory module Ranks • Enable greater memory capacity and speeds while reducing “load” • Require special buffer chips Eight-Rank modules now possible with less loading on memory bus • Will enable faster 32GB, 64GB, 128GB LRDIMMs Supported on 2012+ Intel and AMD servers Included in DDR4 spec with new, more efficient design – Q1 2015 Kingston launched Q1 2013 DDR3 LRDIMMs KVR13LL9Q4/32 KVR16LL11Q4/32 KVR18L13Q4/32

DDR4 LRDIMMs KVR21L15Q4/32 KVR21L15Q4/64 Coming soon

15 Source: Intel IDF, September 2011 Load-Reduced DIMMs for DDR4 – New Design, Reduced Latency

16 Source IDT: www.idt.com/document/whp/ddr4-lrdimms-both-memory-capacity-and-speed Non-Volatile DIMM (NVDIMM)

What Is It? A non-volatile memory module directly accessed by the memory controller (approximately 10x faster than PCIe interface). Silicon based memory technology is reaching its perfomance and efficiently limit.

Types of NVDIMM standards NVDIMM-F – An all FLASH memory module accessible by the memory controller. NVDIMM-N – A DRAM memory module made persistent by the use of NAND flash silicon. NVDIMM-N2 – Similar to NVDIMM-N but with system accessible Flash. NVDIMM-M – MRAM based memory module accessible by the memory controller. NVDIMM-x – New technology. PRAM? RRAM? HMC?

Viking ArxCis-NV DDR3 NVDIMM Diablo/SanDisk ULLtra DIMM SK Hynix 16GB DDR4 NVDIMM 17 Source: Google Images Kingston Memory Product Lines ValueRAM System Specific o o Generic / JEDEC Specification Also known as “Branded” o Guaranteed compatibility o Ideal for system builders or small o Ideal for users who want to upgrade datacenters OEM machines

OEM / Private Label HyperX o Customized in box solutions o Enthusiast gaming market

o Contracted supply agreements o Overclocking

o Brand focused

18 System Specific Memory

Guaranteed Compatible • Testing conducted in the system in which the memory will be installed Importance of Branded Memory • Different chips provide varying performance levels • Chip Compatibility can vary • Possible constraints in module height or width • Presence Detect Configuration

19 ValueRAM Product Decoder • Kingston ValueRAM is built to industry standard specifications outlined by J.E.D.E.C., the Joint Electron Device Engineering Council, on which Kingston holds a seat. Intel, AMD, and other computing chipset architects adhere to these standards for industry agreement and cross-platform compatibility. • J.E.D.E.C. provides the specifications for Pin Count, Form Factor, Speed, Power, and Memory Technology Types

20 ValueRAM Server Focused Programs

Intel Certified intended specifically for customers who are using Intel branded server/workstation motherboards or systems • Qualified and supported by Intel • Controlled builds (BOMs) • PCNs with 45-90 Day Notice Server Premier intended for server builders with fully controlled BOM requirements (PCB, chip brand/die revision, etc) regardless of brand of motherboard used • Fully BOM controlled • PCNs with 45-90 Day Notice • Easily manage what is ordered • No price premium

21 ValueRAM Qualifications

In addition to the strategic partnerships Kingston has with ODMs for submitting memory for qualification, we also work with independent memory test labs AVL and CMTL Labs for specific platform, motherboard, or system qualification. Test certificates are available on our website.

http://www.kingston.com/us/memory/valueram/server

Example:

22 Module Types For Server Platforms

Registered DIMM Load Reduced DIMM ECC Unbuffered DIMM

288-pin DIMM

Registered DIMM Load Reduced DIMM ECC Unbuffered DIMM

240-pin DIMM

• Optimizing Memory: • DDR4 is NOT compatible with DDR3 based systems (module will not fit) • Install memory in kits of identical modules according to memory architecture • Configure all processors the same way for most bandwidth • Do Not Mix Module Types within a server. • Mixing DDR3 (1.5V) and DDR3L (1.35V) will default the memory to 1.5V.

23 DRAM Chip Types Used in Servers

Multi Bit Error Detection and Registered DIMMs Correction x4 LRDIMMs

1 2 3 4

Registered DIMMs LRDIMMs x8 Single Bit Error Unbuffered ECC DIMMs Detection and Unbuffered DIMMs Correction 1 2 3 4 5 6 7 8 SODIMMs

24 64-bit Memory Loads (aka Ranks)

• Ranks are virtual modules on a single DIMM, Single Rank (1R) designed to save space on the motherboard • Only one Rank is active at a time, the other ranks on the DIMM go into standby saving power 64bits • 2R and 4R DIMMs consume less power than 1R DIMMs

Dual Rank (2R) • Memory lanes on the motherboard become congested with more memory loads, so server systems are limited by the number of ranks 64bits 64bits that can be used, as the memory slows down with more ranks populated, just line cars in Quad Rank (4R) heavy traffic

64bits 64bits 64bits 64bits

25 Why Add Channels and Increase The Memory Speed?

DDR3 & DDR4 Aggregate Server Memory Bandwidth 120

100

GB/s 60 DDR3-800 DDR3-1066 40 DDR3-1333 DDR3-1600 20 DDR3-1866 0 DDR4-2133 Single Channel Dual Channel Quad Channel (1 DIMM per CPU) (2 DIMMs per CPU) (4 DIMMs per CPU) 2 x 16GB = 32GB 4 x 16GB = 64GB 8 x 16GB = 128GB

26 Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel DDR3 and DDR4 platforms Memory Configuration Options by Processor

Memory Supported Memory Configuration Xeon 5500 Series DDR3-1333 Triple-Channel = K3 kits Xeon 5600 Series DDR3-1333 / DDR3L Triple-Channel = K3 kits Quad-channel = K4 kits Xeon E5 Series DDR3-1600 / DDR3L-1333 Triple-channel = K3 kits Quad-channel = K4 kits Xeon E5 v2 Series DDR3-1866 / DDR3L-1600 Triple-channel = K3 kits Xeon E5 v3 Series DDR4-2133 Quad-channel = K4 kits DDR3-1066 Xeon 6500 / 7500 / E7 Series Dual-channel = K2 / K4 kits DDR3L-1066 on E7 Series Xeon E7 v2 Series DDR3-1600 / DDR3L-1600 Quad-channel = K4 kits Xeon W3500 / 3600 / E3 DDR3-1066 / DDR3L (W3600) Dual-channel = K2 / K4 kits Xeon E3 v2 / v3 Series DDR3/L-1600 / DDR3-1866 Opteron 8400 / 2400 Series DDR2-800 Dual-channel = K2 kits Opteron 6100 - 6300 Series DDR3 / DDR3L 1333-1866* Quad-channel = K4 kits Opteron 4100 - 4300 Series DDR3 / DDR3L 1333-1866* Dual-channel = K2 kits * Subject to motherboard compatibility

27 Kingston Performance Upgrades

• Introduction to Kingston • Kingston Solutions and Memory Roadmap • Memory Configurations for Intel server platforms

28 Microarchitecture Roadmap

1600 1866 2133 2400

29 Intel Xeon E5 / E5 v2 / E5 v3 “EP” Series Server Memory Configuration

HP ProLiant Gen 8 / Gen 9 ® Cisco UCS M3 ®

IBM System xSeries M4 ® Dell PowerEdge ®

30 Picture sources: Google images Intel Xeon E5 Series Summary

31 Source: Intel public marketing Intel Xeon E5-2600/4600 v1/v2/v3 Architecture Quad Channel – up to 12 DIMMs per CPU (24 per 2-CPU System, 48 per 4-CPU System)

32 Intel Xeon E5-2400 v1/v2 Architecture Triple Channel – up to 6 DIMMs per CPU (12 per 2-CPU System)

33 Haswell-EP Intel Xeon E5-2600 v3 DDR4 Memory Speeds 1.2V

1 DIMM per 2 DIMMs per 3 DIMMs per Module Type Channel Channel Channel (No Mixing in a Server) 1 DPC 2 DPC 3DPC Registered DIMM (RDIMM) 2133 1866 1600 Load Reduced DIMM (LRDIMM) 2133 2133 1600

2133 Processor Model Determines Maximum Memory Speed

1866

1600

34 Optimizing Intel Xeon E5 v3 Servers 16 x KVR21R15S4/8 128GB 8GB DDR4-2133 RDIMMs 1Rx4 1.2v DDR4 Ch 3 Ch 4 Ch 2 Ch 1 Ch 3 Ch 4 Ch 2 Ch 1

Server 3DPC 1DPC 1DPC 3DPC 3DPC 1DPC 1DPC 3DPC 1 2 3 1 2 3 3 2 1 3 2 1 1 2 3 1 2 3 3 2 1 3 2 1 Intel S2600WT

Unbalanced E5-2650v3 E5-2650v3 Configuration (128GB) CPU1 CPU2 Memory Bandwidth 67.464 GB/s

31.41% Ch 3 Ch 4 Ch 2 Ch 1 Ch 3 Ch 4 Ch 2 Ch 1 (30.89 GB/s) 2DPC 2DPC 2DPC 2DPC 2DPC 2DPC 2DPC 2DPC 1 2 3 1 2 3 3 2 1 3 2 1 1 2 3 1 2 3 3 2 1 3 2 1

E5-2650v3 E5-2650v3

Balanced Configuration CPU1 CPU2 (128GB)

2 DIMMs Per Channel

Memory Bandwidth 98.357 GB/s

35 Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel S2600WT Performance Gains for Intel DDR4 Servers

Bandwidth comparison using 32GB DDR3 and DDR4 Load Reduced DIMMs

2200 2133 2133

2000 13%

1866 25% 1800 1600 1600 1600

1400

Xeon E5 v3 35% Memory Speed Memory 1200 Xeon E5 v2 1066 1000 1 DPC 2 DPC 3 DPC 256GB 512GB 768GB

36 Source: http://www.intel.com/performance Intel Xeon E5 v2 Processors

Processor Model Determines Maximum Memory Speed

37 http://www.intel.com/content/www/us/en/processors/xeon/xeon-e5-brief.html Ivybridge-EP Intel Xeon E5-2600 v2/4600 v2 DDR3 RDIMM Memory Speeds 1.5V

1 DIMM per 2 DIMMs per 3 DIMMs per Module Ranks Channel Channel Channel Module Type System 1R/2R=Single/Dual (No Mixing in a Server) Configuration 4R = Quad 1 DPC 2 DPC 3DPC 1 Socket Per 1R/2R 1866 Registered Channel 4R 1066 DIMM 2 or 3 Sockets 1R/2R 1866 1600 1066 (RDIMM) Per Channel 4R 1066 800 NO QR 1.35V

1 DIMM per 2 DIMMs per 3 DIMMs per Module Ranks Channel Channel Channel Module Type System 1R/2R=Single/Dual (No Mixing in a Server) Configuration 4R = Quad 1 DPC 2 DPC 3DPC 1DPC 2DPC 3DPC 1 Socket Per 1R/2R 1600 Registered Channel 4R 800 DIMM 2 or 3 Sockets 1R/2R 1333 1333 800 (RDIMM) Per Channel 4R 800 800 NO QR

38 Ivybridge-EP Intel Xeon E5-2600 v2/4600 v2 DDR3 LRDIMM & ECC UDIMM Memory Speeds

1.5V / 1.35V

1 DIMM per 2 DIMMs per 3 DIMMs per Module Type Module Ranks Channel Channel Channel (No Mixing in a Server) 4R = Quad 1 DPC 2 DPC 3DPC Load Reduced 4R at 1.5V 1866 1600 1066 DIMM 4R at 1.35V 1600 1600 1066 (LRDIMM) 1DPC 2DPC 3DPC 1.5V 1.35V

1 DIMM per 2 DIMMs 3 DIMMs 1 DIMM per 2 DIMMs per 3 DIMMs per Module Type Channel per Channel per Channel Channel Channel Channel (No Mixing in a Server) System Configuration 1 DPC 2 DPC 3DPC 1 DPC 2 DPC 3DPC ECC 1 Socket Per 1866 1600 Unbuffered Channel DIMM 2 or 3 Sockets 1866 1600 NO 1333 1333 NO (ECC UDIMM) Per Channel UDIMM UDIMM

39 16 x KVR16LR11D4/16 Optimizing Intel Xeon E5 v1/v2 Servers 256GB 16GB DDR3-1600 RDIMMs 2Rx4 1.5v

Unbalanced

34.88%

Balanced

40 Benchmark by SiSoftware SANDRA 2013.05.19.44 on Intel S2600GZ Kingston Summary

• Top buyer of memory chips/only the best chips • Ranked #1 in DRAM • Ranked #1 in USB drives • Listed on America’s largest private companies • Worldwide presence/financially solid • Channel leadership – OEM, channel, Retail, eTail • Established relationships with technology leaders • Value-Add to Flash products • Major investments in technology • Most extensive testing in the industry • Guaranteed compatibility/certifications • Managing the quality process – from wafer to module

41 Questions?

42 43