The Solid State Storage Revolution

Andy Bechtolsheim Sun Microsystems

 1956: IBM introduces 350 Storage Unit, first Hard disk (5 MB)  1962: IBM introduces Model 1301 Advanced Disk File (25 MB)  1970: IBM introduces Model 3330 DAS Disk Drive (100 MB)  1980: IBM introduces Model 3380 DAS Disk Drive (1000 MB)  2004: 3.5” 7200 RPM Disks store 250 GB (250,000 MB)  2006: 50 Year Anniversary of the Hard disk drive  2008: 3.5” 7200 RPM Disks store 1 TB (1,000,000 MB)

$/GB

TIme

300

200

100

0 7200 RPM 10K RPM 15K RPM 125 IOPS 180 IOPS 250 IOPS

 Great improvements in:  Density (average 60% / year growth)  Cost per GB (tracking density)  Some improvements in:  Transfer rate (higher bit density)  Interface speed (SAS-SATA I/II/III)  No significant improvements in:  IOPS per disk (more or less same)  Reliability per disk (more or less same)

 HDD have made enormous improvements in capacity, cost per GB and reliability per GB  Very limited improvements in performance  IOPS per GB are continuously declining  Server performance is increasing 50% per year  Major CPU versus I/O performance “crisis”

 1985: Toshiba shows first Flash chip (256 Kbit)  1995: First Multi-level Flash chip (32 Mbit)  2002: Samsung delivers 1 Gbit chips (120 nm)  2005: Samsung delivers 4 Gbit chips (70 nm)  2006: Flash Market hits $12B in revenue  2008: Toshiba delivers 16 Gbit chips (56 nm)

 SLC: Single-level Flash: 1 Bit / T  Most Robust Flash Technology  500,000 Write Cycles  Ideal for enterprise applications

 MLC: Multi-level Flash: 2-3-4 Bits/T  Most Cost-effective Flash Technology  10,000 Write Cycles  Ideal for consumer applications

 Random Read Access Time: 100 us (10K IOPS)  This is per device or device channel  Typical controller has 4 channels  Typical controller read performance is 30K IOPS

 Write cycles are considerably slower  Erase cycles are painfully slow  Controller hides these with write buffering  Typical controller write performance is 10K IOPS

 FMC is Key for Reliability and Performance  Functions of FMC:  Error correction  Wear leveling  Bad Block mapping  Scrubbing  Write caching  Read caching  Significant Improvements Ahead

Flash Flash Flash Flash

uCode FMC DRAM

SAS/SATA Interface

 Flash is more than 100X faster than disk  Flash consumes less power than hard disk  Flash is more reliable than hard disk  Flash is physically smaller than hard disk

 Major issue is cost  Flash costs a lot more per GB than Disk

 Ideal use Model is to use Flash as a Cache  Most Stored data is not actively being accessed  Working set is typically a few % of total capacity  Caching the actively used data in Flash makes the entire disk storage appear as fast as Flash  Significant gain for I/O intensive workloads  No change required for the application

 ZFS already supports a DRAM memory cache  Cache efficiency is limited to size of main memory  Trivial to extend mem-cache to FLASH  Flash can be much larger than main memory  Flash cache divided into two segments:  Readzilla for Read Caching  Logzilla for Transactional Commits  Delivers significant cost-performance gains

 What is the speed limit with Flash?  Are multi-million IOPS achievable?  What storage fabrics can support this?  How much space/power/cooling will this take?

 Speed limit is HCAs, not Flash  HCAs were not designed for these I/O rates  One million IOPS achievable today  More requires next generation HCAs  Existing HDD arrays are problematic for SSDs  There are simply not enough I/O channels  SSDs significantly improve IOPS power efficiency  Almost a factor of 100X compared to hard disk

 Use Flash as a boot disk replacement  No-brainer, just a question of cost  Saves space and power  Make Flash part of the memory hierarchy  Unfortunately this is not application transparent  Flash is 1000X slower than DRAM  Replace existing disk access protocols  Memory-memory transfers more efficient than SCSI  May be required to scale to Multi-million IOPS

 Flash is ~ 100X faster than Hard disks  Flash is ~ 100X more IOPS power efficient  Flash prices are dropping about 50% / year  This makes Flash suddenly very interesting  Using Flash as cache looks very promising  Hard Disks are not going away anytime soon