Getting Real: Lessons in Transitioning Research Simulations into Hardware Systems Mohit Saxena, Yiying Zhang Michael M. Swift, Andrea C. Arpaci-Dusseau and Remzi H. Arpaci-Dusseau University of Wisconsin-Madison msaxena,yyzhang,swift,dusseau,remzi @cs.wisc.edu { } Abstract of changing their command interface. Most such knowl- Flash-based solid-state drives have revolutionized stor- edge has remained the intellectual property of SSD man- age with their high performance. Their sophisticated in- ufacturers [18, 26, 11, 12], who release little about the ternal mechanisms have led to a plethora of research on internal workings of their devices. This limits the oppor- how to optimize applications, file systems, and internal tunities for research innovation on new flash interfaces, SSD designs. Due to the closed nature of commercial de- on OS designs to better integrate flash in the storage hi- vices though, most research on the internals of an SSD, erarchy, and on adapting the SSD internal block manage- such as enhanced flash-translation layers, is performed ment for application requirements. using simulation or emulation. Without implementation Simulators and emulators suffer from two major in real devices, it can be difficult to judge the true benefit sources of inaccuracy. First, they are limited by the of the proposed designs. quality of performance models, which may miss impor- In this paper, we describe our efforts to implement two tant real-world effects. Second, simulators often simplify new SSD designs that change both the internal workings systems and may leave out important components, such of the device and its interface to the host operating sys- as the software stack used to access an SSD. tem. Using the OpenSSD Jasmine board, we develop We sought to validate two recently proposed SSD de- a prototype of FlashTier’s Solid State Cache (SSC) and signs by implementing them as hardware prototypes. of the Nameless Write SSD. While the flash-translation FlashTier’s Solid-State Cache (SSC) design [29] im- layer changes were straightforward, we discovered un- proves caching performance with changes to the block expected complexities in implementing extensions to the interface, flash management algorithms within the de- storage interface. vice, and the OS storage stack. Nameless Writes [33] We describe our implementation process and extract a simplifies garbage collection and improves write perfor- set of lessons applicable to other SSD prototypes. With mance by moving the address-mapping function out of our prototype we validate the performance claims of the device and into the file system, and requires changes FlashTier and show a 45-52% performance improvement to the same components as SSCs plus the file system. over caching with an SSD and a 90% reduction in erases. Therefore, these designs are ideal candidates for study- ing the disruptive nature of such systems. 1 Introduction We prototype both systems with the OpenSSD Jas- mine hardware platform [30]. The OpenSSD evaluation Due to the high performance, commercial success, board is composed of commodity SSD parts, including a and complex internal mechanisms of solid-state drives commercial flash controller, and supports standard stor- (SSDs), there has been a great deal of work on op- age interfaces (SATA). It allows the firmware to be com- timizing their use (e.g., caching [24, 27]), optimizing pletely replaced, and therefore enables the introduction their internal algorithms (e.g., garbage collection [7, 16, of new commands or changes to existing commands in 17]), and extending their interface (e.g., caching opera- addition to changes to the FTL algorithms. As a real stor- tions [29]). However, most research looking at internal age device with performance comparable to commercial SSD mechanisms relies on simulation rather than direct SSDs, it allows us to test new SSD designs with existing experimentation [2, 27, 29, 33]. file-system benchmarks and real application workloads. Thus, there is little known about real-world implemen- During prototyping, we faced several challenges not tation trade-offs relevant to SSD design, such as the cost foreseen in published work on new flash interfaces. First, 1 USENIX Association 11th USENIX Conference on File and Storage Technologies (FAST ’13) 215 Controller ARM7TDMI-S Frequency 87.5 MHz SDRAM 64 MB (4 B ECC/128 B) Frequency 175 MHz Flash 256 GB Overprovisioning 7% Type MLC async mode Packages 4 Dies/package 2 Banks/package 4 Channel Width 2 bytes Ways 2 Physical Page 8 KB (448 B spare) Physical Block 2 MB Virtual Page 32 KB Virtual Block 4 MB Table 1: OpenSSD device configuration. With our SSC prototype, we validate that FlashTier’s projected benefits, including better write performance and reliability, are indeed possible. Compared to caching with an SSD, the SSC design provides 45–52% bet- ter write performance. Our Nameless-Write prototype, demonstrates that the split-FTL approach may be a use- ful method of implementing new interface designs rely- Figure 1: OpenSSD Architecture: Major components of ing on upcalls from an SSD to host. OpenSSD platform are the Indilinx Barefoot SSD controller, internal SRAM, SDRAM, NAND flash, specialized hardware 2 Background for buffer management, flash control, and memory utility func- tions; and debugging UART/JTAG ports. We use the OpenSSD platform [30] as it is the most up-to-date open platform available today for prototyp- ing new SSD designs. It uses a commercial flash con- we found that the baseline performance of the OpenSSD troller for managing flash at speeds close to commodity board was low, and that the effect of new designs was SSDs. We prototype our own designs — FlashTier SSC drowned out by other performance problems. We intro- and Nameless Writes SSD — to verify their practicality duced a merge buffer to align writes to internal bound- and validate if they perform as we projected in simula- aries and a read buffer for efficient random reads. tion earlier. Second, we found that passing new commands from the file-system layer through the Linux storage stack and 2.1 OpenSSD Research Platform into the device firmware raised substantial engineering The OpenSSD board is designed as a platform for im- hurdles. For example, the I/O scheduler must know plementing and evaluating SSD firmware and is spon- which commands can be merged and reordered. We sored primarily by Indilinx, an SSD-controller manufac- developed novel techniques to tunnel new commands turer [30]. The board is composed of commodity SSD through the storage stack and hardware as variations of parts: an Indilinx Barefoot ARM-based SATA controller, existing commands, which limits software changes to the introduced in 2009 for second generation SSDs and still layers at the ends of the tunnel. For example, we return used in many commercial SSDs; 96 KB SRAM; 64 MB cache-miss errors in the data field of a read. DRAM for storing the flash translation mapping and for Third, we encountered practical hardware limitations SATA buffers; and 8 slots holding up to 256 GB of MLC within the firmware. The OpenSSD board lacks an ac- NAND flash. The controller runs firmware that can send cessible out-of-band (OOB) area, and has limited SRAM read/write/erase and copyback (copy data within a bank) and DRAM within the device. This made us redesign operations to the flash banks over a 16-bit I/O channel. the mechanisms and semantics for providing consistency The chips use two planes and have 8 KB physical pages. and durability guarantees in an SSC. We change the se- The device uses large 32 KB virtual pages, which im- mantics to provide consistency only when demanded by prove performance by striping data across physical pages higher level-software via explicit commands. on 2 planes on 2 chips within a flash bank. Erase blocks For the SSC design, we implemented the interface are 4 MB and composed of 128 contiguous virtual pages. functionality entirely within the firmware running on the The controller provides hardware support to acceler- OpenSSD board. For Nameless Writes, the challenges ate command processing in the form of command queues in passing data between the device and the host OS led and a buffer manager. The command queues provide a us to a split-FTL design. A minimal FTL on the device FIFO for incoming requests to decouple FTL operations exports primitive operations, while an FTL within the from receiving SATA requests. The hardware provides host OS uses these primitives to implement higher-level separate read and write command queues, into which ar- functionality. This split design simplifies the FTL imple- riving commands can be placed. The queue provides mentation and provides a convenient mechanism to work a fast path for performance-sensitive commands. Less around hardware limitations, such as limited DRAM or common commands, such as ATA flush, idle and standby fixed-function hardware. are executed on a slow path that waits for all queued 2 216 11th USENIX Conference on File and Storage Technologies (FAST ’13) USENIX Association to map disk addresses to logical flash address that the Host FTL translates to physical flash addresses. In addition, Command Event SSCs provide silent eviction with which they can evict Queue Queue clean data without notifying the host as an alternative to garbage collection. FTL FlashTier relies on a block-level cache manager to Bad Block NAND interpose on disk requests from the file system and Manager transparently shunt them to the SSC. If the SSC does Buffer Manager not contain the data, the manager sends the request to the backing disk. Compared to caching with an SSD, Host Interface Logic Flash Interface Logic FlashTier promises better write performance, because it can evict data silently instead of performing expensive DRAM garbage collection; reduced memory on the host, because the disk-block-to-flash-location mapping is stored in the Figure 2: OpenSSD Internals: Major components of OpenSSD internal design are host interface logic, flash inter- SSC; and better reliability, again by avoiding expensive face logic and flash translation layer.