Linux on Flash

Total Page：16

File Type：pdf, Size：1020Kb

Linux on Flash The Benefits & Pitfalls of Flash Memory in Open Source Environments Thom Denholm - Datalight Flash Memory Summit 2011 Santa Clara, CA 1 Agenda . Growth of Linux and Flash Memory . Options for Managing Flash . Factors to Consider • System Boot vector • Discard Interface • Design Upgrades • Support Growth of Linux 45% 40% 35% 30% 25% 20% 15% 10% 2010 5% 0% 2011 Source: EE Times Embedded Market Study 2011 Growth of Flash Memory . 2011 - $22 billion; 19.3 billion gigabytes Challenges of Working with Flash on Linux . Flash must be “managed” . Driver required for more than basic functionality . Flash File System (YAFFS, JFFS2, UBIFS) Flash Management ECC Block Use NAND Software Driver EZ-NAND OS e•MMC Software alternatives Linux Kernel File System Cache Software Solution Configuration NAND Registers Erase Units to Disk Sectors Physical Erase Block Logical File Block Logical Erase Block File System Block Pitfalls . Write Amplification . ECC - Bad Blocks . Wear Leveling Wear Leveling Effectiveness – Dynamic Only Uneven usage! Only dynamic data is wear leveled Underused Optimal Overused Wear Leveling Effectiveness – Complete Nearly uniform wear across static & dynamic data Underused Optimal Overused FACTORS TO CONSIDER Boot Cycle uboot mtd Full Flash Management Discard interface . File system delete . Hardware notified . Saves flash life, wear leveling Testing 350000 300000 Write 131072 bytes 250000 ext3 ext4 Nitro 200000 ubifs jffs2 150000 Rewrite ext3 ext4 100000 Nitro ubifs 50000 jffs2 0 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 File System Connection Linux Kernel Flash File System NAND File System Connection Linux Kernel New File System e•MMC Design Upgrades . Flash Part End-of-Life . Kernel updates Support . Evaluation . Field Failures . Flash Image Tools.

Copy Link

Recommended publications

Development of a Verified Flash File System ⋆
Development of a Verified Flash File System ? Gerhard Schellhorn, Gidon Ernst, JörgPfähler,Dominik Haneberg, and Wolfgang Reif Institute for Software & Systems Engineering University of Augsburg, Germany fschellhorn,ernst,joerg.pfaehler,haneberg,reifg @informatik.uni-augsburg.de Abstract. This paper gives an overview over the development of a for- mally verified file system for flash memory. We describe our approach that is based on Abstract State Machines and incremental modular re- finement. Some of the important intermediate levels and the features they introduce are given. We report on the verification challenges addressed so far, and point to open problems and future work. We furthermore draw preliminary conclusions on the methodology and the required tool support. 1 Introduction Flaws in the design and implementation of file systems already lead to serious problems in mission-critical systems. A prominent example is the Mars Explo- ration Rover Spirit [34] that got stuck in a reset cycle. In 2013, the Mars Rover Curiosity also had a bug in its file system implementation, that triggered an au- tomatic switch to safe mode. The first incident prompted a proposal to formally verify a file system for flash memory [24,18] as a pilot project for Hoare's Grand Challenge [22]. We are developing a verified flash file system (FFS). This paper reports on our progress and discusses some of the aspects of the project. We describe parts of the design, the formal models, and proofs, pointing out challenges and solutions. The main characteristic of flash memory that guides the design is that data cannot be overwritten in place, instead space can only be reused by erasing whole blocks.
[Show full text]
Membrane: Operating System Support for Restartable File Systems Swaminathan Sundararaman, Sriram Subramanian, Abhishek Rajimwale, Andrea C
Membrane: Operating System Support for Restartable File Systems Swaminathan Sundararaman, Sriram Subramanian, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Michael M. Swift Computer Sciences Department, University of Wisconsin, Madison Abstract and most complex code bases in the kernel. Further, We introduce Membrane, a set of changes to the oper- file systems are still under active development, and new ating system to support restartable file systems. Mem- ones are introduced quite frequently. For example, Linux brane allows an operating system to tolerate a broad has many established file systems, including ext2 [34], class of file system failures and does so while remain- ext3 [35], reiserfs [27], and still there is great interest in ing transparent to running applications; upon failure, the next-generation file systems such as Linux ext4 and btrfs. file system restarts, its state is restored, and pending ap- Thus, file systems are large, complex, and under develop- plication requests are serviced as if no failure had oc- ment, the perfect storm for numerous bugs to arise. curred. Membrane provides transparent recovery through Because of the likely presence of flaws in their imple- a lightweight logging and checkpoint infrastructure, and mentation, it is critical to consider how to recover from includes novel techniques to improve performance and file system crashes as well. Unfortunately, we cannot di- correctness of its fault-anticipation and recovery machin- rectly apply previous work from the device-driver litera- ery. We tested Membrane with ext2, ext3, and VFAT. ture to improving file-system fault recovery. File systems, Through experimentation, we show that Membrane in- unlike device drivers, are extremely stateful, as they man- duces little performance overhead and can tolerate a wide age vast amounts of both in-memory and persistent data; range of file system crashes.
[Show full text]
Ext4 File System and Crash Consistency
1 Ext4 file system and crash consistency Changwoo Min 2 Summary of last lectures • Tools: building, exploring, and debugging Linux kernel • Core kernel infrastructure • Process management & scheduling • Interrupt & interrupt handler • Kernel synchronization • Memory management • Virtual file system • Page cache and page fault 3 Today: ext4 file system and crash consistency • File system in Linux kernel • Design considerations of a file system • History of file system • On-disk structure of Ext4 • File operations • Crash consistency 4 File system in Linux kernel User space application (ex: cp) User-space Syscalls: open, read, write, etc. Kernel-space VFS: Virtual File System Filesystems ext4 FAT32 JFFS2 Block layer Hardware Embedded Hard disk USB drive flash 5 What is a file system fundamentally? int main(int argc, char *argv[]) { int fd; char buffer[4096]; struct stat_buf; DIR *dir; struct dirent *entry; /* 1. Path name -> inode mapping */ fd = open("/home/lkp/hello.c" , O_RDONLY); /* 2. File offset -> disk block address mapping */ pread(fd, buffer, sizeof(buffer), 0); /* 3. File meta data operation */ fstat(fd, &stat_buf); printf("file size = %d\n", stat_buf.st_size); /* 4. Directory operation */ dir = opendir("/home"); entry = readdir(dir); printf("dir = %s\n", entry->d_name); return 0; } 6 Why do we care EXT4 file system? • Most widely-deployed file system • Default file system of major Linux distributions • File system used in Google data center • Default file system of Android kernel • Follows the traditional file system design 7 History of file system design 8 UFS (Unix File System) • The original UNIX file system • Design by Dennis Ritche and Ken Thompson (1974) • The first Linux file system (ext) and Minix FS has a similar layout 9 UFS (Unix File System) • Performance problem of UFS (and the first Linux file system) • Especially, long seek time between an inode and data block 10 FFS (Fast File System) • The file system of BSD UNIX • Designed by Marshall Kirk McKusick, et al.
[Show full text]
CS 152: Computer Systems Architecture Storage Technologies
CS 152: Computer Systems Architecture Storage Technologies Sang-Woo Jun Winter 2019 Storage Used To be a Secondary Concern Typically, storage was not a first order citizen of a computer system o As alluded to by its name “secondary storage” o Its job was to load programs and data to memory, and disappear o Most applications only worked with CPU and system memory (DRAM) o Extreme applications like DBMSs were the exception Because conventional secondary storage was very slow o Things are changing! Some (Pre)History Magnetic core memory Rope memory (ROM) 1960’s Drum memory 1950~1970s 72 KiB per cubic foot! 100s of KiB (1024 bits in photo) Hand-woven to program the 1950’s Apollo guidance computer Photos from Wikipedia Some (More Recent) History Floppy disk drives 1970’s~2000’s 100 KiBs to 1.44 MiB Hard disk drives 1950’s to present MBs to TBs Photos from Wikipedia Some (Current) History Solid State Drives Non-Volatile Memory 2000’s to present 2010’s to present GB to TBs GBs Hard Disk Drives Dominant storage medium for the longest time o Still the largest capacity share Data organized into multiple magnetic platters o Mechanical head needs to move to where data is, to read it o Good sequential access, terrible random access • 100s of MB/s sequential, maybe 1 MB/s 4 KB random o Time for the head to move to the right location (“seek time”) may be ms long • 1000,000s of cycles! Typically “ATA” (Including IDE and EIDE), and later “SATA” interfaces o Connected via “South bridge” chipset Ding Yuan, “Operating Systems ECE344 Lecture 11: File
[Show full text]
Elinos Product Overview
SYSGO Product Overview ELinOS 7 Industrial Grade Linux ELinOS is a SYSGO Linux distribution to help developers save time and effort by focusing on their application. Our Industrial Grade Linux with user-friendly IDE goes along with the best selection of software packages to meet our cog linux Qt LOCK customers needs, and with the comfort of world-class technical support. ELinOS now includes Docker support Feature LTS Qt Open SSH Configurator Kernel embedded Open VPN in order to isolate applications running on the same system. laptop Q Bug Shield-Virus Docker Eclipse-based QEMU-based Application Integrated Docker IDE HW Emulators Debugging Firewall Support ELINOS FEATURES MANAGING EMBEDDED LINUX VERSATILITY • Industrial Grade Creating an Embedded Linux based system is like solving a puzzle and putting • Eclipse-based IDE for embedded the right pieces together. This requires a deep knowledge of Linux’s versatility Systems (CODEO) and takes time for the selection of components, development of Board Support • Multiple Linux kernel versions Packages and drivers, and testing of the whole system – not only for newcomers. incl. Kernel 4.19 LTS with real-time enhancements With ELinOS, SYSGO offers an ‘out-of-the-box’ experience which allows to focus • Quick and easy target on the development of competitive applications itself. ELinOS incorporates the system configuration appropriate tools, such as a feature configurator to help you build the system and • Hardware Emulation (QEMU) boost your project success, including a graphical configuration front-end with a • Extensive file system support built-in integrity validation. • Application debugging • Target analysis APPLICATION & CONFIGURATION ENVIRONMENT • Runs out-of-the-box on PikeOS • Validated and tested for In addition to standard tools, remote debugging, target system monitoring and PowerPC, x86, ARM timing behaviour analyses are essential for application development.
[Show full text]
AMD Alchemy™ Processors Building a Root File System for Linux® Incorporating Memory Technology Devices
AMD Alchemy™ Processors Building a Root File System for Linux® Incorporating Memory Technology Devices 1.0 Scope This document outlines a step-by-step process for building and deploying a Flash-based root file system for Linux® on an AMD Alchemy™ processor-based development board, using an approach that incorporates Memory Technology Devices (MTDs) with the JFFS2 file system. Note: This document describes creating a root file system on NOR Flash memory devices, and does not apply to NAND Flash devices. 1.1 Journaling Flash File System JFFS2 is the second generation of the Journaling Flash File System (JFFS). This file system provides a crash-safe and powerdown-safe Linux file system for use with Flash memory devices. The home page for the JFFS project is located at http://developer.axis.com/software/jffs. 1.2 Memory Technology Device The MTD subsystem provides a generic Linux driver for a wide range of memory devices, including Flash memory devices. This driver creates an abstracted device used by JFFS2 to interface to the actual Flash memory hardware. The home page for the MTD project is located at http://www.linux-mtd.infradead.org. 2.0 Building the Root File System Before being deployed to an AMD Alchemy platform, the file system must first be built on an x86 Linux host PC. The pri- mary concern when building a Flash-based root file system is often the size of the image. The file system must be designed so that it fits within the available space of the Flash memory, with enough extra space to accommodate any runtime-created files, such as temporary or log files.
[Show full text]
Filesystem Considerations for Embedded Devices ELC2015 03/25/15
Filesystem considerations for embedded devices ELC2015 03/25/15 Tristan Lelong Senior embedded software engineer Filesystem considerations ABSTRACT The goal of this presentation is to answer a question asked by several customers: which filesystem should you use within your embedded design’s eMMC/SDCard? These storage devices use a standard block interface, compatible with traditional filesystems, but constraints are not those of desktop PC environments. EXT2/3/4, BTRFS, F2FS are the first of many solutions which come to mind, but how do they all compare? Typical queries include performance, longevity, tools availability, support, and power loss robustness. This presentation will not dive into implementation details but will instead summarize provided answers with the help of various figures and meaningful test results. 2 TABLE OF CONTENTS 1. Introduction 2. Block devices 3. Available filesystems 4. Performances 5. Tools 6. Reliability 7. Conclusion Filesystem considerations ABOUT THE AUTHOR • Tristan Lelong • Embedded software engineer @ Adeneo Embedded • French, living in the Pacific northwest • Embedded software, free software, and Linux kernel enthusiast. 4 Introduction Filesystem considerations Introduction INTRODUCTION More and more embedded designs rely on smart memory chips rather than bare NAND or NOR. This presentation will start by describing: • Some context to help understand the differences between NAND and MMC • Some typical requirements found in embedded devices designs • Potential filesystems to use on MMC devices 6 Filesystem considerations Introduction INTRODUCTION Focus will then move to block filesystems. How they are supported, what feature do they advertise. To help understand how they compare, we will present some benchmarks and comparisons regarding: • Tools • Reliability • Performances 7 Block devices Filesystem considerations Block devices MMC, EMMC, SD CARD Vocabulary: • MMC: MultiMediaCard is a memory card unveiled in 1997 by SanDisk and Siemens based on NAND flash memory.
[Show full text]
BEC Brochure for Partners
3724 Heron Way, Palo Alto, CA 94303, USA +1 (650) 272-03-84 (USA and Canada) +7 (812) 926-64-74 (Europe and other regions) BELKASOFT EVIDENCE CENTER All-in-one forensic solution for locating, extracting, and analyzing digital evidence stored inside computers and mobile devices and mobile devices, RAM and cloud. Belkasoft Evidence Center is designed with forensic experts and investigators in mind: it automatically performs multiple tasks without requiring your presence, allowing you to speed up the investigation; at the same time, the product has convenient interface, making it a powerful, yet easy-to-use tool for data extraction, search, and analysis. INCLUDES Fully automated extraction and Advanced low-level expertise analysis of 1000+ types of evidence Concise and adjustable reports, Destroyed and hidden evidence accepted by courts recovery via data carving Case Management and a possibility Live RAM analysis to create a portable case to share with a colleague at no cost TYPES OF EVIDENCE SUPPORTED BY EVIDENCE CENTER Office documents System files, including Windows 10 Email clients timeline and TOAST, macOS plists, smartphone Wi-Fi and Bluetooth Pictures and videos configurations etc. Mobile application data Cryptocurrencies Web browser histories, cookies, cache, passwords, etc. Registry files Chats and instant messenger histories SQLite databases Social networks and cloud services Peer-to-peer software Encrypted files and volumes Plist files TYPES OF ANALYSIS PERFORMED BY EVIDENCE CENTER Existing files search and analysis. Low-level investigation using Hex Viewer Timeline analysis - ability to display and filter all user activities and system events in a single aggregated view Full-text search through all types of collected evidence.
[Show full text]
Recursive Updates in Copy-On-Write File Systems - Modeling and Analysis
2342 JOURNAL OF COMPUTERS, VOL. 9, NO. 10, OCTOBER 2014 Recursive Updates in Copy-on-write File Systems - Modeling and Analysis Jie Chen*, Jun Wang†, Zhihu Tan*, Changsheng Xie* *School of Computer Science and Technology Huazhong University of Science and Technology, China *Wuhan National Laboratory for Optoelectronics, Wuhan, Hubei 430074, China [email protected], {stan, cs_xie}@hust.edu.cn †Dept. of Electrical Engineering and Computer Science University of Central Florida, Orlando, Florida 32826, USA [email protected] Abstract—Copy-On-Write (COW) is a powerful technique recursive update. Recursive updates can lead to several for data protection in file systems. Unfortunately, it side effects to a storage system, such as write introduces a recursively updating problem, which leads to a amplification (also can be referred as additional writes) side effect of write amplification. Studying the behaviors of [4], I/O pattern alternation [5], and performance write amplification is important for designing, choosing and degradation [6]. This paper focuses on the side effects of optimizing the next generation file systems. However, there are many difficulties for evaluation due to the complexity of write amplification. file systems. To solve this problem, we proposed a typical Studying the behaviors of write amplification is COW file system model based on BTRFS, verified its important for designing, choosing, and optimizing the correctness through carefully designed experiments. By next generation file systems, especially when the file analyzing this model, we found that write amplification is systems uses a flash-memory-based underlying storage greatly affected by the distributions of files being accessed, system under online transaction processing (OLTP) which varies from 1.1x to 4.2x.
[Show full text]
F2punifycr: a Flash-Friendly Persistent Burst-Buffer File System
F2PUnifyCR: A Flash-friendly Persistent Burst-Buffer File System ThanOS Department of Computer Science Florida State University Tallahassee, United States I. ABSTRACT manifold depending on the workloads it is handling for With the increased amount of supercomputing power, applications. In order to leverage the capabilities of burst it is now possible to work with large scale data that buffers to the utmost level, it is very important to have a pose a continuous opportunity for exascale computing standardized software interface across systems. It has to that puts immense pressure on underlying persistent data deal with an immense amount of data during the runtime storage. Burst buffers, a distributed array of node-local of the applications. persistent flash storage devices deployed on most of Using node-local burst buffer can achieve scalable the leardership supercomputers, are means to efficiently write bandwidth as it lets each process write to the handling the bursty I/O invoked through cutting-edge local flash drive, but when the files are shared across scientific applications. In order to manage these burst many processes, it puts the management of metadata buffers, many ephemeral user level file system solutions, and object data of the files under huge challenge. In like UnifyCR, are present in the research and industry order to handle all the challenges posed by the bursty arena. Because of the intrinsic nature of the flash devices and random I/O requests by the Scientific Applica- due to background processing overhead, like Garbage tions running on leadership Supercomputing clusters, Collection, peak write bandwidth is hard to get.
[Show full text]
Yaffs Direct Interface
Yaffs Direct Interface Charles Manning 2017-06-07 This document describes the Yaffs Direct Interface (YDI), which allows Yaffs to be simply integrated with embedded systems, with or without an RTOS. Table of Contents 1 Background.............................................................................................................................3 2 Licensing.................................................................................................................................3 3 What are Yaffs and Yaffs Direct Interface?.............................................................................3 4 Why use Yaffs?........................................................................................................................4 5 Source Code and Yaffs Resources ..........................................................................................4 6 System Requirements..............................................................................................................5 7 How to integrate Yaffs with an RTOS/Embedded system......................................................5 7.1 Source Files......................................................................................................................6 7.2 Integrating the POSIX Application Interface...................................................................9 8 RTOS Integration Interface...................................................................................................11 9 Yaffs NAND Model..............................................................................................................13
[Show full text]
Foot Prints Feel the Freedom of Fedora!
The Fedora Project: Foot Prints Feel The Freedom of Fedora! RRaahhuull SSuunnddaarraamm SSuunnddaarraamm@@ffeeddoorraapprroojjeecctt..oorrgg FFrreeee ((aass iinn ssppeeeecchh aanndd bbeeeerr)) AAddvviiccee 101011:: KKeeeepp iitt iinntteerraaccttiivvee!! Credit: Based on previous Fedora presentations from Red Hat and various community members. Using the age old wisdom and Indian, Free software tradition of standing on the shoulders of giants. Who the heck is Rahul? ( my favorite part of this presentation) ✔ Self elected Fedora project monkey and noisemaker ✔ Fedora Project Board Member ✔ Fedora Ambassadors steering committee member. ✔ Fedora Ambassador for India.. ✔ Editor for Fedora weekly reports. ✔ Fedora Websites, Documentation and Bug Triaging projects volunteer and miscellaneous few grunt work. Agenda ● Red Hat Linux to Fedora & RHEL - Why? ● What is Fedora ? ● What is the Fedora Project ? ● Who is behind the Fedora Project ? ● Primary Principles. ● What are the Fedora projects? ● Features, Future – Fedora Core 5 ... The beginning: Red Hat Linux 1994-2003 ● Released about every 6 months ● More stable “ .2” releases about every 18 months ● Rapid innovation ● Problems with retail channel sales model ● Impossible to support long-term ● Community Participation: ● Upstream Projects ● Beta Team / Bug Reporting The big split: Fedora and RHEL Red Hat had two separate, irreconcilable goals: ● To innovate rapidly. To provide stability for the long-term ● Red Hat Enterprise Linux (RHEL) ● Stable and supported for 7 years plus. A platform for 3rd party standardization ● Free as in speech ● Fedora Project / Fedora Core ● Rapid releases of Fedora Core, every 6 months ● Space to innovate. Fedora Core in the tradition of Red Hat Linux (“ FC1 == RHL10” ) Free as in speech, free as in beer, free as in community support ● Built and sponsored by Red Hat ● ...with increased community contributions.
[Show full text]