Evaluation of Flash File Systems for Large NAND Flash Memory - Introduction of 2009 CELF Embedded Linux Conference
Total Page:16
File Type:pdf, Size:1020Kb
Load more
Recommended publications
-
Development of a Verified Flash File System ⋆
Development of a Verified Flash File System ? Gerhard Schellhorn, Gidon Ernst, J¨orgPf¨ahler,Dominik Haneberg, and Wolfgang Reif Institute for Software & Systems Engineering University of Augsburg, Germany fschellhorn,ernst,joerg.pfaehler,haneberg,reifg @informatik.uni-augsburg.de Abstract. This paper gives an overview over the development of a for- mally verified file system for flash memory. We describe our approach that is based on Abstract State Machines and incremental modular re- finement. Some of the important intermediate levels and the features they introduce are given. We report on the verification challenges addressed so far, and point to open problems and future work. We furthermore draw preliminary conclusions on the methodology and the required tool support. 1 Introduction Flaws in the design and implementation of file systems already lead to serious problems in mission-critical systems. A prominent example is the Mars Explo- ration Rover Spirit [34] that got stuck in a reset cycle. In 2013, the Mars Rover Curiosity also had a bug in its file system implementation, that triggered an au- tomatic switch to safe mode. The first incident prompted a proposal to formally verify a file system for flash memory [24,18] as a pilot project for Hoare's Grand Challenge [22]. We are developing a verified flash file system (FFS). This paper reports on our progress and discusses some of the aspects of the project. We describe parts of the design, the formal models, and proofs, pointing out challenges and solutions. The main characteristic of flash memory that guides the design is that data cannot be overwritten in place, instead space can only be reused by erasing whole blocks. -
LSP 1.20 Davinci Linux NOR Flash Device Driver
LSP 1.20 DaVinci Linux NOR Flash Driver User's Guide Literature Number: SPRUF10 April 2008 2 SPRUF10–April 2008 Submit Documentation Feedback Contents 1 Overview............................................................................................................................. 5 1.1 System Requirements .................................................................................................... 5 1.2 Design Overview .......................................................................................................... 6 2 Installation Guide................................................................................................................. 7 2.1 List of Installable Components .......................................................................................... 7 2.2 Component Folder ........................................................................................................ 7 2.3 Development Tools ....................................................................................................... 7 2.4 Build......................................................................................................................... 8 2.5 Steps to Load/Unload the NOR Flash Driver.......................................................................... 8 3 NOR Flash Driver Porting...................................................................................................... 9 3.1 Customizing the NOR-flash partitions ................................................................................. -
Membrane: Operating System Support for Restartable File Systems Swaminathan Sundararaman, Sriram Subramanian, Abhishek Rajimwale, Andrea C
Membrane: Operating System Support for Restartable File Systems Swaminathan Sundararaman, Sriram Subramanian, Abhishek Rajimwale, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, Michael M. Swift Computer Sciences Department, University of Wisconsin, Madison Abstract and most complex code bases in the kernel. Further, We introduce Membrane, a set of changes to the oper- file systems are still under active development, and new ating system to support restartable file systems. Mem- ones are introduced quite frequently. For example, Linux brane allows an operating system to tolerate a broad has many established file systems, including ext2 [34], class of file system failures and does so while remain- ext3 [35], reiserfs [27], and still there is great interest in ing transparent to running applications; upon failure, the next-generation file systems such as Linux ext4 and btrfs. file system restarts, its state is restored, and pending ap- Thus, file systems are large, complex, and under develop- plication requests are serviced as if no failure had oc- ment, the perfect storm for numerous bugs to arise. curred. Membrane provides transparent recovery through Because of the likely presence of flaws in their imple- a lightweight logging and checkpoint infrastructure, and mentation, it is critical to consider how to recover from includes novel techniques to improve performance and file system crashes as well. Unfortunately, we cannot di- correctness of its fault-anticipation and recovery machin- rectly apply previous work from the device-driver litera- ery. We tested Membrane with ext2, ext3, and VFAT. ture to improving file-system fault recovery. File systems, Through experimentation, we show that Membrane in- unlike device drivers, are extremely stateful, as they man- duces little performance overhead and can tolerate a wide age vast amounts of both in-memory and persistent data; range of file system crashes. -
Ext4 File System and Crash Consistency
1 Ext4 file system and crash consistency Changwoo Min 2 Summary of last lectures • Tools: building, exploring, and debugging Linux kernel • Core kernel infrastructure • Process management & scheduling • Interrupt & interrupt handler • Kernel synchronization • Memory management • Virtual file system • Page cache and page fault 3 Today: ext4 file system and crash consistency • File system in Linux kernel • Design considerations of a file system • History of file system • On-disk structure of Ext4 • File operations • Crash consistency 4 File system in Linux kernel User space application (ex: cp) User-space Syscalls: open, read, write, etc. Kernel-space VFS: Virtual File System Filesystems ext4 FAT32 JFFS2 Block layer Hardware Embedded Hard disk USB drive flash 5 What is a file system fundamentally? int main(int argc, char *argv[]) { int fd; char buffer[4096]; struct stat_buf; DIR *dir; struct dirent *entry; /* 1. Path name -> inode mapping */ fd = open("/home/lkp/hello.c" , O_RDONLY); /* 2. File offset -> disk block address mapping */ pread(fd, buffer, sizeof(buffer), 0); /* 3. File meta data operation */ fstat(fd, &stat_buf); printf("file size = %d\n", stat_buf.st_size); /* 4. Directory operation */ dir = opendir("/home"); entry = readdir(dir); printf("dir = %s\n", entry->d_name); return 0; } 6 Why do we care EXT4 file system? • Most widely-deployed file system • Default file system of major Linux distributions • File system used in Google data center • Default file system of Android kernel • Follows the traditional file system design 7 History of file system design 8 UFS (Unix File System) • The original UNIX file system • Design by Dennis Ritche and Ken Thompson (1974) • The first Linux file system (ext) and Minix FS has a similar layout 9 UFS (Unix File System) • Performance problem of UFS (and the first Linux file system) • Especially, long seek time between an inode and data block 10 FFS (Fast File System) • The file system of BSD UNIX • Designed by Marshall Kirk McKusick, et al. -
O'reilly Linux Kernel in a Nutshell.Pdf
,title.4229 Page i Friday, December 1, 2006 9:52 AM LINUX KERNEL IN A NUTSHELL ,title.4229 Page ii Friday, December 1, 2006 9:52 AM Other Linux resources from O’Reilly Related titles Building Embedded Linux Running Linux Systems Understanding Linux Linux Device Drivers Network Internals Linux in a Nutshell Understanding the Linux Linux Pocket Guide Kernel Linux Books linux.oreilly.com is a complete catalog of O’Reilly’s Resource Center books on Linux and Unix and related technologies, in- cluding sample chapters and code examples. Conferences O’Reilly brings diverse innovators together to nurture the ideas that spark revolutionary industries. We spe- cialize in documenting the latest tools and systems, translating the innovator’s knowledge into useful skills for those in the trenches. Visit conferences.oreilly.com for our upcoming events. Safari Bookshelf (safari.oreilly.com) is the premier on- line reference library for programmers and IT professionals. Conduct searches across more than 1,000 books. Subscribers can zero in on answers to time-critical questions in a matter of seconds. Read the books on your Bookshelf from cover to cover or sim- ply flip to the page you need. Try it today for free. ,title.4229 Page iii Friday, December 1, 2006 9:52 AM LINUX KERNEL IN A NUTSHELL Greg Kroah-Hartman Beijing • Cambridge • Farnham • Köln • Paris • Sebastopol • Taipei • Tokyo ,LKNSTOC.fm.8428 Page v Friday, December 1, 2006 9:55 AM Chapter 1 Table of Contents Preface . ix Part I. Building the Kernel 1. Introduction . 3 Using This Book 4 2. Requirements for Building and Using the Kernel . -
ZBD: Using Transparent Compression at the Block Level to Increase Storage Space Efficiency
ZBD: Using Transparent Compression at the Block Level to Increase Storage Space Efficiency Thanos Makatos∗†, Yannis Klonatos∗†, Manolis Marazakis∗, Michail D. Flouris∗, and Angelos Bilas∗† ∗ Foundation for Research and Technology - Hellas (FORTH) Institute of Computer Science (ICS) 100 N. Plastira Av., Vassilika Vouton, Heraklion, GR-70013, Greece † Department of Computer Science, University of Crete P.O. Box 2208, Heraklion, GR 71409, Greece {mcatos, klonatos, maraz, flouris, bilas}@ics.forth.gr Abstract—In this work we examine how transparent compres- • Logical to physical block mapping: Block-level compres- sion in the I/O path can improve space efficiency for online sion imposes a many-to-one mapping from logical to storage. We extend the block layer with the ability to compress physical blocks, as multiple compressed logical blocks and decompress data as they flow between the file-system and the disk. Achieving transparent compression requires extensive must be stored in the same physical block. This requires metadata management for dealing with variable block sizes, dy- using a translation mechanism that imposes low overhead namic block mapping, block allocation, explicit work scheduling in the common I/O path and scales with the capacity and I/O optimizations to mitigate the impact of additional I/Os of the underlying devices as well as a block alloca- and compression overheads. Preliminary results show that online tion/deallocation mechanism that affects data placement. transparent compression is a viable option for improving effective • storage capacity, it can improve I/O performance by reducing Increased number of I/Os: Using compression increases I/O traffic and seek distance, and has a negative impact on the number of I/Os required on the critical path during performance only when single-thread I/O latency is critical. -
CS 152: Computer Systems Architecture Storage Technologies
CS 152: Computer Systems Architecture Storage Technologies Sang-Woo Jun Winter 2019 Storage Used To be a Secondary Concern Typically, storage was not a first order citizen of a computer system o As alluded to by its name “secondary storage” o Its job was to load programs and data to memory, and disappear o Most applications only worked with CPU and system memory (DRAM) o Extreme applications like DBMSs were the exception Because conventional secondary storage was very slow o Things are changing! Some (Pre)History Magnetic core memory Rope memory (ROM) 1960’s Drum memory 1950~1970s 72 KiB per cubic foot! 100s of KiB (1024 bits in photo) Hand-woven to program the 1950’s Apollo guidance computer Photos from Wikipedia Some (More Recent) History Floppy disk drives 1970’s~2000’s 100 KiBs to 1.44 MiB Hard disk drives 1950’s to present MBs to TBs Photos from Wikipedia Some (Current) History Solid State Drives Non-Volatile Memory 2000’s to present 2010’s to present GB to TBs GBs Hard Disk Drives Dominant storage medium for the longest time o Still the largest capacity share Data organized into multiple magnetic platters o Mechanical head needs to move to where data is, to read it o Good sequential access, terrible random access • 100s of MB/s sequential, maybe 1 MB/s 4 KB random o Time for the head to move to the right location (“seek time”) may be ms long • 1000,000s of cycles! Typically “ATA” (Including IDE and EIDE), and later “SATA” interfaces o Connected via “South bridge” chipset Ding Yuan, “Operating Systems ECE344 Lecture 11: File -
Elinos Product Overview
SYSGO Product Overview ELinOS 7 Industrial Grade Linux ELinOS is a SYSGO Linux distribution to help developers save time and effort by focusing on their application. Our Industrial Grade Linux with user-friendly IDE goes along with the best selection of software packages to meet our cog linux Qt LOCK customers needs, and with the comfort of world-class technical support. ELinOS now includes Docker support Feature LTS Qt Open SSH Configurator Kernel embedded Open VPN in order to isolate applications running on the same system. laptop Q Bug Shield-Virus Docker Eclipse-based QEMU-based Application Integrated Docker IDE HW Emulators Debugging Firewall Support ELINOS FEATURES MANAGING EMBEDDED LINUX VERSATILITY • Industrial Grade Creating an Embedded Linux based system is like solving a puzzle and putting • Eclipse-based IDE for embedded the right pieces together. This requires a deep knowledge of Linux’s versatility Systems (CODEO) and takes time for the selection of components, development of Board Support • Multiple Linux kernel versions Packages and drivers, and testing of the whole system – not only for newcomers. incl. Kernel 4.19 LTS with real-time enhancements With ELinOS, SYSGO offers an ‘out-of-the-box’ experience which allows to focus • Quick and easy target on the development of competitive applications itself. ELinOS incorporates the system configuration appropriate tools, such as a feature configurator to help you build the system and • Hardware Emulation (QEMU) boost your project success, including a graphical configuration front-end with a • Extensive file system support built-in integrity validation. • Application debugging • Target analysis APPLICATION & CONFIGURATION ENVIRONMENT • Runs out-of-the-box on PikeOS • Validated and tested for In addition to standard tools, remote debugging, target system monitoring and PowerPC, x86, ARM timing behaviour analyses are essential for application development. -
AMD Alchemy™ Processors Building a Root File System for Linux® Incorporating Memory Technology Devices
AMD Alchemy™ Processors Building a Root File System for Linux® Incorporating Memory Technology Devices 1.0 Scope This document outlines a step-by-step process for building and deploying a Flash-based root file system for Linux® on an AMD Alchemy™ processor-based development board, using an approach that incorporates Memory Technology Devices (MTDs) with the JFFS2 file system. Note: This document describes creating a root file system on NOR Flash memory devices, and does not apply to NAND Flash devices. 1.1 Journaling Flash File System JFFS2 is the second generation of the Journaling Flash File System (JFFS). This file system provides a crash-safe and powerdown-safe Linux file system for use with Flash memory devices. The home page for the JFFS project is located at http://developer.axis.com/software/jffs. 1.2 Memory Technology Device The MTD subsystem provides a generic Linux driver for a wide range of memory devices, including Flash memory devices. This driver creates an abstracted device used by JFFS2 to interface to the actual Flash memory hardware. The home page for the MTD project is located at http://www.linux-mtd.infradead.org. 2.0 Building the Root File System Before being deployed to an AMD Alchemy platform, the file system must first be built on an x86 Linux host PC. The pri- mary concern when building a Flash-based root file system is often the size of the image. The file system must be designed so that it fits within the available space of the Flash memory, with enough extra space to accommodate any runtime-created files, such as temporary or log files. -
Filesystem Considerations for Embedded Devices ELC2015 03/25/15
Filesystem considerations for embedded devices ELC2015 03/25/15 Tristan Lelong Senior embedded software engineer Filesystem considerations ABSTRACT The goal of this presentation is to answer a question asked by several customers: which filesystem should you use within your embedded design’s eMMC/SDCard? These storage devices use a standard block interface, compatible with traditional filesystems, but constraints are not those of desktop PC environments. EXT2/3/4, BTRFS, F2FS are the first of many solutions which come to mind, but how do they all compare? Typical queries include performance, longevity, tools availability, support, and power loss robustness. This presentation will not dive into implementation details but will instead summarize provided answers with the help of various figures and meaningful test results. 2 TABLE OF CONTENTS 1. Introduction 2. Block devices 3. Available filesystems 4. Performances 5. Tools 6. Reliability 7. Conclusion Filesystem considerations ABOUT THE AUTHOR • Tristan Lelong • Embedded software engineer @ Adeneo Embedded • French, living in the Pacific northwest • Embedded software, free software, and Linux kernel enthusiast. 4 Introduction Filesystem considerations Introduction INTRODUCTION More and more embedded designs rely on smart memory chips rather than bare NAND or NOR. This presentation will start by describing: • Some context to help understand the differences between NAND and MMC • Some typical requirements found in embedded devices designs • Potential filesystems to use on MMC devices 6 Filesystem considerations Introduction INTRODUCTION Focus will then move to block filesystems. How they are supported, what feature do they advertise. To help understand how they compare, we will present some benchmarks and comparisons regarding: • Tools • Reliability • Performances 7 Block devices Filesystem considerations Block devices MMC, EMMC, SD CARD Vocabulary: • MMC: MultiMediaCard is a memory card unveiled in 1997 by SanDisk and Siemens based on NAND flash memory. -
Ensuring Data Confidentiality Via Plausibly Deniable Encryption and Secure Deletion – a Survey Qionglu Zhang1,2,3, Shijie Jia1,2*, Bing Chang4 and Bo Chen5*
Zhang et al. Cybersecurity (2018) 1:1 Cybersecurity https://doi.org/10.1186/s42400-018-0005-8 SURVEY Open Access Ensuring data confidentiality via plausibly deniable encryption and secure deletion – a survey Qionglu Zhang1,2,3, Shijie Jia1,2*, Bing Chang4 and Bo Chen5* Abstract Ensuring confidentiality of sensitive data is of paramount importance, since data leakage may not only endanger data owners’ privacy, but also ruin reputation of businesses as well as violate various regulations like HIPPA and Sarbanes-Oxley Act. To provide confidentiality guarantee, the data should be protected when they are preserved in the personal computing devices (i.e., confidentiality during their lifetime); and also, they should be rendered irrecoverable after they are removed from the devices (i.e., confidentiality after their lifetime). Encryption and secure deletion are used to ensure data confidentiality during and after their lifetime, respectively. This work aims to perform a thorough literature review on the techniques being used to protect confidentiality of the data in personal computing devices, including both encryption and secure deletion. Especially for encryption, we mainly focus on the novel plausibly deniable encryption (PDE), which can ensure data confidentiality against both a coercive (i.e., the attacker can coerce the data owner for the decryption key) and a non-coercive attacker. Keywords: Data confidentiality, Plausibly deniable encryption, Secure deletion Introduction Spahr LLP: State and local governments move swiftly to Modern computing devices (e.g., desktops, laptops, smart sue equifax 2017); Third, it will directly violate regulations phones, tablets, wearable devices) are increasingly used like HIPAA (Congress 1996), Gramm-Leach-Bliley Act to process sensitive or even mission critical data. -
BEC Brochure for Partners
3724 Heron Way, Palo Alto, CA 94303, USA +1 (650) 272-03-84 (USA and Canada) +7 (812) 926-64-74 (Europe and other regions) BELKASOFT EVIDENCE CENTER All-in-one forensic solution for locating, extracting, and analyzing digital evidence stored inside computers and mobile devices and mobile devices, RAM and cloud. Belkasoft Evidence Center is designed with forensic experts and investigators in mind: it automatically performs multiple tasks without requiring your presence, allowing you to speed up the investigation; at the same time, the product has convenient interface, making it a powerful, yet easy-to-use tool for data extraction, search, and analysis. INCLUDES Fully automated extraction and Advanced low-level expertise analysis of 1000+ types of evidence Concise and adjustable reports, Destroyed and hidden evidence accepted by courts recovery via data carving Case Management and a possibility Live RAM analysis to create a portable case to share with a colleague at no cost TYPES OF EVIDENCE SUPPORTED BY EVIDENCE CENTER Office documents System files, including Windows 10 Email clients timeline and TOAST, macOS plists, smartphone Wi-Fi and Bluetooth Pictures and videos configurations etc. Mobile application data Cryptocurrencies Web browser histories, cookies, cache, passwords, etc. Registry files Chats and instant messenger histories SQLite databases Social networks and cloud services Peer-to-peer software Encrypted files and volumes Plist files TYPES OF ANALYSIS PERFORMED BY EVIDENCE CENTER Existing files search and analysis. Low-level investigation using Hex Viewer Timeline analysis - ability to display and filter all user activities and system events in a single aggregated view Full-text search through all types of collected evidence.