EROFS: a Compression-Friendly Readonly File System for Resource-Scarce Devices

Total Page：16

File Type：pdf, Size：1020Kb

EROFS: A Compression-friendly Readonly File System for Resource-scarce Devices Xiang Gao#, Mingkai Dong*, Xie Miao#, Wei Du#, Chao Yu#, Haibo Chen*,# #Huawei Technologies Co., Ltd. *IPADS, Shanghai Jiao Tong University USENIX ATC 2019, Renton, WA, USA System resources in Android /oem /system 3072 2654 /system Partition Size (MB) /odm 840 654 650 512 /vendor 2.3.6 4.3 5.1 6.0 7.0 8.0 Android Version ... System resources consume insignificant storage! System resources in Android Read-only Read-only /oem /system 3072 2654 Read-only /system Partition Size (MB) /odm Read-only 840 654 650 512 /vendor 2.3.6 4.3 5.1 6.0 7.0 8.0 Android Version ... System resources consume insignificant storage! Read-only system partitions → compressed read-only file systems Squashfs on system partitions /system /oem /vendor /odm Squashfs on system partitions /system /oem Squashfs Squashfs /vendor /odm Squashfs Squashfs Squashfs on system partitions /system /oem Squashfs Squashfs /vendor /odm Squashfs Squashfs Squashfs on system partitions Fixed-sized input compression /system /oem Squashfs Squashfs /vendor /odm Squashfs Squashfs Squashfs on system partitions Read Amplification 1 byte decompression amplification /system /oem ... ... Squashfs Squashfs I/O /vendor /odm amplification Squashfs Squashfs Squashfs on system partitions Read Amplification Massive Memory Consumption buffer_head 1 byte ... decompression amplification decompress Page cache copy /system /oem ... ... Squashfs Squashfs I/O /vendor /odm amplification Squashfs Squashfs EROFS /system /oem Squashfs Squashfs /vendor /odm Squashfs Squashfs EROFS Fixed-sized output compression /system /oem Squashfs Squashfs /vendor /odm Squashfs Squashfs EROFS Read Amplification 84% Fixed-sized output compression /system /oem Squashfs Squashfs /vendor /odm Squashfs Squashfs EROFS Read Amplification 84% Memory Consumption 89% Fixed-sized output compression Vmap Buffer In-place Rolling 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 /system /oem Squashfs Squashfs /vendor /odm Squashfs Squashfs EROFS Read Amplification 84% Memory Consumption 89% Fixed-sized output compression Vmap Buffer In-place Rolling 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 üSaves >30% storage used by system resources /system /oem EROFS EROFS üComparable or even better performance than Ext4 /vendor /odm üUpstreamed to Linux 4.19 EROFS EROFS üRunning on 10,000,000+ smartphones EROFS: A Compression-friendly Readonly File System for Resource-scarce Devices Hope to see you at our presentation! Wednesday July 10th, 3:00 PM Track I Filesystems EROFSEROFSEROFS Thanks!.

Copy Link

Recommended publications

Huawei Announces EROFS Linux File-System, Might Eventually Be Used
ARTICLES & REVIEWS NEWS ARCHIVE FORUMS PREMIUM CATEGORIES Custom Search Search Latest Linux News Huawei Announces EROFS Linux File-System, Might Huawei Announces EROFS Linux File- Eventually Be Used By Android Devices System, Might Eventually Be Used By Android Devices Written by Michael Larabel in Linux Storage on 31 May 2018 at 09:00 AM EDT. 3 Comments Mesa 18.0.5 Is The Last Planned Release In Huawei's Gao Xiang has announced the EROFS open-source Linux file-system The Series intended for Android devices, but still at its very early stages of AMD K8 Support Stripped Out Of Coreboot development. NVIDIA’s Next Generation Mainstream GPU Will At Least Be Detailed In August EROFS is the company's new approach for a read-only file-system that would work well for Android devices. EROFS is short for the Extendable Read-Only GNOME 3 Might Be Too Resource Hungry To File-System and they began developing it with being unsatisfied with other read-only file- Ever Run Nicely On The Raspberry Pi system alternatives. XWayland Gets Patch To Automatically Use EGLStreams For NVIDIA Support When EROFS is designed to offer better performance than other read-only alternatives while still Needed focusing upon saving storage space. As part of EROFS is also a compression mode pursuing BPFILTER Landing For Linux 4.18 For a different design approach than other file-systems: the compression numbers shared in Eventually Better Firewall / Packet Filtering today's announcement on both server hardware and a Kirin 970 are compelling for being in AMDGPU Patches Prepping JPEG Support For the early stages of development.
[Show full text]
Z/OS Distributed File Service Zseries File System Implementation Z/OS V1R13
Front cover z/OS Distributed File Service zSeries File System Implementation z/OS V1R13 Defining and installing a zSeries file system Performing backup and recovery, sysplex sharing Migrating from HFS to zFS Paul Rogers Robert Hering ibm.com/redbooks International Technical Support Organization z/OS Distributed File Service zSeries File System Implementation z/OS V1R13 October 2012 SG24-6580-05 Note: Before using this information and the product it supports, read the information in “Notices” on page xiii. Sixth Edition (October 2012) This edition applies to version 1 release 13 modification 0 of IBM z/OS (product number 5694-A01) and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright International Business Machines Corporation 2010, 2012. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . xiii Trademarks . xiv Preface . .xv The team who wrote this book . .xv Now you can become a published author, too! . xvi Comments welcome. xvi Stay connected to IBM Redbooks . xvi Chapter 1. zFS file systems . 1 1.1 zSeries File System introduction. 2 1.2 Application programming interfaces . 2 1.3 zFS physical file system . 3 1.4 zFS colony address space . 4 1.5 zFS supports z/OS UNIX ACLs. 4 1.6 zFS file system aggregates. 5 1.6.1 Compatibility mode aggregates. 5 1.6.2 Multifile system aggregates. 6 1.7 Metadata cache. 7 1.8 zFS file system clones . 7 1.8.1 Backup file system . 8 1.9 zFS log files.
[Show full text]
A Study of Failure Recovery and Logging of High-Performance Parallel File Systems
1 A Study of Failure Recovery and Logging of High-Performance Parallel File Systems RUNZHOU HAN, OM RAMESHWAR GATLA, MAI ZHENG, Iowa State University JINRUI CAO, State University of New York at Plattsburgh DI ZHANG, DONG DAI, North Carolina University at Charlotte YONG CHEN, Texas Tech University JONATHAN COOK, New Mexico State University Large-scale parallel file systems (PFSes) play an essential role in high performance computing (HPC). However, despite the importance, their reliability is much less studied or understood compared with that of local storage systems or cloud storage systems. Recent failure incidents at real HPC centers have exposed the latent defects in PFS clusters as well as the urgent need for a systematic analysis. To address the challenge, we perform a study of the failure recovery and logging mechanisms of PFSes in this paper. First, to trigger the failure recovery and logging operations of the target PFS, we introduce a black- box fault injection tool called PFault, which is transparent to PFSes and easy to deploy in practice. PFault emulates the failure state of individual storage nodes in the PFS based on a set of pre-defined fault models, and enables examining the PFS behavior under fault systematically. Next, we apply PFault to study two widely used PFSes: Lustre and BeeGFS. Our analysis reveals the unique failure recovery and logging patterns of the target PFSes, and identifies multiple cases where the PFSes are imperfect in terms of failure handling. For example, Lustre includes a recovery component called LFSCK to detect and fix PFS-level inconsistencies, but we find that LFSCK itself may hang or trigger kernel panicswhen scanning a corrupted Lustre.
[Show full text]
Lustre* Software Release 2.X Operations Manual Lustre* Software Release 2.X: Operations Manual Copyright © 2010, 2011 Oracle And/Or Its Affiliates
Lustre* Software Release 2.x Operations Manual Lustre* Software Release 2.x: Operations Manual Copyright © 2010, 2011 Oracle and/or its affiliates. (The original version of this Operations Manual without the Intel modifications.) Copyright © 2011, 2012, 2013 Intel Corporation. (Intel modifications to the original version of this Operations Man- ual.) Notwithstanding Intel’s ownership of the copyright in the modifications to the original version of this Operations Manual, as between Intel and Oracle, Oracle and/or its affiliates retain sole ownership of the copyright in the unmodified portions of this Operations Manual. Important Notice from Intel INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IM- PLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSO- EVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR IN- FRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. A "Mission Critical Application" is any application in which failure of the Intel Product could result, directly or indirectly, in personal injury or death. SHOULD YOU PURCHASE OR USE INTEL'S PRODUCTS FOR ANY SUCH MISSION CRITICAL APPLICATION, YOU SHALL IN- DEMNIFY AND HOLD INTEL AND ITS SUBSIDIARIES, SUBCONTRACTORS AND AFFILIATES, AND THE DIRECTORS, OFFICERS, AND EMPLOYEES OF EACH, HARMLESS AGAINST ALL CLAIMS COSTS, DAMAGES, AND EXPENSES AND REASONABLE AT- TORNEYS' FEES ARISING OUT OF, DIRECTLY OR INDIRECTLY, ANY CLAIM OF PRODUCT LIABILITY, PERSONAL INJURY, OR DEATH ARISING IN ANY WAY OUT OF SUCH MISSION CRITICAL APPLICATION, WHETHER OR NOT INTEL OR ITS SUBCON- TRACTOR WAS NEGLIGENT IN THE DESIGN, MANUFACTURE, OR WARNING OF THE INTEL PRODUCT OR ANY OF ITS PARTS.
[Show full text]
Error Propagation Analysis for File Systems!
Error Propagation Analysis for File Systems! Cindy Rubio-González, Haryadi S. Gunawi, Ben Liblit, Remzi H. Arpaci-Dusseau, and Andrea C. Arpaci-Dusseau! University of Wisconsin-Madison PLDI’09 ECS 289C Seminar on Program Analysis February 17th, 2015 Motivation! • Systems software plays an important role! o Designed to operate the computer hardware! o Provides platform for running application software! • Particular focus on file systems! o Store massive amounts of data! o Used everywhere!! § Home use: photos, movies, tax returns! § Servers: network file servers, search engines! • Incorrect error handling → serious consequences! o Silent data corruption, data loss, system crashes, etc.! • Broken systems software → Broken user applications!! ! 2 Error Handling! Error Propagation + Error Recovery! USER-LEVEL APPLICATION SYSTEMS SOFTWARE Error Recovery:! ERROR ERROR • Logging! • Notifying! • Re-trying! Run-time ERROR ERROR Errors! HARDWARE Incorrect error handling → longstanding problem! 3 Return-Code Idiom! • Widely used in systems software written in C! • Also used in large C++ applications! • Run-time errors represented as integer values! • Propagated through variable assignments and function return values! 4 Error Codes in Linux! 5 Example of Error-Propagation Bug! 1 int txCommit(...) { 2 ... 3 if (isReadOnly(...)) { 4 rc = -EROFS; read-only file-system error! 5 ... 6 goto TheEnd; 7 } ... 8 if (rc = diWrite(...)) diWrite may return EIO error! 9 txAbort(...); 10 TheEnd: return rc; function returns variable rc 11 } 13 int diFree(...) { 14 ..
[Show full text]
Authenticated and Encypted Storage on Embedded Linux
Authenticated and Encrypted Storage on Embedded Linux ELC Europe 2019 Jan Lübbe – [email protected] https://www.pengutronix.de Linux Storage Stack 2/21 Transparent Authentication and Encryption 3/21 Crypto‽ https://www.pengutronix.de https://www.instructables.com/id/Laser-Cut-Cryptex/ Quick Crypto Refresher Hash: one-way function, fixed output size (SHA*) HMAC: data authentication using hash and shared secret Signature: data authentication using public key cryptography (keys & certificates, RSA & ECDSA) Unauthenticated encryption: attacker can‘t read private data, but could modify it (AES-CBC, AES-XTS, …) Authenticated encryption: attacker can‘t read private data and modification is detected (AEAD: AES GCM, AEGIS) 5/21 Overview Building Blocks authentication encryption authenticated encryption General Considerations 6/21 dm-verity (since 2012, v3.4) filesystem authentication via hash tree: read-only dm-verity hash used by Chrome OS & Android for rootfs tree root hash provided via out-of-band (kernel cmdline) or via signature in super block (since 5.4) can be created and configured via veritysetup (LUKS2) combine with ext4, SquashFS or EROFS hash-tree image ⇒ best choice for RO data 7/21 fsverity (since 2019, v5.4) “dm-verity for files”: efficient authentication of (large) read- only files via a hash tree root hash provided out-of-band integrated into ext4 could be integrated with IMA/EVM to improve performance ⇒ Android will likely be the main user (for .apk authentication) 8/21 dm-integrity (since 2017, v4.12) emulates
[Show full text]
Man Pages Section 7 Device and Network Interfaces
man pages section 7: Device and Network Interfaces Beta Part No: 819–2254–33 February 2010 Copyright ©2010 Sun Microsystems, Inc. All Rights Reserved 4150 Network Circle, Santa Clara, CA 95054 U.S.A. All rights reserved. Sun Microsystems, Inc. has intellectual property rights relating to technology embodied in the product that is described in this document. In particular, and without limitation, these intellectual property rights may include one or more U.S. patents or pending patent applications in the U.S. and in other countries. U.S. Government Rights – Commercial software. Government users are subject to the Sun Microsystems, Inc. standard license agreement and applicable provisions of the FAR and its supplements. This distribution may include materials developed by third parties. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, the Sun logo, the Solaris logo, the Java Coffee Cup logo, docs.sun.com, Java, and Solaris are trademarks or registered trademarks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and SunTM Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry.
[Show full text]
Cross-Checking Semantic Correctness: the Case of Finding File System Bugs
Cross-checking Semantic Correctness: The Case of Finding File System Bugs Paper #171 Abstract 1. Introduction Today, systems software is too complex to be bug-free. To System software is buggy. It is often implemented in un- find bugs in systems software, developers often rely oncode safe languages (e.g., C) for achieving better performance or checkers, like Sparse in Linux. However, the capability of directly accessing the hardware, thereby facilitating the intro- existing tools used in commodity, large-scale systems is duction of tedious bugs. At the same time, it is also complex. limited to finding only shallow bugs that tend to be introduced For example, Linux consists of 17 millions lines of pure code by the simple mistakes of programmers and require no deep and accepts around 190 commits every day. understanding of code. Unfortunately, a majority of and To help this situation, researchers often use memory-safe difficult-to-find bugs are semantic ones, which violate high- languages in the first place. For example, Singularity [32] level rules or invariants (e.g., missing a permission check). uses C# and Unikernel [40] use OCaml for their OS develop- Thus, it is difficult for code-checking tools that lack the ment. However, in practice, developers usually rely on code understanding of a programmer’s true intention, to reason checkers. For example, Linux has integrated static code anal- about semantic correctness. ysis tools (e.g., Sparse) in its build process to detect common To solve this problem, we present JUXTA, a tool that au- coding errors (e.g., if system calls validate arguments that tomatically infers high-level semantics directly from source come from userspace).
[Show full text]
Device and Network Interfaces
man pages section 7: Device and Network Interfaces Sun Microsystems, Inc. 4150 Network Circle Santa Clara, CA 95054 U.S.A. Part No: 816–5223–10 December 2002 Copyright 2002 Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 U.S.A. All rights reserved. This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. No part of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any. Third-party software, including font technology, is copyrighted and licensed from Sun suppliers. Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S. and other countries, exclusively licensed through X/Open Company, Ltd. Sun, Sun Microsystems, the Sun logo, docs.sun.com, AnswerBook, AnswerBook2, and Solaris are trademarks, registered trademarks, or service marks of Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarks of SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by Sun Microsystems, Inc. The OPEN LOOK and Sun™ Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges the pioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds a non-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun’s licensees who implement OPEN LOOK GUIs and otherwise comply with Sun’s written license agreements.
[Show full text]
MSL C Reference 9.0
MSL C Reference 9.0 Revised <Date> Revised <Date> Metrowerks and the Metrowerks logo are registered trademarks of Metrowerks Corporation in the United States and/or other countries. CodeWarrior is a trademark or registered trademark of Metrowerks Corporation in the United States and/or other countries. All other trade names and trademarks are the property of their respective owners. Copyright © 2004 Metrowerks Corporation. ALL RIGHTS RESERVED. No portion of this document may be reproduced or transmitted in any form or by any means, electronic or me- chanical, without prior written permission from Metrowerks. Use of this document and related materials are governed by the license agreement that accompanied the product to which this manual pertains. This document may be printed for non-commercial personal use only in accordance with the aforementioned license agreement. If you do not have a copy of the license agreement, contact your Metrowerks representative or call 1-800-377- 5416 (if outside the U.S., call +1-512-996-5300). Metrowerks reserves the right to make changes to any product described or referred to in this document without further notice. Metrowerks makes no warranty, representation or guarantee regarding the merchantability or fitness of its prod- ucts for any particular purpose, nor does Metrowerks assume any liability arising out of the application or use of any product described herein and specifically disclaims any and all liability. Metrowerks software is not authorized for and has not been designed, tested, manufactured, or intended for use in developing applications where the fail- ure, malfunction, or any inaccuracy of the application carries a risk of death, serious bodily injury, or damage to tangible property, including, but not limited to, use in factory control systems, medical devices or facilities, nuclear facilities, aircraft navigation or communication, emergency systems, or other applications with a similar degree of potential hazard.
[Show full text]
Plumbers Report
Status of Embedded Linux Linux Plumbers Conference Report and Community Status Update September 2019 Tim Bird Sr. Staff Software Engineer, Sony Electronics 1 110/23/2014 PA1 Confidential What this talk covers • Report on recent conferences • ELC North America • Linux Plumbers Conference • CKI Hackfest • Community discussions • Mostly from ksummit-discuss • Recent kernel releases • A few miscellaneous notes • Not comprehensive! • Just stuff that I saw 2 210/23/2014 PA1 Confidential Outline ELC Hot maintainer topics Plumbers CKI hackfest Kernel releases Miscellany 3 310/23/2014 PA1 Confidential Outline ELC Hot maintainer topics Plumbers CKI hackfest Kernel releases Miscellany 4 410/23/2014 PA1 Confidential ELC report • August 21-23 in San Diego, California • My report: • Sessions I saw • Discussions I had • General impressions • For most presentations see: • https://elinux.org/ELC_2019_Presentations 510/23/2014 PA1 Confidential Sessions I attended • You might be a maintainer and not know it • by Frank Rowand • Regression Testing with Fuego • by Hirotaka Motai • gcc/clang optimizations for embedded • by Khem Raj • CVE Monitoring and Management • by Ashkay Bhat • Testing Laboratory API • by Pawer Wieczorek 610/23/2014 PA1 Confidential Sessions (cont.) • Open Source license variations in Linux and Android (OSS NA session) • by Peter Shin • Toybox vs. Busybox • by Rob Landley • Creating a BT PAN/USB RNDIS Router using OpenWrt • By Koichi Okamoto & Masayuki Ishikawa • USB arsenal for the masses • by Krzysztof Opasiak 710/23/2014 PA1 Confidential
[Show full text]
System Interface Module Manual
Oracle® Hospitality Simphony System Interface Module (SIM) Manual Release 18.1 E95482-01 July 2018 Copyright © 2007, 2018, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this software or related documentation is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government. This software or hardware is developed for general use in a variety of information management applications.
[Show full text]