DOCSLIB.ORG

EROFS File System Introduction

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
EROFS File System Introduction EROFS file system Gao Xiang <[email protected]> Self-introduction & Background 1. Graduated from university and joined HUAWEI for 3 years, working for OS !ernel Lab %ept., mainly fo'using on #inux )"e system ●im&rovement for HUAWEI mo$ile phones+ ● ,-, su&&ort . bugfi(/ New solutions of requirements from our &rodu'ts+ 12HUAWEI sd'ardfs; -2ERO, / 32Other useful so"utions w*i'* are under deve"o&ment... -. User availa$"e s&ace becomes tig*t wit* t*e increase of Android RO4, especial"y for our low5end devices (some of t*em are stil" wit* 17GB tota" storage, saving 19-GB more storage spa'e is usefu" for users wit* t*e hel& of ERO, 2/ 3. ome solution to leave more spa'e and maintain *ig*5 &erforman'e to end users: 3 Questions 1. What kind of fi"e types 'an be 'ompressed pra'ti'a""y? and w*ere are t*ey stored in: ❌ a. <*otos, <i'tures,❓ 4ovies? 6no gain to 'om&ress "oss"ess"y again2 $. =e(t materia"?❌ ( 'ompressible $ut t*e total amount de&ends on end-user be*aviors2 c. %atabase: ( some &art of overwrite IO &atterns 'ou"d ki"" t*e✔️ &erforman'e 2 d. 8I0s, s*ared "ibrary, some A<!s, &re"oad con)guration )"es? (most"y read5on"y )"es2 -. Is t*ere some so"ution to reso"ve our 'om&ression re1uirements: s1uas*fs is not good enoug* for real5time de'om&ression (es&ecia""y for em$edded devices wit* "imited memory, su'* as mo$ile &*ones2 sin'e 1. e(tra memory over*ead/ 2. noti'eab"e read am&"ifi'ation 6> based on its default 12?!8 $"ock size) 3. some metadata &arsing *ave to be done syn'*ronous"y limited by its on5disk design. 3. =*e &erformance 'an be even better wit* 'om&ressing effectively in addition to reso"ving t*e above squashfs issues, as shown in the fo""owing pages. ; Fixed-output compression Advantage: 1. improved storage density (hig* C3 -D less IO for most patterns -D read performan'e improvement2/ -. decompression in5&"a'e (no mem'&y2 com&ared with )(ed-input 'om&a'ted compression/ 3. 'an be on5disk com&atib"e wit* b"o'k boundary a"igned compression (eg. 8trfs2 in order to ar'hive zero IO read am&"i)'ation (However, in my opinion, it depends on a'tua" IO patterns and "ess usefu" for fi(ed-output decompression sin'e a"" compressed data in t*e b"o'k can be compressed.2 #m LmE1 #m+- #mE3 #m #m+1 #mE- #m+3 #m #mE1 #mE- <n <nE1 <nE- <nE3 Pn PnE1 <nE- <nE3 Pn Pn+1 PnE- blo'k $oundary 'om&ression 6$trfs2 )(ed5in&ut 'om&a'ted com&ression (s1uashfs, 'ramfs2 it is only useful if C3 DD - 6more t*an BFG oA2, )(ed5out&ut 'om&ression 6erofs2 it is toug* for ;k 'om&ress $lo'k B Comparsion of erofs/squashfs/btrfs image sizes Support 4k compressed cluster only in current erofs kernel implementation due to our requirement: dataset testcase output size CR enwikH enwikH 1,FFF,FFF,FFF8 1 enwikHI;k.s1uashfs.img 7-1,-11,7;?8 1.71 enwik9_4k.erofs.img 558,133,248 1.79 enwikHI?k.s1uashfs.img BB7,1H1,N;;8 1.?F enwikHI1-?k.s1uashfs.img 3H?,-F;,H-?8 -.B1 657,608,704 enwikHI1-?k.$trfs.img> 8 1.B- si"esia.tar si"esia.tar -11,HBN,N7F8 1 si"esia.tarI;k.s1uas*fs.img 11;,B-;,17F8 1.?B si"esiaI?k.s1uashfs.img 1F7,FH;,BH-8 -.FF si"esia.tar_4k.erofs.img 1#5,2$!,2## 2.01 si"esia.tarI1-?k.s1uashfs.img ?1,N7?,;;?8 -.BH 2.03 si"esia.tarI1-?k.$trfs.img> 1F;,7-?,--;8 'om&ressed wit* "@; 1.8.3 with 'ommand "ines+ 12 mksquashfs enwikH enwikHI;k.s1uashfs.img 5'omp "@; 5J*' 5$ ;FH7 5noappend -2 mkfs.erofs 5@"@;*' enwikHI;k.erofs.img enwikH > for $trfs, mounting with Knodatasum,'om&ress5for'eL"@o”, observed with 'ompsi@e, thus no metadata at all. 7 Fixed-output compression (cont.) =ypica""y, t*ere are 2 ways for compress file system to decom&ress+ 1.read a"" 'om&ressed data into bdev page 'a'*e (like all metadata do2 and decompress O squas*fs/ 2.read com&ressed data to tem&orary $uAer and de'om&ress ⇒ e.g., $trfs. ●However, bot* 2 ways takes e(tra over*ead sin'e 3e'"aimed ,or 1), it 'ou"d 'ause &age 'a'*e t*ras*ing, imagine a$out B0% 'om&ressed data of the origina" )"e si@e are added into &age 'a'*e ina'tive #3U "ist at "east for em$eded devi'es, 'om&ressed $"ock needed to read many of t*em are used5once or at a very "ow fre1uency/ furt*ermore, it’s *ard to de'om&ress dire'tly if some of 'om&ressed &ages are re'"aimed. ● ,or -), it 'an *ard"y use t*e remaining 'om&ressed data since tem&orary $uAer 'ou"d $e freed immediate"y. ,ixed-output compression can make fu"" use of data in 'ompressed b"o'k in prin'ip"e (a"" com&ressed data can be de'ompressed if needed.2 N Our solution ERO, *as de'om&ression strategies in - dimensions+ %Dimension 1' Cached or In5&"a'e IO decom&ression ● Ca'*ed de'om&ression 'urrently for in'om&"ete 'om&ression read / ● In5&"a'e IO de'om&ression for 'om&"ete 'om&ression read. KComp"ete or in'omp"ete” means whether or not data in the 'ompressed $"ock are a"" re1uested. =*e &oli'y 'an $e re)ned "ater since for sma"" C3 it 'an $e fu""y de'om&ressed in advance since it takes t*e simi"ar amount of memory if t*e 'om&ressed data are 'a'*ed in memory. %Dimension 2' yn' or async de'om&ression ● yn' de'om&ression for sma"" num$er of &age re1uests, w*i'* de'om&ress in its 'a""er thread/ ● Asyn' de'om&ression for "arge read re1uests, w*i'* de'om&ress in work1ueue 'onte(t. =*e &oli'y 'an $e re)ned "ater as we"", since erofs 'annot make diAerence $etween asyn'Qsyn' reada*ead in .read&ages(). ? In-place IO decompression Why we need t*is? main"y for t*ree reasons+ 1.doing ca'hed decompression on"y cou"d cause ca'*e t*rashing/ -.'onsidering many read requests are pending for IO ending, temporary $uAers cannot be freed. A""o'ating immediate buAers wil" cause more memory rec"aim as wel" com&ared to un'ompressed generic file system su'* as e(t; O espe'ia""y for HUAWEI camera heavy memory work"oad/ 3.it wil" be later used for decompressing in5&"a'e. In this way, compressed data are read in its last decompressed pages as mu'h as possib"e, as il"ustrated below+ )"e &ages #m #m+1 #mE- R Ln68&P2 %ire'tly read 8& into #n 'om&ressed '"usters 8& H Decompression In-place As mentioned before, decom&ression in5&"a'e can be im&"emented wit* fixed- out&ut de'ompression, w*ich is hard for lega'y fixed-in&ut de'ompression. some margin )"e &ages #m #m+1 R Ln68&P2 IO submission read 8& dire'tly into #n 'om&ressed $"ocks 8& <er5C<U some margin buAer de'om&ress input decompression #m #m+1 R Ln68&P2 )"e &ages #m #m+1 R Ln68&P2 de'om&ress output 1F Limited bounced buffers in'e a"" "@5$ased compression a"gorit*ms use s"iding5window tec*no"ogy, E3OF can use limited (7;!8 for l@;2 boun'ed $uAers, whic* minimi@e memory 'onsum&tion as wel". Limited $oun'ed $uffers is a"so im&"emented in new decompression ba'kend. before #m #m+1 #mE- #m+3 R #m+17 #mE18 #mE19 R Ln68&P2 after #m #mE1 #mE- #mE3 R #m+1N #m+18 #m+19 R #n68&P2 11 On-disk brief introduction • • #ittle5endian • 4k $lock si@e 'urrently 6no$*2/ 4i(ed metadata with data 6In other words, • Se(ible enough for mkfs to &lay with it)/ • 32 6v12 or 7; 6v-25$yte inode $ase si@e/ 7; bit + ns timestam&, ea'* )"e 'an *ave • its own timestam& for v-/ • u&&ort tai"5end data in"ine; • u&&ort JA==R, PO IXAC# 65.-E2/ • u&&ort stat( 65.3E2/ • u&&ort 'om&acted inde(es 6B.3+)/ *ttps:QQgit.kernel.orgQ&u$Qs'mQlinu(Qkerne"QgitQtorvalds/linu(.gitQtreeQdrivers/ stagingQerofs/%ocumentationQ)lesystems/erofs.t(t:*Lv5.1 1- Microbenchmark C<U: Intel632 Core(=42 i55?-BFU C<U @ 1.6FGHz (; cores, 8 t*reads2 %%3: 8G D: I0TEL S %<E!!,37FGNHenwik9 FIO (psync, 1 thread) 800 700 600 500 erofs_4k squashfs_4k squashfs_8k 400 squashfs_16k squashfs_128k 300 200 100 0 seq rand rand1m 13 Real app launching C<U: M=7N7B (? Corte(5AB3 cores2 D%3: 2G8 eMMC: 3-G8 App. # 1 2 3 4 5 6 7 8 9 10 11 12 13 w/o FIO workload -16.4 -3.5 +4.2 -4.0 -7.5 -1.4 -6.8 +6.3 -2.2 -18.4 -3.3 -7.6 -4.5 w/ FIO workload -2.8 -12.9 -5.4 +3.9 -7.6 +3.7 +4.4 -2.6 +9.9 +4.0 -11.1 -10.3 -15.1 Relative boot time of thirteen applications.
Load more

Recommended publications
  • Huawei Announces EROFS Linux File-System, Might Eventually Be Used
    ARTICLES & REVIEWS NEWS ARCHIVE FORUMS PREMIUM CATEGORIES Custom Search Search Latest Linux News Huawei Announces EROFS Linux File-System, Might Huawei Announces EROFS Linux File- Eventually Be Used By Android Devices System, Might Eventually Be Used By Android Devices Written by Michael Larabel in Linux Storage on 31 May 2018 at 09:00 AM EDT. 3 Comments Mesa 18.0.5 Is The Last Planned Release In Huawei's Gao Xiang has announced the EROFS open-source Linux file-system The Series intended for Android devices, but still at its very early stages of AMD K8 Support Stripped Out Of Coreboot development. NVIDIA’s Next Generation Mainstream GPU Will At Least Be Detailed In August EROFS is the company's new approach for a read-only file-system that would work well for Android devices. EROFS is short for the Extendable Read-Only GNOME 3 Might Be Too Resource Hungry To File-System and they began developing it with being unsatisfied with other read-only file- Ever Run Nicely On The Raspberry Pi system alternatives. XWayland Gets Patch To Automatically Use EGLStreams For NVIDIA Support When EROFS is designed to offer better performance than other read-only alternatives while still Needed focusing upon saving storage space. As part of EROFS is also a compression mode pursuing BPFILTER Landing For Linux 4.18 For a different design approach than other file-systems: the compression numbers shared in Eventually Better Firewall / Packet Filtering today's announcement on both server hardware and a Kirin 970 are compelling for being in AMDGPU Patches Prepping JPEG Support For the early stages of development.
    [Show full text]
  • Study of File System Evolution
    Study of File System Evolution Swaminathan Sundararaman, Sriram Subramanian Department of Computer Science University of Wisconsin {swami, srirams} @cs.wisc.edu Abstract File systems have traditionally been a major area of file systems are typically developed and maintained by research and development. This is evident from the several programmer across the globe. At any point in existence of over 50 file systems of varying popularity time, for a file system, there are three to six active in the current version of the Linux kernel. They developers, ten to fifteen patch contributors but a single represent a complex subsystem of the kernel, with each maintainer. These people communicate through file system employing different strategies for tackling individual file system mailing lists [14, 16, 18] various issues. Although there are many file systems in submitting proposals for new features, enhancements, Linux, there has been no prior work (to the best of our reporting bugs, submitting and reviewing patches for knowledge) on understanding how file systems evolve. known bugs. The problems with the open source We believe that such information would be useful to the development approach is that all communication is file system community allowing developers to learn buried in the mailing list archives and aren’t easily from previous experiences. accessible to others. As a result when new file systems are developed they do not leverage past experience and This paper looks at six file systems (Ext2, Ext3, Ext4, could end up re-inventing the wheel. To make things JFS, ReiserFS, and XFS) from a historical perspective worse, people could typically end up doing the same (between kernel versions 1.0 to 2.6) to get an insight on mistakes as done in other file systems.
    [Show full text]
  • F2FS) Overview
    Flash Friendly File System (F2FS) Overview Leon Romanovsky [email protected] www.leon.nu November 17, 2012 Leon Romanovsky [email protected] F2FS Overview Disclaimer Everything in this lecture shall not, under any circumstances, hold any legal liability whatsoever. Any usage of the data and information in this document shall be solely on the responsibility of the user. This lecture is not given on behalf of any company or organization. Leon Romanovsky [email protected] F2FS Overview Introduction: Flash Memory Definition Flash memory is a non-volatile storage device that can be electrically erased and reprogrammed. Challenges block-level access wear leveling read disturb bad blocks management garbage collection different physics different interfaces Leon Romanovsky [email protected] F2FS Overview Introduction: Flash Memory Definition Flash memory is a non-volatile storage device that can be electrically erased and reprogrammed. Challenges block-level access wear leveling read disturb bad blocks management garbage collection different physics different interfaces Leon Romanovsky [email protected] F2FS Overview Introduction: General System Architecture Leon Romanovsky [email protected] F2FS Overview Introduction: File Systems Optimized for disk storage EXT2/3/4 BTRFS VFAT Optimized for flash, but not aware of FTL JFFS/JFFS2 YAFFS LogFS UbiFS NILFS Leon Romanovsky [email protected] F2FS Overview Background: LFS vs. Unix FS Leon Romanovsky [email protected] F2FS Overview Background: LFS Overview Leon Romanovsky [email protected] F2FS Overview Background: LFS Garbage Collection 1 A victim segment is selected through referencing segment usage table. 2 It loads parent index structures of all the data in the victim identified by segment summary blocks.
    [Show full text]
  • In Search of Optimal Data Placement for Eliminating Write Amplification in Log-Structured Storage
    In Search of Optimal Data Placement for Eliminating Write Amplification in Log-Structured Storage Qiuping Wangy, Jinhong Liy, Patrick P. C. Leey, Guangliang Zhao∗, Chao Shi∗, Lilong Huang∗ yThe Chinese University of Hong Kong ∗Alibaba Group ABSTRACT invalidated by a live block; a.k.a. the death time [12]) to achieve Log-structured storage has been widely deployed in various do- the minimum possible WA. However, without obtaining the fu- mains of storage systems for high performance. However, its garbage ture knowledge of the BIT pattern, how to design an optimal data collection (GC) incurs severe write amplification (WA) due to the placement scheme with the minimum WA remains an unexplored frequent rewrites of live data. This motivates many research stud- issue. Existing temperature-based data placement schemes that ies, particularly on data placement strategies, that mitigate WA in group blocks by block temperatures (e.g., write/update frequencies) log-structured storage. We show how to design an optimal data [7, 16, 22, 27, 29, 35, 36] are arguably inaccurate to capture the BIT placement scheme that leads to the minimum WA with the fu- pattern and fail to group the blocks with similar BITs [12]. ture knowledge of block invalidation time (BIT) of each written To this end, we propose SepBIT, a novel data placement scheme block. Guided by this observation, we propose SepBIT, a novel data that aims to minimize the WA in log-structured storage. It infers placement algorithm that aims to minimize WA in log-structured the BITs of written blocks from the underlying storage workloads storage.
    [Show full text]
  • ECE 598 – Advanced Operating Systems Lecture 19
    ECE 598 { Advanced Operating Systems Lecture 19 Vince Weaver http://web.eece.maine.edu/~vweaver [email protected] 7 April 2016 Announcements • Homework #7 was due • Homework #8 will be posted 1 Why use FAT over ext2? • FAT simpler, easy to code • FAT supported on all major OSes • ext2 faster, more robust filename and permissions 2 btrfs • B-tree fs (similar to a binary tree, but with pages full of leaves) • overwrite filesystem (overwite on modify) vs CoW • Copy on write. When write to a file, old data not overwritten. Since old data not over-written, crash recovery better Eventually old data garbage collected • Data in extents 3 • Copy-on-write • Forest of trees: { sub-volumes { extent-allocation { checksum tree { chunk device { reloc • On-line defragmentation • On-line volume growth 4 • Built-in RAID • Transparent compression • Snapshots • Checksums on data and meta-data • De-duplication • Cloning { can make an exact snapshot of file, copy-on- write different than link, different inodles but same blocks 5 Embedded • Designed to be small, simple, read-only? • romfs { 32 byte header (magic, size, checksum,name) { Repeating files (pointer to next [0 if none]), info, size, checksum, file name, file data • cramfs 6 ZFS Advanced OS from Sun/Oracle. Similar in idea to btrfs indirect still, not extent based? 7 ReFS Resilient FS, Microsoft's answer to brtfs and zfs 8 Networked File Systems • Allow a centralized file server to export a filesystem to multiple clients. • Provide file level access, not just raw blocks (NBD) • Clustered filesystems also exist, where multiple servers work in conjunction.
    [Show full text]
  • Open Source Licensing Information for Cisco IP Phone 8800 Series
    Open Source Used In Cisco IP Phone 8800 Series 12.1(1) Cisco Systems, Inc. www.cisco.com Cisco has more than 200 offices worldwide. Addresses, phone numbers, and fax numbers are listed on the Cisco website at www.cisco.com/go/offices. Text Part Number: 78EE117C99-163803748 Open Source Used In Cisco IP Phone 8800 Series 12.1(1) 1 This document contains licenses and notices for open source software used in this product. With respect to the free/open source software listed in this document, if you have any questions or wish to receive a copy of any source code to which you may be entitled under the applicable free/open source license(s) (such as the GNU Lesser/General Public License), please contact us at [email protected]. In your requests please include the following reference number 78EE117C99-163803748 Contents 1.1 bluez 4.101 :MxC-1.1C R4.0 1.1.1 Available under license 1.2 BOOST C++ Library 1.63.0 1.2.1 Available under license 1.3 busybox 1.21.0 1.3.1 Available under license 1.4 Busybox 1.23.1 1.4.1 Available under license 1.5 cjose 0.4.1 1.5.1 Available under license 1.6 cppformat 2.0.0 1.6.1 Available under license 1.7 curl 7.26.0 1.7.1 Available under license 1.8 dbus 1.4.1 :MxC-1.1C R4.0 1.8.1 Available under license 1.9 DirectFB library and utilities 1.4.5 1.9.1 Available under license 1.10 dnsmasq 2.46 1.10.1 Available under license 1.11 flite 2.0.0 1.11.1 Available under license 1.12 glibc 2.13 1.12.1 Available under license 1.13 hostapd 2.0 :MxC-1.1C R4.0 1.13.1 Available under license Open Source Used
    [Show full text]
  • De-Anonymizing Live Cds Through Physical Memory Analysis
    De-Anonymizing Live CDs through Physical Memory Analysis Andrew Case [email protected] Digital Forensics Solutions Abstract Traditional digital forensics encompasses the examination of data from an offline or “dead” source such as a disk image. Since the filesystem is intact on these images, a number of forensics techniques are available for analysis such as file and metadata examination, timelining, deleted file recovery, indexing, and searching. Live CDs present a serious problem for this investigative model, however, since the OS and applications execute in a RAM-only environment and do not save data on non-volatile storage devices such as the local disk. In order to solve this problem, we present a number of techniques that support complete recovery of a live CD’s in-memory filesystem and partial recovery of its deleted contents. We also present memory analysis of the popular Tor application, since it is used by a number of live CDs in an attempt to keep network communications encrypted and anonymous. 1 Introduction Traditional digital forensics encompasses the examination of data from an offline or “dead” source such as a disk image. Under normal circumstances, evidence is obtained by first creating an exact, bit-for-bit copy of the target disk, followed by hashing of both the target disk and the new copy. If these hashes match then it is known that an exact copy has been made, and the hash is recorded to later prove that evidence was not modified during the investigation. Besides satisfying legal requirements, obtaining a bit-for-bit copy of data provides investigators with a wealth of information to examine and makes available a number of forensics techniques.
    [Show full text]
  • Hardware-Driven Evolution in Storage Software by Zev Weiss A
    Hardware-Driven Evolution in Storage Software by Zev Weiss A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy (Computer Sciences) at the UNIVERSITY OF WISCONSIN–MADISON 2018 Date of final oral examination: June 8, 2018 ii The dissertation is approved by the following members of the Final Oral Committee: Andrea C. Arpaci-Dusseau, Professor, Computer Sciences Remzi H. Arpaci-Dusseau, Professor, Computer Sciences Michael M. Swift, Professor, Computer Sciences Karthikeyan Sankaralingam, Professor, Computer Sciences Johannes Wallmann, Associate Professor, Mead Witter School of Music i © Copyright by Zev Weiss 2018 All Rights Reserved ii To my parents, for their endless support, and my cousin Charlie, one of the kindest people I’ve ever known. iii Acknowledgments I have taken what might be politely called a “scenic route” of sorts through grad school. While Ph.D. students more focused on a rapid graduation turnaround time might find this regrettable, I am glad to have done so, in part because it has afforded me the opportunities to meet and work with so many excellent people along the way. I owe debts of gratitude to a large cast of characters: To my advisors, Andrea and Remzi Arpaci-Dusseau. It is one of the most common pieces of wisdom imparted on incoming grad students that one’s relationship with one’s advisor (or advisors) is perhaps the single most important factor in whether these years of your life will be pleasant or unpleasant, and I feel exceptionally fortunate to have ended up iv with the advisors that I’ve had.
    [Show full text]
  • NOVA: a Log-Structured File System for Hybrid Volatile/Non
    NOVA: A Log-structured File System for Hybrid Volatile/Non-volatile Main Memories Jian Xu and Steven Swanson, University of California, San Diego https://www.usenix.org/conference/fast16/technical-sessions/presentation/xu This paper is included in the Proceedings of the 14th USENIX Conference on File and Storage Technologies (FAST ’16). February 22–25, 2016 • Santa Clara, CA, USA ISBN 978-1-931971-28-7 Open access to the Proceedings of the 14th USENIX Conference on File and Storage Technologies is sponsored by USENIX NOVA: A Log-structured File System for Hybrid Volatile/Non-volatile Main Memories Jian Xu Steven Swanson University of California, San Diego Abstract Hybrid DRAM/NVMM storage systems present a host of opportunities and challenges for system designers. These sys- Fast non-volatile memories (NVMs) will soon appear on tems need to minimize software overhead if they are to fully the processor memory bus alongside DRAM. The result- exploit NVMM’s high performance and efficiently support ing hybrid memory systems will provide software with sub- more flexible access patterns, and at the same time they must microsecond, high-bandwidth access to persistent data, but provide the strong consistency guarantees that applications managing, accessing, and maintaining consistency for data require and respect the limitations of emerging memories stored in NVM raises a host of challenges. Existing file sys- (e.g., limited program cycles). tems built for spinning or solid-state disks introduce software Conventional file systems are not suitable for hybrid mem- overheads that would obscure the performance that NVMs ory systems because they are built for the performance char- should provide, but proposed file systems for NVMs either in- acteristics of disks (spinning or solid state) and rely on disks’ cur similar overheads or fail to provide the strong consistency consistency guarantees (e.g., that sector updates are atomic) guarantees that applications require.
    [Show full text]
  • Z/OS Distributed File Service Zseries File System Implementation Z/OS V1R13
    Front cover z/OS Distributed File Service zSeries File System Implementation z/OS V1R13 Defining and installing a zSeries file system Performing backup and recovery, sysplex sharing Migrating from HFS to zFS Paul Rogers Robert Hering ibm.com/redbooks International Technical Support Organization z/OS Distributed File Service zSeries File System Implementation z/OS V1R13 October 2012 SG24-6580-05 Note: Before using this information and the product it supports, read the information in “Notices” on page xiii. Sixth Edition (October 2012) This edition applies to version 1 release 13 modification 0 of IBM z/OS (product number 5694-A01) and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright International Business Machines Corporation 2010, 2012. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . xiii Trademarks . xiv Preface . .xv The team who wrote this book . .xv Now you can become a published author, too! . xvi Comments welcome. xvi Stay connected to IBM Redbooks . xvi Chapter 1. zFS file systems . 1 1.1 zSeries File System introduction. 2 1.2 Application programming interfaces . 2 1.3 zFS physical file system . 3 1.4 zFS colony address space . 4 1.5 zFS supports z/OS UNIX ACLs. 4 1.6 zFS file system aggregates. 5 1.6.1 Compatibility mode aggregates. 5 1.6.2 Multifile system aggregates. 6 1.7 Metadata cache. 7 1.8 zFS file system clones . 7 1.8.1 Backup file system . 8 1.9 zFS log files.
    [Show full text]
  • Elinos Product Overview
    SYSGO Product Overview ELinOS 7 Industrial Grade Linux ELinOS is a SYSGO Linux distribution to help developers save time and effort by focusing on their application. Our Industrial Grade Linux with user-friendly IDE goes along with the best selection of software packages to meet our cog linux Qt LOCK customers needs, and with the comfort of world-class technical support. ELinOS now includes Docker support Feature LTS Qt Open SSH Configurator Kernel embedded Open VPN in order to isolate applications running on the same system. laptop Q Bug Shield-Virus Docker Eclipse-based QEMU-based Application Integrated Docker IDE HW Emulators Debugging Firewall Support ELINOS FEATURES MANAGING EMBEDDED LINUX VERSATILITY • Industrial Grade Creating an Embedded Linux based system is like solving a puzzle and putting • Eclipse-based IDE for embedded the right pieces together. This requires a deep knowledge of Linux’s versatility Systems (CODEO) and takes time for the selection of components, development of Board Support • Multiple Linux kernel versions Packages and drivers, and testing of the whole system – not only for newcomers. incl. Kernel 4.19 LTS with real-time enhancements With ELinOS, SYSGO offers an ‘out-of-the-box’ experience which allows to focus • Quick and easy target on the development of competitive applications itself. ELinOS incorporates the system configuration appropriate tools, such as a feature configurator to help you build the system and • Hardware Emulation (QEMU) boost your project success, including a graphical configuration front-end with a • Extensive file system support built-in integrity validation. • Application debugging • Target analysis APPLICATION & CONFIGURATION ENVIRONMENT • Runs out-of-the-box on PikeOS • Validated and tested for In addition to standard tools, remote debugging, target system monitoring and PowerPC, x86, ARM timing behaviour analyses are essential for application development.
    [Show full text]
  • A Study of Failure Recovery and Logging of High-Performance Parallel File Systems
    1 A Study of Failure Recovery and Logging of High-Performance Parallel File Systems RUNZHOU HAN, OM RAMESHWAR GATLA, MAI ZHENG, Iowa State University JINRUI CAO, State University of New York at Plattsburgh DI ZHANG, DONG DAI, North Carolina University at Charlotte YONG CHEN, Texas Tech University JONATHAN COOK, New Mexico State University Large-scale parallel file systems (PFSes) play an essential role in high performance computing (HPC). However, despite the importance, their reliability is much less studied or understood compared with that of local storage systems or cloud storage systems. Recent failure incidents at real HPC centers have exposed the latent defects in PFS clusters as well as the urgent need for a systematic analysis. To address the challenge, we perform a study of the failure recovery and logging mechanisms of PFSes in this paper. First, to trigger the failure recovery and logging operations of the target PFS, we introduce a black- box fault injection tool called PFault, which is transparent to PFSes and easy to deploy in practice. PFault emulates the failure state of individual storage nodes in the PFS based on a set of pre-defined fault models, and enables examining the PFS behavior under fault systematically. Next, we apply PFault to study two widely used PFSes: Lustre and BeeGFS. Our analysis reveals the unique failure recovery and logging patterns of the target PFSes, and identifies multiple cases where the PFSes are imperfect in terms of failure handling. For example, Lustre includes a recovery component called LFSCK to detect and fix PFS-level inconsistencies, but we find that LFSCK itself may hang or trigger kernel panicswhen scanning a corrupted Lustre.
    [Show full text]