DOCSLIB.ORG

Performance Evaluation of Linux Discard Support (Overview, Benchmark Results, Current Status)

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

Performance evaluation of Linux Discard Support (Overview, benchmark results, current status) Red Hat Luk´aˇsCzerner February 12, 2011 Copyright © 2011 Luk´aˇsCzerner, Red Hat. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.3 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the COPYING file. Part I Discard Background Agenda 1 SSD Description 2 Thinly Provisioned Storage 3 Introducing Linux Discard Support SSD Description Solid-State Drive Flash memory block device Wear-leveling needed Firmware = black box SSD Description ATA TRIM Command Helps handle garbage collection overhead Subsequent READ of TRIM'ed blocks 1 Read data should NOT change between READ's 2 Read data should NOT be retrieved from data previously written to any other LBA. As long as the device has enough free pages to write to we do not necessarily need it. In a nutshell: TRIM command tells the device what LBA'a is not used by the OS anymore. Thinly Provisioned Storage Thin Provisioning Unlike in traditional storage, there is no fixed one-to-one logical bock to physical storage mapping More efficient use of storage space Block reclamation interface needed Thinly Provisioned Storage SCSI UNMAP / WRITE SAME Storage space reclamation interface Subsequent READ of unmapped blocks 1 Read data should NOT change between READ's 2 Read data should NOT be retrieved from data previously written to any other LBA. Unlike with SSD's we can not afford to wait until we run out of space for reclamation. Introducing Linux Discard Support Linux Discard Implementation Abstraction for the two underlying specifications: 1 ATA TRIM Command 2 SCSI UNMAP / WRITE SAME API for user-space BLKDISCARD ioctl Added with v2.6.27-rc9-30-gd30a260 API for File Sytems 1 sb issue discard() 2 blkdev issue discard() Part II Discard Performance Agenda 4 Testing Methodology 5 Results Testing Methodology What do we need to find out ? Does discard really work ? Is it reliable ? How fast/slow is it ? Is there any difference between devices from different vendors ? What is the ideal discard size ? SSD performance degradation Testing Methodology How do we test it ? BLKDISCARD ioctl() Automatic discard of different ranges Different discard patterns 1 sequential performance 2 random IO peformance 3 discard already discarded blocks test-discard - discard benchmarking tool http://sourceforge.net/projects/test-discard/ impression - filesystem aging tool Results Sequential discard performance 300 1200 Duration Summary Throughput 250 1000 200 800 150 600 Duration [s] 100 400 Throughput [MB/s] 50 200 0 0 10 100 1000 Record size [kB] Results Different modes comparison 1200 Throughput (sequential) Throughput (random IO) Throughput (discard2) 1000 800 600 Throughput [MB/s] 400 200 0 10 100 1000 Record size [kB] Results Difference between various vendors 1200 Throughput Vendor 1 Throughput Vendor 2 Throughput Vendor 3 1000 800 600 Throughput [MB/s] 400 200 0 10 100 1000 Record size [kB] Results SSD performance degradation 70 Vendor1 Vendor2 Vendor3 60 Vendor3 discard 50 40 30 Write performance [MB/s] 20 10 0 50 100 150 200 250 300 350 400 Filesystem saturation [%] Part III Discard Support for Linux File systems Agenda 6 Periodic Discard 7 Discard Batching 8 Different Approach Periodic Discard Periodic discard Easy to implement File system support 1 ext4 (v2.6.27-5185-g8a0aba7) 2 btrfs (since upstream) 3 gfs2 (v2.6.29-9-gf15ab56) 4 fat, swap, nilfs mount -o discard /dev/sdc /mnt/test TRIM is non-queueable command - implications ? Periodic Discard Benchmarking periodic discard Expectations ? Testing methodology 1 Metadata intensive load 2 Load with removing files 3 Reasonable file size distribution Discard-kit 1 Using PostMark 2 http://sourceforge.net/projects/test-discard/files/ Periodic Discard Ext4 performance (18% hit) 800 Deleted/s Append/s Read/s 700 Files-created/s Transactions/s Read[B/s] 600 Write[B/s] 500 400 Opetation/s 300 200 100 nodiscard discard Periodic Discard Performance with various file systems 750 ext4 700 btrfs 650 gfs2 600 550 500 Transactions/s 450 400 350 nodiscard discard nodiscard discard nodiscard discard -18% -7% -63% Discard Batching Discard Batching - The idea Fine-grained discard is not necessarily needed Small extents are slow With time, freed extents tends to coalesce Disadvantages 1 There is a price for tracking freed extents 2 Discarding already discarded blocks should be easy, but... 3 Daemon (in-kernel, user-space) needed. 4 File system independent solution would most likely be pain to do right (if possible). Discard Batching Batched discard support File system specific solution Provide ioctl() interface - FITRIM Do not disturb other ongoing IO too much 1 Prevent allocations while trimming 2 How to handle huge filesystem ? File system support 1 ext4 (v2.6.36-rc6-35-g7360d17) 2 ext3 (v2.6.37-11-g9c52749) 3 xfs (v2.6.37-rc4-63-ga46db60) Discard Batching FITRIM ioctl Ioctl with one RW parameter defined in linux/fs.h struct fstrim range f u64 start; u64 len; u64 minlen; g fstrim tool http://sourceforge.net/projects/fstrim/ util-linux-ng Since v2.18-165-gd9e2d0d Discard Batching Batched discard benchmark results 6 FITRIM on ext4 BLKDISCARD 5 4 3 Duration [s] 2 1 0 0 20 40 60 80 100 Filesystem saturation [%] Different Approach Alternative approach It is always a compromise The future of SSD's and thinly provisioned LUN's (???) Part IV Discard Support in user-space Agenda 9 e2fsprogs 10 Other utilities e2fsprogs Discard in e2fsprogs tools Using BLKDISCARD ioctl() mke2fs 1 Refresh SSD's garbage collector 2 discard zeroes data - significant speed boost 3 mkfs.ext4 -E discard /dev/sdc e2fsck 1 After the last check discard free space 2 Non detected file system errors ? oops 3 fsck.ext4 -E discard /dev/sdc resize2fs 1 Refresh SSD's garbage collector 2 discard zeroes data - significant speed boost 3 resize2fs -E discard /dev/sdc e2fsprogs File system creation 30 nodiscard discard 25 20 15 Duration [s] 10 5 0 EXT4 XFS File system Other utilities Fstrim tool Very simple tool to invoke FITRIM ioctl on mounted file system Stand-alone tool http://sourceforge.net/projects/fstrim/ Since v2.18-165-gd9e2d0d part of util-linux-ng Part V Summary Summary Linux Discard support is a abstraction for underlying specification Exported via BLKDISCARD ioctl to user-space and blkdev issue discard() for filesystems Discard testing kit (Discard-kit) 1 test-discard 2 PostMark Filesystem support 1 Fine grained (online) discard - mount -o discard 2 Batched discard support - fstrim from util-linux-ng Support in user-space utilities 1 Filesystem creation (mkfs) 2 e2fsprogs - mkfs,e2fsck,resize2fs 3 xfsprogs - mkfs 4 fstrim The end. Thanks for listening. Useful links http://sourceforge.net/projects/fstrim/ http://sourceforge.net/projects/test-discard/ http://people.redhat.com/lczerner/discard/.
Recommended publications
  • Package 'Slurmr'

    Package 'Slurmr'

    Package ‘slurmR’ September 3, 2021 Title A Lightweight Wrapper for 'Slurm' Version 0.5-1 Description 'Slurm', Simple Linux Utility for Resource Management <https://slurm.schedmd.com/>, is a popular 'Linux' based software used to schedule jobs in 'HPC' (High Performance Computing) clusters. This R package provides a specialized lightweight wrapper of 'Slurm' with a syntax similar to that found in the 'parallel' R package. The package also includes a method for creating socket cluster objects spanning multiple nodes that can be used with the 'parallel' package. Depends R (>= 3.3.0), parallel License MIT + file LICENSE BugReports https://github.com/USCbiostats/slurmR/issues URL https://github.com/USCbiostats/slurmR, https://slurm.schedmd.com/ Encoding UTF-8 RoxygenNote 7.1.1 Suggests knitr, rmarkdown, covr, tinytest Imports utils VignetteBuilder knitr Language en-US NeedsCompilation no Author George Vega Yon [aut, cre] (<https://orcid.org/0000-0002-3171-0844>), Paul Marjoram [ctb, ths] (<https://orcid.org/0000-0003-0824-7449>), National Cancer Institute (NCI) [fnd] (Grant Number 5P01CA196569-02), Michael Schubert [rev] (JOSS reviewer, <https://orcid.org/0000-0002-6862-5221>), Michel Lang [rev] (JOSS reviewer, <https://orcid.org/0000-0001-9754-0393>) Maintainer George Vega Yon <[email protected]> Repository CRAN Date/Publication 2021-09-03 04:20:02 UTC 1 2 expand_array_indexes R topics documented: expand_array_indexes . .2 JOB_STATE_CODES . .3 makeSlurmCluster . .4 new_rscript . .6 opts_slurmR . .7 parse_flags . .9 random_job_name . .9 read_sbatch . 10 slurmR . 11 slurmr_docker . 11 slurm_available . 12 Slurm_clean . 15 Slurm_collect . 16 Slurm_env . 17 Slurm_EvalQ . 18 slurm_job . 19 Slurm_log . 21 Slurm_Map . 22 snames . 25 sourceSlurm . 25 status . 28 the_plan .
  • TEE Internal Core API Specification V1.1.2.50

    TEE Internal Core API Specification V1.1.2.50

    GlobalPlatform Technology TEE Internal Core API Specification Version 1.1.2.50 (Target v1.2) Public Review June 2018 Document Reference: GPD_SPE_010 Copyright 2011-2018 GlobalPlatform, Inc. All Rights Reserved. Recipients of this document are invited to submit, with their comments, notification of any relevant patents or other intellectual property rights (collectively, “IPR”) of which they may be aware which might be necessarily infringed by the implementation of the specification or other work product set forth in this document, and to provide supporting documentation. The technology provided or described herein is subject to updates, revisions, and extensions by GlobalPlatform. This documentation is currently in draft form and is being reviewed and enhanced by the Committees and Working Groups of GlobalPlatform. Use of this information is governed by the GlobalPlatform license agreement and any use inconsistent with that agreement is strictly prohibited. TEE Internal Core API Specification – Public Review v1.1.2.50 (Target v1.2) THIS SPECIFICATION OR OTHER WORK PRODUCT IS BEING OFFERED WITHOUT ANY WARRANTY WHATSOEVER, AND IN PARTICULAR, ANY WARRANTY OF NON-INFRINGEMENT IS EXPRESSLY DISCLAIMED. ANY IMPLEMENTATION OF THIS SPECIFICATION OR OTHER WORK PRODUCT SHALL BE MADE ENTIRELY AT THE IMPLEMENTER’S OWN RISK, AND NEITHER THE COMPANY, NOR ANY OF ITS MEMBERS OR SUBMITTERS, SHALL HAVE ANY LIABILITY WHATSOEVER TO ANY IMPLEMENTER OR THIRD PARTY FOR ANY DAMAGES OF ANY NATURE WHATSOEVER DIRECTLY OR INDIRECTLY ARISING FROM THE IMPLEMENTATION OF THIS SPECIFICATION OR OTHER WORK PRODUCT. Copyright 2011-2018 GlobalPlatform, Inc. All Rights Reserved. The technology provided or described herein is subject to updates, revisions, and extensions by GlobalPlatform.
  • Powerview Command Reference

    Powerview Command Reference

    PowerView Command Reference TRACE32 Online Help TRACE32 Directory TRACE32 Index TRACE32 Documents ...................................................................................................................... PowerView User Interface ............................................................................................................ PowerView Command Reference .............................................................................................1 History ...................................................................................................................................... 12 ABORT ...................................................................................................................................... 13 ABORT Abort driver program 13 AREA ........................................................................................................................................ 14 AREA Message windows 14 AREA.CLEAR Clear area 15 AREA.CLOSE Close output file 15 AREA.Create Create or modify message area 16 AREA.Delete Delete message area 17 AREA.List Display a detailed list off all message areas 18 AREA.OPEN Open output file 20 AREA.PIPE Redirect area to stdout 21 AREA.RESet Reset areas 21 AREA.SAVE Save AREA window contents to file 21 AREA.Select Select area 22 AREA.STDERR Redirect area to stderr 23 AREA.STDOUT Redirect area to stdout 23 AREA.view Display message area in AREA window 24 AutoSTOre ..............................................................................................................................
  • Xshell 6 User Guide Secure Terminal Emualtor

    Xshell 6 User Guide Secure Terminal Emualtor

    Xshell 6 User Guide Secure Terminal Emualtor NetSarang Computer, Inc. Copyright © 2018 NetSarang Computer, Inc. All rights reserved. Xshell Manual This software and various documents have been produced by NetSarang Computer, Inc. and are protected by the Copyright Act. Consent from the copyright holder must be obtained when duplicating, distributing or citing all or part of this software and related data. This software and manual are subject to change without prior notice for product functions improvement. Xlpd and Xftp are trademarks of NetSarang Computer, Inc. Xmanager and Xshell are registered trademarks of NetSarang Computer, Inc. Microsoft Windows is a registered trademark of Microsoft. UNIX is a registered trademark of AT&T Bell Laboratories. SSH is a registered trademark of SSH Communications Security. Secure Shell is a trademark of SSH Communications Security. This software includes software products developed through the OpenSSL Project and used in OpenSSL Toolkit. NetSarang Computer, Inc. 4701 Patrick Henry Dr. BLDG 22 Suite 137 Santa Clara, CA 95054 http://www.netsarang.com/ Contents About Xshell ............................................................................................................................................... 1 Key Functions ........................................................................................................... 1 Minimum System Requirements .................................................................................. 3 Install and Uninstall ..................................................................................................
  • BSD UNIX Toolbox 1000+ Commands for Freebsd, Openbsd

    BSD UNIX Toolbox 1000+ Commands for Freebsd, Openbsd

    76034ffirs.qxd:Toolbox 4/2/08 12:50 PM Page iii BSD UNIX® TOOLBOX 1000+ Commands for FreeBSD®, OpenBSD, and NetBSD®Power Users Christopher Negus François Caen 76034ffirs.qxd:Toolbox 4/2/08 12:50 PM Page ii 76034ffirs.qxd:Toolbox 4/2/08 12:50 PM Page i BSD UNIX® TOOLBOX 76034ffirs.qxd:Toolbox 4/2/08 12:50 PM Page ii 76034ffirs.qxd:Toolbox 4/2/08 12:50 PM Page iii BSD UNIX® TOOLBOX 1000+ Commands for FreeBSD®, OpenBSD, and NetBSD®Power Users Christopher Negus François Caen 76034ffirs.qxd:Toolbox 4/2/08 12:50 PM Page iv BSD UNIX® Toolbox: 1000+ Commands for FreeBSD®, OpenBSD, and NetBSD® Power Users Published by Wiley Publishing, Inc. 10475 Crosspoint Boulevard Indianapolis, IN 46256 www.wiley.com Copyright © 2008 by Wiley Publishing, Inc., Indianapolis, Indiana Published simultaneously in Canada ISBN: 978-0-470-37603-4 Manufactured in the United States of America 10 9 8 7 6 5 4 3 2 1 Library of Congress Cataloging-in-Publication Data is available from the publisher. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permis- sion should be addressed to the Legal Department, Wiley Publishing, Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4355, or online at http://www.wiley.com/go/permissions.
  • Filesystem Considerations for Embedded Devices ELC2015 03/25/15

    Filesystem Considerations for Embedded Devices ELC2015 03/25/15

    Filesystem considerations for embedded devices ELC2015 03/25/15 Tristan Lelong Senior embedded software engineer Filesystem considerations ABSTRACT The goal of this presentation is to answer a question asked by several customers: which filesystem should you use within your embedded design’s eMMC/SDCard? These storage devices use a standard block interface, compatible with traditional filesystems, but constraints are not those of desktop PC environments. EXT2/3/4, BTRFS, F2FS are the first of many solutions which come to mind, but how do they all compare? Typical queries include performance, longevity, tools availability, support, and power loss robustness. This presentation will not dive into implementation details but will instead summarize provided answers with the help of various figures and meaningful test results. 2 TABLE OF CONTENTS 1. Introduction 2. Block devices 3. Available filesystems 4. Performances 5. Tools 6. Reliability 7. Conclusion Filesystem considerations ABOUT THE AUTHOR • Tristan Lelong • Embedded software engineer @ Adeneo Embedded • French, living in the Pacific northwest • Embedded software, free software, and Linux kernel enthusiast. 4 Introduction Filesystem considerations Introduction INTRODUCTION More and more embedded designs rely on smart memory chips rather than bare NAND or NOR. This presentation will start by describing: • Some context to help understand the differences between NAND and MMC • Some typical requirements found in embedded devices designs • Potential filesystems to use on MMC devices 6 Filesystem considerations Introduction INTRODUCTION Focus will then move to block filesystems. How they are supported, what feature do they advertise. To help understand how they compare, we will present some benchmarks and comparisons regarding: • Tools • Reliability • Performances 7 Block devices Filesystem considerations Block devices MMC, EMMC, SD CARD Vocabulary: • MMC: MultiMediaCard is a memory card unveiled in 1997 by SanDisk and Siemens based on NAND flash memory.
  • Journaling File Systems

    Journaling File Systems

    Linux Journaling File Systems Linux onzSeries Journaling File Systems Volker Sameske ([email protected]) Linux on zSeries Development IBM Lab Boeblingen, Germany Share Anaheim,California February27 –March 4,2005 Session 9257 ©2005 IBM Corporation Linux Journaling File Systems Agenda o File systems. • Overview, definitions. • Reliability, scalability. • File system features. • Common grounds & differences. o Volume management. • LVM, EVMS, MD. • Striping. o Measurement results. • Hardware/software setup. • throughput. • CPU load. 2 Session 9257 © 2005 IBM Corporation Linux Journaling File Systems A file system should... o ...store data o ...organize data o ...administrate data o ...organize data about the data o ...assure integrity o ...be able to recover integrity problems o ...provide tools (expand, shrink, check, ...) o ...be able to handle many and large files o ...be fast o ... 3 Session 9257 © 2005 IBM Corporation Linux Journaling File Systems File system-definition o Informally • The mechanism by which computer files are stored and organized on a storage device. o More formally, • A set of abstract data types that are necessary for the storage, hierarchical organization, manipulation, navigation, access and retrieval of data. 4 Session 9257 © 2005 IBM Corporation Linux Journaling File Systems Why a journaling file system? o Imagine your Linux system crashs while you are saving an edited file: • The system crashs after the changes have been written to disk à good crash • The system crashs before the changes have been written to disk à bad crash but bearable if you have an older version • The sytem crashs just in the moment your data will be written: à very bad crash your file could be corrupted and in worst case the file system could be corrupted à That‘s why you need a journal 5 Session 9257 © 2005 IBM Corporation Linux Journaling File Systems Somefilesystemterms o Meta data • "Data about the data" • File system internal data structure (e.g.
  • State of the Art: Where We Are with the Ext3 Filesystem

    State of the Art: Where We Are with the Ext3 Filesystem

    State of the Art: Where we are with the Ext3 filesystem Mingming Cao, Theodore Y. Ts’o, Badari Pulavarty, Suparna Bhattacharya IBM Linux Technology Center {cmm, theotso, pbadari}@us.ibm.com, [email protected] Andreas Dilger, Alex Tomas, Cluster Filesystem Inc. [email protected], [email protected] Abstract 1 Introduction Although the ext2 filesystem[4] was not the first filesystem used by Linux and while other filesystems have attempted to lay claim to be- ing the native Linux filesystem (for example, The ext2 and ext3 filesystems on Linux R are when Frank Xia attempted to rename xiafs to used by a very large number of users. This linuxfs), nevertheless most would consider the is due to its reputation of dependability, ro- ext2/3 filesystem as most deserving of this dis- bustness, backwards and forwards compatibil- tinction. Why is this? Why have so many sys- ity, rather than that of being the state of the tem administrations and users put their trust in art in filesystem technology. Over the last few the ext2/3 filesystem? years, however, there has been a significant amount of development effort towards making There are many possible explanations, includ- ext3 an outstanding filesystem, while retaining ing the fact that the filesystem has a large and these crucial advantages. In this paper, we dis- diverse developer community. However, in cuss those features that have been accepted in our opinion, robustness (even in the face of the mainline Linux 2.6 kernel, including direc- hardware-induced corruption) and backwards tory indexing, block reservation, and online re- compatibility are among the most important sizing.
  • Outline of Ext4 File System & Ext4 Online Defragmentation Foresight

    Outline of Ext4 File System & Ext4 Online Defragmentation Foresight

    Outline of Ext4 File System & Ext4 Online Defragmentation Foresight LinuxCon Japan/Tokyo 2010 September 28, 2010 Akira Fujita <[email protected]> NEC Software Tohoku, Ltd. Self Introduction ▐ Name: Akira Fujita Japan ▐ Company: NEC Software Tohoku, Ltd. in Sendai, Japan. Sendai ● ▐ Since 2004, I have been working at NEC Software Tohoku developing Linux file system, mainly ext3 and ● ext4 filesystems. Tokyo Currently, I work on the quality evaluation of ext4 for enterprise use, and also develop the ext4 online defragmentation. Page 2 Copyright(C) 2010 NEC Software Tohoku, Ltd. All Rights Reserved. Outline ▐ What is ext4 ▐ Ext4 features ▐ Compatibility ▐ Performance measurement ▐ Recent ext4 topics ▐ What is ext4 online defrag ▐ Relevant file defragmentation ▐ Current status / future plan Page 3 Copyright(C) 2010 NEC Software Tohoku, Ltd. All Rights Reserved. What is ext4 ▐ Ext4 is the successor of ext3 which is developed to solve performance issues and scalability bottleneck on ext3 and also provide backward compatibility with ext3. ▐ Ext4 development began in 2006. Included in stable kernel 2.6.19 as EXPERIMENTAL (ext4dev). Since kernel 2.6.28, ext4 has been released as stable (Renamed from ext4dev to ext4 in kernel 2.6.28). ▐ Maintainers Theodore Ts'o [email protected] , Andreas Dilger [email protected] ▐ ML [email protected] ▐ Ext4 Wiki http://ext4.wiki.kernel.org Page 4 Copyright(C) 2010 NEC Software Tohoku, Ltd. All Rights Reserved. Ext4 features Page 5 Copyright(C) 2010 NEC Software Tohoku, Ltd. All Rights Reserved. Ext4 features Bigger file/filesystem size support. Compared to ext3, ext4 is: 8 times larger in file size, 65536 times(!) larger in filesystem size.
  • Lock-Free Programming

    Lock-Free Programming

    Lock-Free Programming Geoff Langdale L31_Lockfree 1 Desynchronization ● This is an interesting topic ● This will (may?) become even more relevant with near ubiquitous multi-processing ● Still: please don’t rewrite any Project 3s! L31_Lockfree 2 Synchronization ● We received notification via the web form that one group has passed the P3/P4 test suite. Congratulations! ● We will be releasing a version of the fork-wait bomb which doesn't make as many assumptions about task id's. – Please look for it today and let us know right away if it causes any trouble for you. ● Personal and group disk quotas have been grown in order to reduce the number of people running out over the weekend – if you try hard enough you'll still be able to do it. L31_Lockfree 3 Outline ● Problems with locking ● Definition of Lock-free programming ● Examples of Lock-free programming ● Linux OS uses of Lock-free data structures ● Miscellanea (higher-level constructs, ‘wait-freedom’) ● Conclusion L31_Lockfree 4 Problems with Locking ● This list is more or less contentious, not equally relevant to all locking situations: – Deadlock – Priority Inversion – Convoying – “Async-signal-safety” – Kill-tolerant availability – Pre-emption tolerance – Overall performance L31_Lockfree 5 Problems with Locking 2 ● Deadlock – Processes that cannot proceed because they are waiting for resources that are held by processes that are waiting for… ● Priority inversion – Low-priority processes hold a lock required by a higher- priority process – Priority inheritance a possible solution L31_Lockfree
  • Waste Transfer Stations: a Manual for Decision-Making Acknowledgments

    Waste Transfer Stations: a Manual for Decision-Making Acknowledgments

    Waste Transfer Stations: A Manual for Decision-Making Acknowledgments he Office of Solid Waste (OSW) would like to acknowledge and thank the members of the Solid Waste Association of North America Focus Group and the National Environmental Justice Advisory Council Waste Transfer Station Working Group for reviewing and providing comments on this draft document. We would also like to thank Keith Gordon of Weaver Boos & Gordon, Inc., for providing a technical Treview and donating several of the photographs included in this document. Acknowledgements i Contents Acknowledgments. i Introduction . 1 What Are Waste Transfer Stations?. 1 Why Are Waste Transfer Stations Needed?. 2 Why Use Waste Transfer Stations? . 3 Is a Transfer Station Right for Your Community? . 4 Planning and Siting a Transfer Station. 7 Types of Waste Accepted . 7 Unacceptable Wastes . 7 Public Versus Commercial Use . 8 Determining Transfer Station Size and Capacity . 8 Number and Sizing of Transfer Stations . 10 Future Expansion . 11 Site Selection . 11 Environmental Justice Considerations . 11 The Siting Process and Public Involvement . 11 Siting Criteria. 14 Exclusionary Siting Criteria . 14 Technical Siting Criteria. 15 Developing Community-Specific Criteria . 17 Applying the Committee’s Criteria . 18 Host Community Agreements. 18 Transfer Station Design and Operation . 21 Transfer Station Design . 21 How Will the Transfer Station Be Used? . 21 Site Design Plan . 21 Main Transfer Area Design. 22 Types of Vehicles That Use a Transfer Station . 23 Transfer Technology . 25 Transfer Station Operations. 27 Operations and Maintenance Plans. 27 Facility Operating Hours . 32 Interacting With the Public . 33 Waste Screening . 33 Emergency Situations . 34 Recordkeeping. 35 Environmental Issues.
  • Wed04-Blagodarenkoartem-Scaling Ldiskfs for the Future. Again. LUG

    Wed04-Blagodarenkoartem-Scaling Ldiskfs for the Future. Again. LUG

    Scaling LDISKFS for the future. Again Artem Blagodarenko LUG 2017 Bloomington, Indiana LUG 2017 LDISKFS still grows As drive size increases ...8TB -> 10TB -> 12TB The maximum backend storage size increases ...16TB -> 500TB LDISKFS quickly exceeded the original design! LUG 2017 The summary of previous work Done Problems ➢code review ➢ inodes➢ count over ➢ Extending inodes testing suite UINT32_MAXcount over ➢patches with fixes ➢ largeUINT32_MAX memory blocks ➢move LDISKFS size allocation➢ Large memory limit to 256TB (LU- ➢ solutionblocks for allocation large 7592). directories➢ Large directories LUG 2017 Inode count limit (LU-1365) Example: a customer requires 16 billions of inodes on MDS Unfortunately we can not make Only 4 billions of inodes on 16 billions inodes on one MDT one MDT because of LDISKFS limitation mdt0 mdt1 We can use 4 MDTs with DNE but MDT’s space is not mdt2 mdt3 completely used 16 billions >4 billions inodes on LDISKFS LUG 2017 Inode count limit. Additional fields for ext4_dir_entry Offset Size Name Description 0x0 __le32 inode Inode number 0x4 __le16 rec_len Length of this directory entry 0x6 __u8 name_len Length of the file name 0x7 __u8 file_type File type (0x0F), Dirdata (0xF0) 0x8 __u8 lufid_len OST fid length 0x9 N fid EXT4_DIRENT_LUFID 0x8 + N __u8 hi_inode_len length, always 4 0x8 + N + 1 __le64 hi_inode EXT4_DIRENT_INODE LUG 2017 dirdata pros and cons ➕ less space for 64-bit inodes ➕ smaller dirents for 32-bit inodes ➕ more 32-bit dirents in leaf block ➕ backwards compatible with existing directories ➕ doesn’t