DOCSLIB.ORG

File Systems and Storage

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

File Systems and Storage Rich Sudlow and Paul Brenner University of Notre Dame Center for Research Computing Last Modified : 06/22/11 Last Modified : 06/22/11 Overview • File System Concepts – Aspects and Types – Why do we have so many? • Redundancy and Performance – RAID – Examples on X4500 “thumper” • CRC Supported File Systems – Centralized (ext3, fat32/ntfs) – Distributed (AFS, NFS) – Comparision – capabilities • AFS – crc.nd.edu, nd.edu cells • Backup Storage – Software, Cache, Tape Silo • Using CRC Storage – Scratch space, User workspace, Backup Last Modified : 06/22/11 3 Disclaimer • This is: – A broad overview – Operational viewpoint – Starting point • This is not: – Comprehensive – Authoritative analysis of the industry/technology • For more info consider: – ND courses relevant to this topic – Contacting CRC for specific individual requirements Last Modified : 06/22/11 4 File Systems Concepts • Aspects – Filenames – Meta data (size, # blocks, time, security) – Hierarchical vs Flat – Secure access – Capabilities/Facilities (move, delete, append) – Why so many? Why not use just one? • Types – Disk/Flash – Database & Transactional – Network/Distributed Last Modified :– 06/22/11Special Purpose 5 Wikipedia – List of File Systems http://en.wikipedia.org/wiki/List_of_file_systems Last Modified : 06/22/11 6 Redundancy and Performance • File system design is strongly influenced by the target feature set – Bandwidth, security, latency, distributed access, fault tolerance, etc... • File systems can be tuned and tiered to exploit feature sets of Operating Systems and filesystems. E.g. Solaris vs Linux – Underlying ZFS or ext3 filesystem for NFS/AFS • Scalability – Ability to run across multiple nodes – by multiple users – local vs distributed Last Modified : 06/22/11 7 Wikipedia – Comparison of File Systems http://en.wikipedia.org/wiki/Comparison_of_file_systems Last Modified : 06/22/11 RAID • Redundant Array(s) of Inexpensive/Independent Disks • Utilize multiple/many disks to provide – Capacity – Reliability/Redundancy – Performance – configurable based on user requirements IOPS – bandwidth – various block sizes • Hardware and Software Implementations – Performance, Flexibility, Boot ‘chicken before egg’ • RAID ‘Levels’ – Disk utilization configurations to optimize cost vs capability • Tiered/Nested RAID Levels – One RAID level on top another raid 0+1 vs 10 Last Modified : 06/22/11 9 RAID Levels • RAID 0 – Striped : Performance and Capacity • RAID 1 – Mirrored : Read Performance and Fault Tolerance (FT) • RAID 3 & 4 – Striped with Dedicated Parity • RAID 5 – Striped with Distributed Parity : Performance, Capacity, N+1 FT • RAID 6 – Striped with Distributed Parity Performance, Capacity, N+2 FT • RAID 0+1 – Striped sets in a mirrored set • RAID 1 + 0 generally just called RAID 10 – Mirrored sets in a striped set • RAID 50 – Striped (0) Across Distributed Parity RAID (5)s Last Modified : 06/22/11 10 RAID Reference http://en.wikipedia.org/wiki/RAID Last Modified : 06/22/11 Raid example - Sun Thumper X4500 Last Modified : 06/22/11 c vicepb – single disk C5 5C4 C7 C6 C1 C0 vicepc – 2 disk stripe vicepd – 3 disk stripe vicepe – 4 disk stripe vicepf – 6 disk stripe vicepg – single disk mirror viceph – 2 disk stripe mirror vicepi – 3 disk stripe mirror vicep{ j, k, l } – 3 disk stripe – only Last Modified : 06/22/11 used for read problem encountered in multiclient test. Links to RAID examples Solaris/Red Hat on Sun X4500 (thumper) UFS tests on Solaris 10 using Sun Volume Manager http://www.nd.edu/~rich/afsbpw2007/thumper01-solaris-ufs-tests UFS tests on Linux (RH4U4) http://www.nd.edu/~rich/afsbpw2007/thumper02-linux-ext3-tests Last Modified : 06/22/11 CRC Supported File Systems Why do we have so many? • Centralized – Ext3 (Linux) – Red Hat 4 & 5 – FAT32/NTFS (Windows) • Distributed – NFS – AFS • Others – ZFS Last Modified : 06/22/11 15 CRC Supported File Systems http://www.nd.edu/~rich/CRC_filesystems.html • Scratch Space • User Workspace • Storage Backup – Available for backup of CRC and research machines on campus. Last Modified : 06/22/11 Scalability • Scalability of filesystem • Bottlenecks • Scalability of network • Scalability of codes • Simple testing tools – http://ndt.hpcc.nd.edu:7123 (simple but not always accurate) nuttcp - /opt/und/local/bin/nuttcp - firewalls nuttcp –t (-r) opteron.hpcc.nd.edu (Don’t abuse) diskrate – diskrate –n 10m –f trash Last Modified : 06/22/11 File Permissions (Linux) • What the heck does this mean? – drwxr-xr-- • File permissions for user, group, and all – 10 spaces the first indicates ‘if directory’ • Triples of rwx indicate read, write, and execute for user, group, and all • Change file permissions with ‘chmod’ – Examples: • chmod a+r filename • chmod go+w filename • chmod 1777 directory Last Modified : 06/22/11• chmod 700 directory 18 File Permissions (AFS) • fs setacl -dir $HOME -acl pat all terry none – fs is the command suite. – setacl is the operation code, which directs the File Server process to set an access control list. – -dir $HOME and -acl pat all terry none are arguments. Implies that terry previously had access – -dir and -acl are switches; -dir indicates the name of the directory on which to set the ACL, and -acl defines the entries to set on it. – $HOME and pat all terry none are instances of the arguments. $HOME defines a specific directory for the directory argument. The -acl argument has two instances specifying two ACL entries: pat all and terry none. • Command abbreviations – fs listacl (full command) , fs lista (abbreviation), fs la (alias) Last Modified : 06/22/11 19 File Permissions (AFS) AFS gives each user the permission to create their own groups Common to use syntax owner:group pts creategroup cvrl:cvrl_group pts adduser rich cvrl:cvrl_group pts membership cvrl:cvrl_group To recursively set permissions: find ./ -type d –print –exec fs setacl {} cvrl:cvrl_group read \; Special groups: system:anyuser, system:authuser, nd_campus IP based Access Control Lists Last Modified : 06/22/11 AFS References AFS Reference Links http://crcmedia.hpcc.nd.edu/wiki/index.php/AFS_References_and_Resources Some AFS / NFS Storage comparisons http://crcmedia.hpcc.nd.edu/wiki/index.php/CRC_Storage_Comparisons Sometimes the system is more than just storage – features are important – but need to be the ones users use. Last Modified : 06/22/11 21 AFS – crc.nd.edu – nd.edu cell nd.edu is the campus legacy OpenAFS cell – started May 1990 – uses ND.EDU Kerberos realm – run by OIT staff Currently the default cell for most CRC logins and batch system – opteron, opterona, stats crc.nd.edu is the “new” cell run by CRC staff – Started October 2007 – uses CRC.ND.EDU The future cell for CRC logins and batch system – target for rollout June 2008 – hardware, and administrative differences. Kerb 4 EOL scheduled for 12 / 2008 for nd.edu cell Last Modified : 06/22/11 AFS – crc.nd.edu – nd.edu cell CRC Wiki Links http://crcmedia.hpcc.nd.edu/wiki/index.php/CRC_AFS_Cell Accessing multiple cells http://crcmedia.hpcc.nd.edu/wiki/index.php/Automatic_CRC/ND_AFS_cell_setup Recommendations on cells for primary access – interactive use Methods to migrating data – tar, up, cp, vos dump/restore, start fresh. Issues with interactive use – references to nd.edu that you don’t know about – e.g. mozilla, etc Last Modified : 06/22/11 CRC Storage Backup –B023 Malloy Hall • Software - Teradactyl Inc. – True Incremental Backup System – TiBS http://www.teradactyl.com • Available for backup of CRC and any research machines in colleges – On-site training June 16-20th, 2008. • Supported architectures include OpenAFS, Solaris, Linux, Windows, MacOSX. • Hardware - Backup server – Dell Power Edge 6950 server – utilizing 10 Gb ethernet & fiber channel interfaces. - Cache – 16 TB Infortrend Fibre Channel Array Last Modified : 06/22/11 24 Storage Backup Sony – Consolidated Storage Management System (CSM 200) Capacity of 604 tapes – 3 LTO4 drives with 1 TB tapes – 2 TB per tape with 2:1 compression- Library will hold > 1 PB without reloading tapes – expands to 2,988 tapes with 96 drives. Last Modified : 06/22/11 References • Wikipedia: FileSystems • Advanced File Systems Issues-Andy Wang FSU http://www.cs.fsu.edu/~awang/courses/cop5611_s2004/ • ND CRC wiki http://crc.nd.edu/wiki • OpenAFS User Guide http://www.openafs.org/doc/index.htm • OpenAFS Best Practices 2007 – Sudlow http://crc.nd.edu/facilities/documents/afsbpw2007.pdf Last Modified : 06/22/11 26 Questions ? • How can we improve this class? – Additional topics? – Cover one topic more thoroughly? – Remove topics? – Thanks for the feedback? Last Modified : 06/22/11 27.
Recommended publications
  • Elastic Storage for Linux on System Z IBM Research & Development Germany

    Elastic Storage for Linux on System Z IBM Research & Development Germany

    Peter Münch T/L Test and Development – Elastic Storage for Linux on System z IBM Research & Development Germany Elastic Storage for Linux on IBM System z © Copyright IBM Corporation 2014 9.0 Elastic Storage for Linux on System z Session objectives • This presentation introduces the Elastic Storage, based on General Parallel File System technology that will be available for Linux on IBM System z. Understand the concepts of Elastic Storage and which functions will be available for Linux on System z. Learn how you can integrate and benefit from the Elastic Storage in a Linux on System z environment. Finally, get your first impression in a live demo of Elastic Storage. 2 © Copyright IBM Corporation 2014 Elastic Storage for Linux on System z Trademarks The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. AIX* FlashSystem Storwize* Tivoli* DB2* IBM* System p* WebSphere* DS8000* IBM (logo)* System x* XIV* ECKD MQSeries* System z* z/VM* * Registered trademarks of IBM Corporation The following are trademarks or registered trademarks of other companies. Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. Cell Broadband Engine is a trademark of Sony Computer Entertainment, Inc. in the United States, other countries, or both and is used under license therefrom. Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.
  • Storage Administration Guide Storage Administration Guide SUSE Linux Enterprise Server 12 SP4

    Storage Administration Guide Storage Administration Guide SUSE Linux Enterprise Server 12 SP4

    SUSE Linux Enterprise Server 12 SP4 Storage Administration Guide Storage Administration Guide SUSE Linux Enterprise Server 12 SP4 Provides information about how to manage storage devices on a SUSE Linux Enterprise Server. Publication Date: September 24, 2021 SUSE LLC 1800 South Novell Place Provo, UT 84606 USA https://documentation.suse.com Copyright © 2006– 2021 SUSE LLC and contributors. All rights reserved. Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or (at your option) version 1.3; with the Invariant Section being this copyright notice and license. A copy of the license version 1.2 is included in the section entitled “GNU Free Documentation License”. For SUSE trademarks, see https://www.suse.com/company/legal/ . All other third-party trademarks are the property of their respective owners. Trademark symbols (®, ™ etc.) denote trademarks of SUSE and its aliates. Asterisks (*) denote third-party trademarks. All information found in this book has been compiled with utmost attention to detail. However, this does not guarantee complete accuracy. Neither SUSE LLC, its aliates, the authors nor the translators shall be held liable for possible errors or the consequences thereof. Contents About This Guide xii 1 Available Documentation xii 2 Giving Feedback xiv 3 Documentation Conventions xiv 4 Product Life Cycle and Support xvi Support Statement for SUSE Linux Enterprise Server xvii • Technology Previews xviii I FILE SYSTEMS AND MOUNTING 1 1 Overview
  • BESIII Physical Analysis on Hadoop Platform

    BESIII Physical Analysis on Hadoop Platform

    20th International Conference on Computing in High Energy and Nuclear Physics (CHEP2013) IOP Publishing Journal of Physics: Conference Series 513 (2014) 032044 doi:10.1088/1742-6596/513/3/032044 BESIII Physical Analysis on Hadoop Platform Jing HUO12, Dongsong ZANG12, Xiaofeng LEI12, Qiang LI12, Gongxing SUN1 1Institute of High Energy Physics, Beijing, China 2University of Chinese Academy of Sciences, Beijing, China E-mail: [email protected] Abstract. In the past 20 years, computing cluster has been widely used for High Energy Physics data processing. The jobs running on the traditional cluster with a Data-to-Computing structure, have to read large volumes of data via the network to the computing nodes for analysis, thereby making the I/O latency become a bottleneck of the whole system. The new distributed computing technology based on the MapReduce programming model has many advantages, such as high concurrency, high scalability and high fault tolerance, and it can benefit us in dealing with Big Data. This paper brings the idea of using MapReduce model to do BESIII physical analysis, and presents a new data analysis system structure based on Hadoop platform, which not only greatly improve the efficiency of data analysis, but also reduces the cost of system building. Moreover, this paper establishes an event pre-selection system based on the event level metadata(TAGs) database to optimize the data analyzing procedure. 1. Introduction 1.1 The current BESIII computing architecture High Energy Physics experiment is a typical data-intensive application. The BESIII computing system now consist of 3PB+ data, 6500 CPU cores, and it is estimated that there will be more than 10PB data produced in the future 5 years.
  • Globalfs: a Strongly Consistent Multi-Site File System

    Globalfs: a Strongly Consistent Multi-Site File System

    GlobalFS: A Strongly Consistent Multi-Site File System Leandro Pacheco Raluca Halalai Valerio Schiavoni University of Lugano University of Neuchatelˆ University of Neuchatelˆ Fernando Pedone Etienne Riviere` Pascal Felber University of Lugano University of Neuchatelˆ University of Neuchatelˆ Abstract consistency, availability, and tolerance to partitions. Our goal is to ensure strongly consistent file system operations This paper introduces GlobalFS, a POSIX-compliant despite node failures, at the price of possibly reduced geographically distributed file system. GlobalFS builds availability in the event of a network partition. Weak on two fundamental building blocks, an atomic multicast consistency is suitable for domain-specific applications group communication abstraction and multiple instances of where programmers can anticipate and provide resolution a single-site data store. We define four execution modes and methods for conflicts, or work with last-writer-wins show how all file system operations can be implemented resolution methods. Our rationale is that for general-purpose with these modes while ensuring strong consistency and services such as a file system, strong consistency is more tolerating failures. We describe the GlobalFS prototype in appropriate as it is both more intuitive for the users and detail and report on an extensive performance assessment. does not require human intervention in case of conflicts. We have deployed GlobalFS across all EC2 regions and Strong consistency requires ordering commands across show that the system scales geographically, providing replicas, which needs coordination among nodes at performance comparable to other state-of-the-art distributed geographically distributed sites (i.e., regions). Designing file systems for local commands and allowing for strongly strongly consistent distributed systems that provide good consistent operations over the whole system.
  • Z/OS Distributed File Service Zseries File System Implementation Z/OS V1R13

    Z/OS Distributed File Service Zseries File System Implementation Z/OS V1R13

    Front cover z/OS Distributed File Service zSeries File System Implementation z/OS V1R13 Defining and installing a zSeries file system Performing backup and recovery, sysplex sharing Migrating from HFS to zFS Paul Rogers Robert Hering ibm.com/redbooks International Technical Support Organization z/OS Distributed File Service zSeries File System Implementation z/OS V1R13 October 2012 SG24-6580-05 Note: Before using this information and the product it supports, read the information in “Notices” on page xiii. Sixth Edition (October 2012) This edition applies to version 1 release 13 modification 0 of IBM z/OS (product number 5694-A01) and to all subsequent releases and modifications until otherwise indicated in new editions. © Copyright International Business Machines Corporation 2010, 2012. All rights reserved. Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Notices . xiii Trademarks . xiv Preface . .xv The team who wrote this book . .xv Now you can become a published author, too! . xvi Comments welcome. xvi Stay connected to IBM Redbooks . xvi Chapter 1. zFS file systems . 1 1.1 zSeries File System introduction. 2 1.2 Application programming interfaces . 2 1.3 zFS physical file system . 3 1.4 zFS colony address space . 4 1.5 zFS supports z/OS UNIX ACLs. 4 1.6 zFS file system aggregates. 5 1.6.1 Compatibility mode aggregates. 5 1.6.2 Multifile system aggregates. 6 1.7 Metadata cache. 7 1.8 zFS file system clones . 7 1.8.1 Backup file system . 8 1.9 zFS log files.
  • Configuring and Managing the Network File System

    Configuring and Managing the Network File System

    ch33.fm Page 639 Saturday, April 29, 2000 1:41 AM THIRTY-THREE 33 Configuring and Managing the Network File System LChapter Syllabus The Network File System (NFS) is used to share files and directories over a network. It is an RPC- 33.1 NFS Concepts based system and works in the client-server model. The NFS server makes some of its file sys- 33.2 Configuring an NFS Server tems shareable with other systems on a network. The NFS client mounts shared file systems from 33.3 Configuring an NFS Client the server to its local mount points. Users of the client machines can use NFS-mounted file systems 33.4 NFS Daemons just like the local file systems. NFS support is available on many platforms including personal 33.5 Troubleshooting computers. On UNIX systems, this has become a de facto standard and provides a transparent way for sharing files. When used in combination with NIS, it facilitates users’ logging in from any machine on the network and finding the same environment, home directories, and mailboxes. NFS is used to save disk space in a number of ways. All files that are static in nature can be shared on a network. These files include applica- tion directories under /opt and operating system files under the /usr directory. The files that are 639 ch33.fm Page 640 Saturday, April 29, 2000 1:41 AM 640 Chapter 33 • Configuring and Managing the Network File System shared by individual users are users’ home directories and mail- boxes. The directories that contain configuration files or log files are usually not shared.
  • GPFS Policy Engine

    GPFS Policy Engine

    General Parallel File System Version 4 Release 1 Administration and Programming Reference SA23-1452-00 General Parallel File System Version 4 Release 1 Administration and Programming Reference SA23-1452-00 Note Before using this information and the product it supports, read the information in “Notices” on page 655. This edition applies to version 4 release 1 modification 0 of the following products, and to all subsequent releases and modifications until otherwise indicated in new editions: v IBM General Parallel File System ordered through Passport Advantage® (product number 5725-Q01) v IBM General Parallel File System ordered through AAS/eConfig (product number 5641-GPF) v IBM General Parallel File System ordered through HVEC/Xcel (product number 5641-GP6, 5641-GP7, or 5641-GP8) Significant changes or additions to the text and illustrations are indicated by a vertical line (|) to the left of the change. IBM welcomes your comments; see the topic “How to send your comments” on page xii. When you send information to IBM, you grant IBM a nonexclusive right to use or distribute the information in any way it believes appropriate without incurring any obligation to you. © Copyright IBM Corporation 2014. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp. Contents Tables ...............vii Backing up a file system using the mmbackup command ..............34 About this information ........ix Backing up a file system using the GPFS policy engine ...............40 Prerequisite and related information ......x Backing up file system configuration information 40 Conventions used in this information ......xi Using APIs to develop backup applications.
  • Managing Network File Systems in Oracle® Solaris 11.4

    Managing Network File Systems in Oracle® Solaris 11.4

    Managing Network File Systems in ® Oracle Solaris 11.4 Part No: E61004 August 2021 Managing Network File Systems in Oracle Solaris 11.4 Part No: E61004 Copyright © 2002, 2021, Oracle and/or its affiliates. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract.
  • Considerations When Choosing a Backup System for AFS

    Considerations When Choosing a Backup System for AFS

    Considerations when Choosing a Backup System for AFS By Kristen J. Webb President and CTO Teradactyl LLC. June 18, 2005 The Andrew File System® has a proven track record as a scalable and secure network file system. Organizations everywhere depend on AFS for the sharing of data both internally and externally. The most telling sign of the longevity of AFS is the adoption of the open source version of the software (OpenAFS). One of the important challenges facing AFS system administrators is the need to provide efficient, cost-effective backup strategies that can scale with ever increasing cell sizes. This paper provides an overview of the issues to consider when putting together a long term backup and recovery strategy. The topics include: Methods available for data backup and recovery; Impact of the backup function on cell performance and scale; Volume management; Centralized vs. Distributed backups; Disk-to-Tape, Disk-to-Disk and Disk-to-Disk-to-Tape comparisons; Disaster Recovery requirements; Advanced Solutions. The reader should have a basic understanding of backup and recovery concepts and familiarity with AFS architecture and cell administration. Introduction The purpose of this paper it to examine the general behavior of backup systems in relation to working with the Andrew File System (AFS). It is written as a guide for system administers to help make them aware of all of the considerations for backup, restore, and disaster recovery that they need to take into account when putting together a backup and recovery strategy for AFS. The paper includes discussions on the features and limitations of the native backup and recovery programs provided with AFS.
  • File Systems

    File Systems

    ;login JUNE 2013 VOL. 38, NO. 3 : File Systems & Linux File System Evolution via Patch Analysis Lanyue Lu, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, and Shan Lu &Interview with Ted Ts’o Rik Farrow & Ganeti: Cluster Virtualization Manager Guido Trotter and Tom Limoncelli & Future Interfaces for Non- Volatile Memory Andy Rudoff Columns Practical Perl Tools: Constants David Blank-Edelman IPython and Notebook David Beazley Sampling and Monitoring Dave Josephsen For Good Measure: Calculating Risk Dan Geer and Dan Conway /dev/random: Clusters and Understanding *nix File Permissions Robert Ferrell Book Reviews Mark Lamourine and Trey Darley Conference Reports FAST ’13: 11th USENIX Conference on File and Storage Technologies UPCOMING EVENTS FOR A COMPLETE LIST OF USENIX AND USENIX CO-SPONSORED EVENTS, SEE WWW.USENIX.ORG/CONFERENCES 2013 USENIX Federated Conferences Week Workshops Co-located with USENIX Security ’13 June 24–28, 2013, San Jose, CA, USA www.usenix.org/conference/fcw13 EVT/WOTE ’13: 2013 Electronic Voting Technology Workshop/Workshop on Trustworthy Elections USENIX ATC ’13: 2013 USENIX Annual Technical August 12–13, 2013 Conference www.usenix.org/conference/evtwote13 June 26–28, 2013 USENIX Journal of Election Technology www.usenix.org/conference/atc13 and Systems (JETS) Published in conjunction with EVT/WOTE ICAC ’13: 10th International Conference on www.usenix.org/jets Autonomic Computing Submissions for Volume 1, Issue 2, due: August 6, 2013 June 26–28, 2013 www.usenix.org/conference/icac13 CSET ’13: 6th Workshop on Cyber
  • Distributed Systems 14

    Distributed Systems 14

    Distributed Systems 14. Network File Systems (Network Attached Storage) Paul Krzyzanowski Rutgers University Fall 2018 October 22, 2018 © 2014-2018 Paul Krzyzanowski 1 Accessing files File sharing with socket-based programs HTTP, FTP, telnet: – Explicit access – User-directed connection to access remote resources We want more transparency – Allow user to access remote resources just as local ones NAS: Network Attached Storage October 22, 2018 © 2014-2018 Paul Krzyzanowski 2 File service models Upload/Download model Remote access model – Read file: copy file from server to client File service provides functional interface: – Write file: copy file from client to server – create, delete, read bytes, write bytes, etc… Advantage: Advantages: – Simple – Client gets only what’s needed – Server can manage coherent view of file system Problems: – Wasteful: what if client needs small Problem: piece? – Possible server and network congestion – Problematic: what if client doesn’t have enough space? • Servers are accessed for duration of file access – Consistency: what if others need to modify the same file? • Same data may be requested repeatedly October 22, 2018 © 2014-2018 Paul Krzyzanowski 3 Semantics of file sharing Sequential Semantics Session Semantics Read returns result of last write Relax the rules Easily achieved if • Changes to an open file are – Only one server initially visible only to the process – Clients do not cache data (or machine) that modified it. BUT • Need to hide or lock file under – Performance problems if no cache modification
  • CIFS SMB2 SMB3 Meets Linux a Year in Review

    CIFS SMB2 SMB3 Meets Linux a Year in Review

    The Future of File Protocols: CIFS SMB2 SMB3 Meets Linux A Year in Review Steve French Senior Engineer – File System Architect IBM Linux Technology Center 1 IBM, Linux, and Building a Smarter Planet © 2012 IBM Corporation Legal Statement – This work represents the views of the author(s) and does not necessarily reflect the views of IBM Corporation – A full list of U.S. trademarks owned by IBM may be found at http://www.ibm.com/legal/copytrade.shtml. – Linux is a registered trademark of Linus Torvalds. – Other company, product, and service names may be trademarks or service marks of others. 2 © 2012 IBM Corporation Who am I? – Steve French ([email protected] or [email protected]) – Author and maintainer of Linux cifs vfs (for accessing Samba, Windows and various SMB/CIFS based NAS appliances) – Wrote initial SMB2 kernel client prototype – Member of the Samba team, coauthor of SNIA CIFS Technical Reference and former SNIA CIFS Working Group chair – Architect: File Systems/NFS/Samba for IBM Linux Technology Center © 2012 IBM Corporation SMB3: Great Feature Set, Broad Deployment, Amazing Performance ● Introduction of new storage features in Windows 8 causing one of most dramatic shifts in storage industry as companies rapidly move to support “SMB3” (aka SMB2.2) ● “SMB2.2 (CIFS) screams over InfiniBand” (Storage CH Blog) • Is (traditional) SAN use going to die? – “Market trends show virtualization workloads moving to NAS” (Dennis Chapman, Technical Director NetApp, SNIA SDC 2011) – “Unstructured data (file-based) storage is growing faster than structured data” (Thomas Pfenning, Microsoft GM, SNIA SDC 2011) – Customers prefer “file” • SMB2/CIFS market share is MUCH larger than NFS.