DOCSLIB.ORG

High-Performance and Highly Reliable File System for the K Computer

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
High-Performance and Highly Reliable File System for the K Computer High-Performance and Highly Reliable File System for the K computer Kenichiro Sakai Shinji Sumimoto Motoyoshi Kurokawa RIKEN and Fujitsu have been developing the world’s fastest supercomputer, the K computer. In addition to over 80 000 compute nodes, the K computer has several tens of petabytes storage capacity and over one terabyte per second of I/O bandwidth. It is the largest and fastest storage system in the world. In order to take advantage of this huge storage system and achieve high scalability and high stability, we developed the Fujitsu Exabyte File System (FEFS), a clustered distributed file system. This paper outlines the K computer’s file system and introduces measures taken in FEFS to address key issues in a large-scale system. 1. Introduction K computer,note)i which is currently the fastest The dramatic jumps in supercomputer supercomputer in the world. performance in recent years led to the This paper outlines the K computer’s file achievement of 10 peta floating point operations system and introduces measures taken in FEFS per second (PFLOPS) in 2011. The increasing to address key issues in a large-scale system. number of compute nodes and cores and the growing amount of on-board memory mean 2. File system for the K that file systems must have greater capacity computer and total throughput. File-system capacity To achieve a level of performance and stable and performance are expected to reach the job execution appropriate for the world’s fastest 100-petabyte (PB) class and 1-terabyte/s (TB/s) supercomputer, We adopted a two-layer model class, respectively, as both capacity and for the file system of the K computer. This model performance are increasing at a nearly tenfold consists of a local file system used as a high- rate every year. For this reason, mainstream file speed, temporary-storage area for jobs only and a systems have been changing from a single-server global file system used as a large-capacity shared- type to a clustered type. storage area for storing user files Figure( 1). We have developed a clustered distributed In this two-tiered model, jobs are executed file system called the Fujitsu Exabyte File through data transfers between these two file System (FEFS) with the aim of achieving a file systems using a file-staging control function. system having a capacity and performance that The following describes the respective roles of can handle the computing performance of the note)i “K computer” is the English name that RIKEN has been using for the supercomputer of this project since July 2010. “K” comes from the Japanese word “Kei,” which means ten peta or 10 to the 16th power. 302 FUJITSU Sci. Tech. J., Vol. 48, No. 3, pp. 302–309 (July 2012) K. Sakai et al.: High-Performance and Highly Reliable File System for the K computer Compute node group Compute Compute Compute Compute Compute Compute Local file system node node node node node node Tofu interconnect I/O node I/O node I/O node I/O node I/O node I/O node (OSS) (OSS) (OSS) jobs Batch (OSS) (OSS) (OSS) MDS MDS (active) (standby) Interactive jobs Data transfer network and management network Staging function Job Login node management node MDS MDS OSS OSS (active) (standby) (active) (active) Reference ReferencePeripheral devices Global file system MDS: Metadata server OSS: Object storage server Figure 1 FigureTwo-layer 1 file system model. Two-layer file system model. the local file system, global file system, and file- user files consisting of job input/output data staging process. and other content. In addition to providing file 1) Local file system access through the login node, the global file This is a high-speed, temporary storage system enables direct file access from a compute area dedicated to job execution for extracting node when executing an interactive job so that a maximum file-I/O performance for applications user can perform program debugging and tuning executed as batch jobs. The local file system while checking the job output in real time. transfers input/output files from and to the global A compute node connects to a group of file file system through the file-staging function and servers in the global file system through an temporarily holds job files that are either under InfiniBand link mounted on the I/O-dedicated or waiting for execution. node. In short, an I/O node relays data between The file server used to access data blocks the Tofu interconnect and InfiniBand while a connects to a compute node through the Tofu compute node accesses file servers via that I/O interconnect using an I/O-dedicated node (I/O node. node) within the compute rack through remote 3) File staging direct memory access (RDMA) communications. File transfer between the local and global As a result, low-latency, high-throughput file file systems is performed automatically by the data transfer is achieved. system through the file-staging function, which 2) Global file system operates in conjunction with job operation This is a large-capacity, shared storage software. Before a job begins, this function area external to the compute racks for storing transfers the input files for that job from the FUJITSU Sci. Tech. J., Vol. 48, No. 3 (July 2012) 303 K. Sakai et al.: High-Performance and Highly Reliable File System for the K computer global file system to the local file system (stage- file system size and maximum file size to the in), and after the job completes, it transfers the 8-exabyte (EB) class relative to existing Lustre output files from the local file system to the specifications. File system size can be extended global file system (stage-out). The user specifies even during system operation by adding more the stage-in and stage-out files in a job script. servers and storage devices. The following sections describe the FEFS 3. FEFS: Ultra-large-scale cluster measures listed below for dealing with an ultra- file system large-scale supercomputer system. We developed FEFS to provide a local • Elimination of I/O contention by using file system and global file system suitable for communication-bus and disk splitting the top-class performance of the K computer. • Continuous availability by using hardware The following objectives were established in duplication and failover developing FEFS: • Hierarchical node monitoring and automatic • World’s fastest I/O performance and high- switching performance Message Passing Interface • QoS-policy selection by type of operation (MPI)-IO • Stable job run times achieved by eliminating 4. Elimination of I/O contention interruptions by using communication-bus • World’s largest file-system capacity and disk splitting • Greater scalability in performance and As a clustered file system, FEFS bundles capacity by adding hardware together a large number of file servers and • High reliability (service continuity and data improves parallel throughput performance preservation) in proportion to the number of file servers. • Ease of use (sharing by many users) However, if access should become concentrated • Fair share (fair use by many users) on specific file servers, communications can To achieve these objectives, We developed become congested and disk-access contention can FEFS on the basis of the Lustre open-source file arise, resulting in a drop in I/O performance and system, which has become a worldwide industry a variation in job run times between compute standard, while extending certain specifications nodes. It is therefore essential that file I/O and functions as needed. As shown in Table 1, contention be completely eliminated to ensure FEFS extends specifications like maximum stable job execution. Table 1 Lustre and FEFS comparison. Function Lustre FEFS File system size 64 PB 8 EB File size 320 TB 8 EB System specifications No. of files 4 G 8 E (max. values) OST volume size 16 TB 1 PB No. of stripes 160 20 000 No. of ACL entries 32 8191 No. of OSSs 1020 20 000 Scalability No. of OSTs 8150 20 000 (max. values) No. of clients 128 units 1 000 000 units Block size (Lustre disk file system) 4 KB up to 512 KB (backend file system) 304 FUJITSU Sci. Tech. J., Vol. 48, No. 3 (July 2012) K. Sakai et al.: High-Performance and Highly Reliable File System for the K computer In FEFS, file I/O contention is eliminated service and system operations from continuing. by separating file I/O access into two stages, However, when configuring FEFS with more namely, in units of jobs and units of compute than several hundred file servers and storage nodes (Figure 2). devices, there are bound to be times when At the job level, each job is assigned a hardware such as network adaptors and servers separate range of I/O nodes corresponding to are in a fault or maintenance state. To ensure file-storage destinations to eliminate file I/O that operations across the entire system continue contention on servers, on the network, and on even under such conditions, it is essential that disks. And on the compute-node level within a faults be automatically detected and detours job, the transmitting and receiving of file data is taken around faulty locations so that file-system performed via I/O nodes constituting a minimum service can continue unimpeded. number of Tofu-communication hops, thereby For this reason, FEFS duplicates hardware minimizing file I/O contention among compute and uses a software-based control process nodes. to initiate failover processing at the time of a hardware fault. This processing switches 5. Continuous availability by the server and I/O communication paths and using hardware duplication restores the service in progress to its normal and failover state.
Load more

Recommended publications
  • STATEMENT of WORK: SGS File System
    ATTACHMENT A STATEMENT OF WORK: SGS File System DOE National Nuclear Security Administration & the DOD Maryland Office April 25, 2001 File Systems SOW April 25, 2001 Table of Contents TABLE OF CONTENTS ..................................................................................................................................2 1.0 OVERVIEW.................................................................................................................................................4 1.1 INTRODUCTION ..........................................................................................................................................4 2.0 MOTIVATION ............................................................................................................................................5 2.1 THE NEED FOR IMPROVED FILE SYSTEMS .................................................................................................5 2.2 I/O CHARACTERIZATION OF IMPORTANT APPLICATIONS...........................................................................6 2.3 CURRENT AND PROJECTED ENVIRONMENTS AT LLNL, LANL, SANDIA, AND THE DOD .........................6 2.4 SUMMARY OF FIVE TECHNOLOGY CATEGORIES ........................................................................................9 3.0 MINIMUM REQUIREMENTS (GO/NO-GO CRITERIA)..................................................................12 3.1 POSIX-LIKE INTERFACE [MANDATORY]........................................................................................12 3.2 INTEGRATION
    [Show full text]
  • Introduction to Computer Concepts
    MANAGING PUBLIC SECTOR RECORDS A Training Programme Understanding Computers: An Overview for Records and Archives Staff INTERNATIONAL INTERNATIONAL RECORDS COUNCIL ON ARCHIVES MANAGEMENT TRUST MANAGING PUBLIC SECTOR RECORDS: A STUDY PROGRAMME UNDERSTANDING COMPUTERS: AN OVERVIEW FOR RECORDS AND ARCHIVES STAFF MANAGING PUBLIC SECTOR RECORDS A STUDY PROGRAMME General Editor, Michael Roper; Managing Editor, Laura Millar UNDERSTANDING COMPUTERS: AN OVERVIEW FOR RECORDS AND ARCHIVES STAFF INTERNATIONAL RECORDS INTERNATIONAL MANAGEMENT TRUST COUNCIL ON ARCHIVES MANAGING PUBLIC SECTOR RECORDS: A STUDY PROGRAMME Understanding Computers: An Overview for Records and Archives Staff © International Records Management Trust, 1999. Reproduction in whole or in part, without the express written permission of the International Records Management Trust, is strictly prohibited. Produced by the International Records Management Trust 12 John Street London WC1N 2EB UK Printed in the United Kingdom. Inquiries concerning reproduction or rights and requests for additional training materials should be addressed to International Records Management Trust 12 John Street London WC1N 2EB UK Tel: +44 (0) 20 7831 4101 Fax: +44 (0) 20 7831 7404 E-mail: [email protected] Website: http://www.irmt.org Version 1/1999 MPSR Project Personnel Project Director Anne Thurston has been working to define international solutions for the management of public sector records for nearly three decades. Between 1970 and 1980 she lived in Kenya, initially conducting research and then as an employee of the Kenya National Archives. She joined the staff of the School of Library, Archive and Information Studies at University College London in 1980, where she developed the MA course in Records and Archives Management (International) and a post-graduate research programme.
    [Show full text]
  • Winreporter Documentation
    WinReporter documentation Table Of Contents 1.1 WinReporter overview ............................................................................................... 1 1.1.1.................................................................................................................................... 1 1.1.2 Web site support................................................................................................. 1 1.2 Requirements.............................................................................................................. 1 1.3 License ....................................................................................................................... 1 2.1 Welcome to WinReporter........................................................................................... 2 2.2 Scan requirements ...................................................................................................... 2 2.3 Simplified Wizard ...................................................................................................... 3 2.3.1 Simplified Wizard .............................................................................................. 3 2.3.2 Computer selection............................................................................................. 3 2.3.3 Validate .............................................................................................................. 4 2.4 Advanced Wizard....................................................................................................... 5 2.4.1
    [Show full text]
  • Chapter 9– Computer File Formats
    PERA Employer Manual Chapter 9 - Computer File Formats IN THIS CHAPTER: • Testing Process Chapter 9– • Contribution (SDR) File Computer File Formats Layout » Contribution File Over 800 employers use PERA’s online Employer Reporting and Header Information System (ERIS) to send computer files to PERA for pro- » Plan Summary Record cessing. » Detail Transactions » Reporting an PERA has the following computer file formats. Adjustment • Contribution Edits 1. The Contribution file (also called the SDR file) provides the pay • Demographic File Layout period salary and contribution data for electronic submission of • Exclusion Report File the Salary Deduction Report (SDR). The file includes three types » Excel Format of records: the header, plan summary, and detail member trans- » Text File Format actions. As a general rule, the Contribution data file must be a fixed length text file. Employers that have more than 50 active members and do not have technical support to convert their pay- roll data into the text format may contact PERA staff at 651-296- 3636 or 1-888-892-7372 option #2 to discuss submitting their data via Excel. Last chapter 9 revision: 2. The Demographic Data Record format has a single record July 9, 2018 type that used to enroll new employees into a PERA plan or to update a member’s retirement account status as employment changes, such as leaves of absence or terminations occur, or to report member’s personal data changes, such as changes in name or address. 3. The Exclusion Report file is used to comply with the annual legal requirement of providing information about all employees – including elected officials – who worked during the reporting 9-1 PERA Employer Manual Chapter 9 - Computer File Formats year and were not members of a PERA Defined Benefit or Defined Contribution Plan or another Minnesota public retirement system.
    [Show full text]
  • File Management – Saving Your Documents
    File Management – Saving Your Documents You will save everything to your H: drive. This is your personal drive that is out on the server and only you have access to it. It is not accessible by others. Your H: drive is accessible from any CCPS computer or from home via Remote Access. Accessing Your H: Drive To access your H: drive via the desktop: 1. Double-click the “This PC” icon. 2. Under Network Locations, double-click the drive that begins with your username and ends with the letter “H”. To access your H: drive via File Explorer: 1. Click the yellow folder at the bottom left corner of the screen. 2. Under Network Locations, double-click the drive that begins with your username and ends with the letter “H”. Accessing Shared Drives If shared drives have been added to your profile, they will be located under Network Locations. If you need access to a departmental or school shared drive, please submit a work order so it can be added to your profile or email [email protected]. Creating Shortcuts on the Desktop Do not save documents/items directly to the desktop. If you would like to place documents on the desktop for quick- access, save your master documents to your H: drive and place a shortcut to the document on the desktop. To create a shortcut on the desktop to a program or file do the following: 1. Right-click an open area on the desktop, Select New, and then click Shortcut. 2. Click Browse. 3. Locate the program or file to which you want to create a shortcut, click the program or file, click Open, and then click Next.
    [Show full text]
  • Extending File Systems Using Stackable Templates
    THE ADVANCED COMPUTING SYSTEMS ASSOCIATION The following paper was originally published in the Proceedings of the USENIX Annual Technical Conference Monterey, California, USA, June 6-11, 1999 Extending File Systems Using Stackable Templates _ Erez Zadok, Ion Badulescu, and Alex Shender Columbia University © 1999 by The USENIX Association All Rights Reserved Rights to individual papers remain with the author or the author's employer. Permission is granted for noncommercial reproduction of the work for educational or research purposes. This copyright notice must be included in the reproduced paper. USENIX acknowledges all trademarks herein. For more information about the USENIX Association: Phone: 1 510 528 8649 FAX: 1 510 548 5738 Email: [email protected] WWW: http://www.usenix.org Extending File Systems Using Stackable Templates Erez Zadok, Ion Badulescu, and Alex Shender Computer Science Department, Columbia University ezk,ion,alex @cs.columbia.edu { } Abstract Kernel-resident native file systems are those that inter- Extending file system functionality is not a new idea, act directly with lower level media such as disks[9] and but a desirable one nonetheless[6, 14, 18]. In the several networks[11, 16]. Writing such file systems is difficult years since stackable file systems were first proposed, only because it requires deep understanding of specific operat- a handful are in use[12, 19]. Impediments to writing new ing system internals, especially the interaction with device file systems include the complexity of operating systems, drivers and the virtual memory system. Once such a file the difficulty of writing kernel-based code, the lack of a system is written, porting it to another operating system true stackable vnode interface[14], and the challenges of is just as difficult as the initial implementation, because porting one file system to another operating system.
    [Show full text]
  • Hexadecimal Numbers
    AP Computer Science Principles How Computers Work: Computer Architecture and Data Storage What You Will Learn . How to disassemble and rebuild a computer . How to choose the parts to build a new computer . Logic Gates . Data storage: . Binary . Hexadecimal . ASCII . Data compression . Computer Forensics . Limits of computing 2 2 Introduction to Computer Hardware 33 The Parts of a Computer 4 Image: By Gustavb. Source Wikimedia. License: CC BY-SA 3.0 4 Main parts of a computer . Case . Power supply . Motherboard . CPU . RAM . Adapter Cards . Internal and External Drives . Internal and External Cables 5 55 The Motherboard . The motherboard provides the electrical connections between the computer components. Comes in different sizes called “form factors.” . All the other components of the computer connect to the motherboard directly or through a cable. Image: By skeeze. Source Pixabay. License: CC BY-SA 3.0 6 66 A Case Needs to Name size (mm) be a Form BTX 325 × 266 ATX 305 × 244 Factor as Larger EATX 305 × 330 or Larger than (Extended) LPX 330 × 229 the microBTX 264 × 267 Motherboard. microATX 244 × 244 Mini-ITX 170 × 170 7 7 Image: By Magnus Manske. Source Wikimedia. License: CC BY-7SA 3.0 The Power Supply . Power Supplies come in different form factors and should match the form factor of the case. Power supplies are rated in watts and should have a rating larger than the total required watts of all the computer components. 8 8 Image: By Danrock Source: Wikimedia License: CC BY-SA 3.0 8 The Power Supply Connectors . The Power Supply will have lots of extra connectors so you can add additional components to your computer at a later time.
    [Show full text]
  • Network File Sharing Protocols
    Network File Sharing Protocols Lucius quests her hotchpotch ethnologically, she contradistinguish it self-righteously. Dialogic Millicent catch prayingly. Sheridan usually foils stintedly or pirouette anaerobiotically when trilobed Saxon disposes genotypically and homeward. It is recommended that this type be accepted by all FTP implementations. So how do you protect yourself from these malware infected movies? Which protocol is even an option for you is based on your use scenario. Learn about the latest issues in cybersecurity and how they affect you. As more clients access the file, share important files digitally and work as a team even sitting miles apart. Friendly name of the printer object. SMB is built in to every version of Windows. As you can see NFS offers a better performance and is unbeatable if the files are medium sized or small. Processes utilizing the network that do not normally have network communication or have never been seen before are suspicious. You are correct, thank you. It syncs files between devices on a local network or between remote devices over the internet. Sets the maximum number of channels for all subsequent sessions. Session control packets Establishes and discontinues a connection to shared server resources. Please fill in all required fields before continuing. File names in CIFS are encoded using unicode characters. The configured natural language of the printer. After entering its resources or network protocols that protocol over the server supports file services while originally connected you must be set. Download a free fully functional evaluation of JSCAPE MFT Server. The File Transfer Protocol follows the specifications of the Telnet protocol for all communications over the control connection.
    [Show full text]
  • EDRM Glossary Version 1.002, April 22, 2016
    EDRM Glossary http://www.edrm.net/resources/glossaries/glossary Version 1.002, April 22, 2016 © 2016 EDRM LLC EDRM Glossary http://www.edrm.net/resources/glossaries/glossary The EDRM Glossary is EDRM's most comprehensive listing of electronic discovery terms. It includes terms from the specialized glossaries listed below as well as terms not in those glossaries. The terms are listed in alphabetical order with definitions and attributions where available. EDRM Collection Standards Glossary The EDRM Collection Standards Glossary is a glossary of terms defined as part of the EDRM Collection Standards. EDRM Metrics Glossary The EDRM Metrics Glossary contains definitions for terms used in connection with the updated EDRM Metrics Model published in June 2013. EDRM Search Glossary The EDRM Search Glossary is a list of terms related to searching ESI. EDRM Search Guide Glossary The EDRM Search Guide Glossary is part of the EDRM Search Guide. The EDRM Search Guide focuses on the search, retrieval and production of ESI within the larger e- discovery process described in the EDRM Model. IGRM Glossary The IGRM Glossary consists of commonly used Information Governance terms. The Grossman-Cormack Glossary of Technology-Assisted Review Developed by Maura Grossman of Wachtell, Lipton, Rosen & Katz and Gordon Cormack of the University of Waterloo, the Grossman-Cormack Glossary of Technology-Assisted Review contains definitions for terms used in connect with the discovery processes referred to by various terms including Computer Assisted Review, Technology Assisted Review, and Predictive Coding. If you would like to submit a new term with definition or a new definition for an existing term, please go to our Submit a Definition page, http://www.edrm.net/23482, and complete and submit the form.
    [Show full text]
  • Python Style Guide
    CS 102: Python Style Guide This documents provides some standards that will help ensure you are writing good code in this course. Good coding standards allow students to clearly demonstrate their plan for a program, and they help avoid confusion and errors. Each coding assignment in this course will be assessed partly on the items below. Heading Comment at Top Each Python file you submit must have a comment at the top that includes your name, the date of the assignment, and the assignment number. Here is an example: # Kowshik Bhowmik # Assignment 1: Style Guide # 08/01/2020 Descriptive Names Variables, parameters, and functions must have descriptive names. While naming all your variables after your favorite fast food menu items or your favorite Pokemon is fun, it won’t make for an understandable program. A good variable or parameter name describes what the variable stores and its role in the program. A good function name describes what the function does. Variable names tend to be nouns while function names are often verbs. Variable and function names will use the python standard. Names will be lower case with multiple words separated by an underscore. Variables that will hold values that are not meant to change are named in all upper case (this has no effect per say, but it is a stylistic convention). For example: total = 10 sales_tax = 20 search() calculate_amount() CONSTATNT = 10 MY_CONSTANT = 20 Descriptive Comments Every function you write should have a comment before it briefly describing what the function does, what the parameters are (if any), what side effects it has (if any), and what it outputs (if anything).
    [Show full text]
  • Windows 10 Fundamentals Module 2-2017-04-01
    Windows 10 Fundamentals Student Guide Module 2 Windows 10 Module Two: File System Introduction In Module Two we will introduce the file system; the way Windows 10 organizes saved files. We will learn how applications create files and name them. We explore how the computer file system is organized and how to view the file locations using the File Explorer. We will create file folders that will store our files and we will use the file system in several student exercises. Topics Permanent and removable data storage devices. View the permanent and removable drives in the computer. Review files names and file types. Hierarchies and their relation to the file system structure. Introduction of the Windows File Explorer File system navigation. Creation of a file folder. Exercises Exercise 2A – Display the Computer Drives Exercise 2B: Navigate through the File System Exercise 2C: Create a folder using the File Explorer Objectives At the end of this module you will be able to: Know where data is stored in a Computer. Find the hard disk properties and where to start a disk file cleanup. Understand how the file system operates in a computer. Understand how to open the File Explorer. Use the File Explorer to move through the file system. Create a new folder. Version: 4/17/17 Page 2.1 Peoples Resource Center Windows 10 Fundamentals Student Guide Module 2 Windows 10 1. Where are files stored? Data files in a Computer are created by applications and stored in one of several data storage devices inside or connected to the computer.
    [Show full text]
  • My Book Desktop Storage User Manual
    My Book™ Desktop Storage User Manual Accessing Online Support Visit our product support website at http://support.wdc.com and choose from these topics: ▪ Downloads—Download drivers, software, and updates for your WD product. ▪ Registration—Register your WD product to get the latest updates and special offers at http://register.wdc.com ▪ Warranty & RMA Services—Get warranty, product replacement (RMA), RMA status, and data recovery information. ▪ Knowledge Base—Search by keyword, phrase, or answer ID. ▪ Installation—Get online installation help for your WD product or software. ▪ WD Community—Share your thoughts and connect with other WD users at http://community.wdc.com Table of Contents _________ Accessing Online Support.................................................................................ii _________ 1 About Your WD Drive.................................................................................... 1 Features.............................................................................................................................1 Kit Contents......................................................................................................................2 Optional Accessories.......................................................................................................2 Operating System Compatibility....................................................................................2 Disk Drive Format............................................................................................................
    [Show full text]