Best Practice Guide - Parallel I/O Sandra Mendez, LRZ, Germany Sebastian Lührs, FZJ, Germany Dominic Sloan-Murphy (Editor), EPCC, United Kingdom Andrew Turner (Editor), EPCC, United Kingdom Volker Weinberg (Editor), LRZ, Germany Version 2.0 by 07-02-2019 1 Best Practice Guide - Parallel I/O Table of Contents 1. Introduction .............................................................................................................................. 4 1.1. About this Document ...................................................................................................... 4 1.2. Guide Structure .............................................................................................................. 4 2. High Performance I/O Systems .................................................................................................... 5 2.1. Introduction ................................................................................................................... 5 2.2. I/O Strategies in Parallel Applications ................................................................................ 5 2.2.1. Serial I/O ............................................................................................................ 6 2.2.2. Parallel I/O ......................................................................................................... 6 2.3. The I/O Software Stack ................................................................................................... 8 2.3.1. Low-level: POSIX ................................................................................................ 8 2.3.2. Middle-level: MPI-IO ............................................................................................ 8 2.3.3. High level I/O software ......................................................................................... 9 2.4. General Pointers for Efficient I/O ...................................................................................... 9 3. Parallel File Systems ................................................................................................................ 10 3.1. Introduction .................................................................................................................. 10 3.2. Lustre ......................................................................................................................... 10 3.2.1. Lustre File Layout (Striping) ................................................................................ 11 3.2.2. Choosing a stripe count ....................................................................................... 11 3.2.3. Choosing a stripe_size ......................................................................................... 12 3.3. IBM Spectrum Scale (formerly GPFS) .............................................................................. 12 3.3.1. Block size ......................................................................................................... 13 3.4. BeeGFS (formerly FhGFS) ............................................................................................. 13 3.5. Object Storage .............................................................................................................. 14 4. MPI-IO .................................................................................................................................. 15 4.1. Introduction .................................................................................................................. 15 4.1.1. MPI-IO data access operations .............................................................................. 16 4.1.2. General Hints ..................................................................................................... 16 4.2. Manipulating Files in MPI .............................................................................................. 16 4.2.1. Opening a File ................................................................................................... 17 4.3. File View .................................................................................................................... 17 4.4. ROMIO optimisation ..................................................................................................... 18 4.5. Setting Hints for MPI-IO ................................................................................................ 19 4.6. MPI-IO General Considerations ....................................................................................... 19 5. File Per Process ...................................................................................................................... 21 5.1. Introduction .................................................................................................................. 21 5.2. File Per Process vs. Shared File ....................................................................................... 21 5.3. Optimising File Per Process ............................................................................................ 22 5.4. Sample Performance Data ............................................................................................... 22 6. High-Level I/O Libraries ........................................................................................................... 24 6.1. Introduction .................................................................................................................. 24 6.2. NetCDF ....................................................................................................................... 24 6.2.1. Architecture of NetCDF APIs and Libraries ............................................................. 24 6.2.2. Parallel I/O with netCDF ..................................................................................... 25 6.3. HDF5 .......................................................................................................................... 26 6.3.1. HDF5 Design ..................................................................................................... 26 6.3.2. HDF5 File Organization and Data Model ................................................................ 27 6.3.3. Selecting a Portion of a Dataspace ......................................................................... 27 6.3.4. Chunking in HDF5 ............................................................................................. 28 6.3.5. Parallel HDF5 .................................................................................................... 29 6.3.6. Parallel HDF5 Example ....................................................................................... 30 6.4. pNetCDF ..................................................................................................................... 31 6.5. SIONLib ...................................................................................................................... 32 6.5.1. SIONLib file format ............................................................................................ 32 6.5.2. SIONLib API and utilities .................................................................................... 32 6.6. Touching on Low-level I/O: POSIX, C and Fortran File Manipulation ..................................... 33 2 Best Practice Guide - Parallel I/O 6.6.1. POSIX and the C Standard Library ........................................................................ 33 6.6.2. Fortran Files ...................................................................................................... 35 6.7. I/O Libraries Summary .................................................................................................. 38 7. I/O Performance Analysis ......................................................................................................... 39 7.1. Introduction .................................................................................................................. 39 7.2. Darshan Tool ............................................................................................................... 39 7.3. Darshan Runtime .......................................................................................................... 39 7.3.1. Using the Darshan Runtime .................................................................................. 40 7.4. Darshan Util ................................................................................................................. 41 7.4.1. Darshan Plots: FLASH-IO Benchmark .................................................................... 41 7.5. Vampir ........................................................................................................................ 44 7.5.1. Using Vampir .................................................................................................... 45 7.6. MPI-IO Reporting with Cray MPICH ............................................................................... 52 Further documentation ................................................................................................................. 54 3 Best Practice Guide - Parallel I/O 1. Introduction 1.1. About this Document This best practice guide provides information about High Performance I/O systems, parallel I/O APIs and I/O optimisation techniques. The guide presents a description of the storage architecture and the I/O software stack. 1.2. Guide Structure The best practice guide is split into chapters covering specific