The HDF Group
HDF5 Overview
Elena Pourmal [email protected] The HDF Group
10/17/15 ICALEPCS 2015 1 www.hdfgroup.org Outline
• The HDF Group company • Products and services • Overview of HDF5 • What is coming in HDF5 1.10.0 release? • Future directions
10/17/15 ICALEPCS 2015 2 www.hdfgroup.org THE HDF GROUP COMPANY
10/17/15 ICALEPCS 2015 3 www.hdfgroup.org Champaign, Illinois, USA
10/17/15 ICALEPCS 2015 4 www.hdfgroup.org The HDF Group www.hdfgroup.org • Not-for-profit company (since 2006), ex-NCSA at University of Illinois • Offices in 5 states • About 40 employees (more than 50% growth in the past 9 years) - Core software developers - Domain specialists - Documentation team - Technical support • Mission-driven
10/17/15 ICALEPCS 2015 5 www.hdfgroup.org The HDF Group Mission
To ensure long-term accessibility of HDF data through sustainable development and support of HDF technologies.
10/17/15 ICALEPCS 2015 6 www.hdfgroup.org The HDF Group philosophy
• Committed to Open Source • HDF software is free • BSD type of license • Community involvement • Testing • Patches • New features (e.g., CMake support) • Serving diverse user base • Remote sensing, HPC, non-destructive testing, medical records, scientific modeling, etc.
10/17/15 ICALEPCS 2015 7 www.hdfgroup.org Revenue by Source
Light Sources 2014 3% 0% Earth science 4% Finance
28% General NASA, NOAA Na onal Labs HPC 62% Oil & gas 3% 0% Par cle science
10/17/15 ICALEPCS 2015 8 www.hdfgroup.org Revenue by Project Type
Revenues by type of project Training and other Consul ng outreach 8% 0% R&D 22%
Development 24%
Premium support 1%
Enterprise support 45%
10/17/15 ICALEPCS 2015 9 www.hdfgroup.org PRODUCTS AND SERVICES
10/17/15 ICALEPCS 2015 10 www.hdfgroup.org The HDF Group products
• Main product: HDF Technology Suite - For managing high volume complex, heterogeneous data - Flagship: HDF5 data store - Flexible and efficient storage and I/O - Portable - Highly customizable - Misc. tools - Specialized software and tools (e.g., JPSS)
10/17/15 ICALEPCS 2015 11 www.hdfgroup.org Data challenges addressed by HDF5 HDF5 IN 5 MINUTES
10/17/15 ICALEPCS 2015 12 www.hdfgroup.org HDF5 Technology Platform
• HDF5 Abstract Data Model • Defines the “building blocks” for data organization and specification • Files, Groups, Links, Datasets, Attributes, Datatypes, Dataspaces
• HDF5 Software • Tools • Language Interfaces (C, Fortran, C++, Java) • HDF5 Library
• HDF5 Binary File Format • Bit-level organization of HDF5 file • Defined by HDF5 File Format Specification
• HDF5 Ecosystem • Tools and services (h5py, MATLAB, IDL, OPeNDAP, etc.) • Communities (Earth Sciences, medical imaging, modeling and visualization) • Community standards (NeXus, HDF-EOS5, h5part, CGNS) • Institutional support and endorsement (NASA, NOAA, DOE)
10/17/15 ICALEPCS 2015 13 www.hdfgroup.org Members of the HDF community
10/17/15 ICALEPCS 2015 14 www.hdfgroup.org Success stories
• Petabytes of NASA remote sensing data in HDF4 and HDF5 file formats • New NASA/JPSS missions chose HDF5 format for data archiving
Need to organize complex collections of data Long term data preservation
lat | lon | temp ----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6
Efficient, scalable storage and access
10/17/15 ICALEPCS 2015 15 www.hdfgroup.org Success story: Trillion Particle Simulation
• Physics plasma simulation at NERSC Cray XE6 • Simulation ran on 120,000 cores using 80% of computing resources 90% of available memory 50% of Lustre scratch system and writing 10 one-trillion particle dumps of 30-42 TBs in HDF5 files; sustained ~ 27 GB/sec; total 350 TBs in HDF5
10/17/15 ICALEPCS 2015 16 www.hdfgroup.org The HDF Group services
• Helpdesk and mailing lists - [email protected] - [email protected] - Open to all users of HDF • HDF5 Documentation https://www.hdfgroup.org/HDF5/doc/index.html • HDF Examples (C, Fortran, C++, Java, Python, MATLAB) https://www.hdfgroup.org/HDF5/examples/
10/17/15 ICALEPCS 2015 17 www.hdfgroup.org The HDF Group services
• Standard support • Assistance in general areas of HDF usage • Premium support • Access to our consulting and training resources • Limited consulting hours are included • Enterprise support • Help with developing common strategies for managing HDF data within organization • Organization shares consulting/troubleshooting services • Training • Consulting, custom development and support
10/17/15 ICALEPCS 2015 18 www.hdfgroup.org New Upcoming Features HDF5 1.10.0 RELEASE
10/17/15 ICALEPCS 2015 19 www.hdfgroup.org Reusing free file space in a file PERSISTENT FILE FREE SPACE TRACKING
10/17/15 ICALEPCS 2015 20 www.hdfgroup.org Unused space in HDF5 file
• HDF5 library currently only tracks free space while file is open • Space from deleted objects • Space from resized compressed chunks • Free space in the file is “lost” after file is closed • h5repack is used to remove “holes” in the file • New function H5Pset_file_space • Sets a property to track free space in the file that can be reused when file is reopened • Allows fine tuning space tracking
10/17/15 ICALEPCS 2015 21 www.hdfgroup.org
Improving performance and saving space SCALABLE CHUNK INDEXING
10/17/15 ICALEPCS 2015 22 www.hdfgroup.org Optimizing chunking storage and performance
• HDF5 has an ability to add more data to existing datasets (data arrays) • Special storage mechanism – chunked storage • B-trees are used to index chunks in the file • O(log n) lookup time • HDF5 takes advantage of the access pattern and properties of the datasets • O(1) lookup time • File space savings when storing HDF5 metadata
10/17/15 ICALEPCS 2015 23 www.hdfgroup.org
Optimizing chunking storage and performance
• B-tree implementation was reworked to use less space in the file • Used for datasets with more than one unlimited dimension • New indexing structures were introduced to achieve O(1) performance and storage savings in special cases
10/17/15 ICALEPCS 2015 24 www.hdfgroup.org Optimizing chunking storage and performance • Examples of O(1) lookup access: • Fixed-size chunked dataset with no compression filters • Algorithmic lookup • Fixed-size chunked dataset with compression filters • Array to index chunks • Fixed-size dataset stored in one chunk (i.e., we now allow compression for contiguous dataset) • No index • Dataset with one unlimited dimension • Extensible array to index chunks
10/17/15 ICALEPCS 2015 25 www.hdfgroup.org
CONCURRENCY: SINGLE-WRITER/MULTIPLE- READER
10/17/15 ICALEPCS 2015 26 www.hdfgroup.org Concurrent Access to Data
New data elements …
Writer Reader
…which can be read … are added by a reader… to a dataset HDF5 File in the file… with no IPC necessary.
10/17/15 ICALEPCS 2015 27 www.hdfgroup.org Managing data stored across HDF5 files VIRTUAL DATASET (VDS)
10/17/15 ICALEPCS 2015 28 www.hdfgroup.org VDS Use Case with NPP satellite data
4 granules in 9 GMODO-SVM07… files
Visualization with IDV
10/17/15 ICALEPCS 2015 29 www.hdfgroup.org VDS Use Case with NPP satellite data
One virtual dataset with 36 granules stored in one file
Visualization with IDV
10/17/15 ICALEPCS 2015 30 www.hdfgroup.org VDS use case: Percival detector
Series of images
D C B A
t3+4k
t1+4k t4 t3 Virtual Dataset VDS has images A, B, C and D interleaved t2 t1 reader
VDS.h5
writer writer writer writer
10/17/15Dataset A Dataset B Dataset C Dataset D A B C D
a.h5 b.h5 c.h5 d.h5 31 www.hdfgroup.org VDS: Conceptual View
32 10/17/15 www.hdfgroup.org Performance boost when opening and closing HDF5 files METADATA CACHE IMAGE
10/17/15 ICALEPCS 2015 33 www.hdfgroup.org Problem: Metadata Cache Image
! HDF5 metadata is typically small and scattered throughout the file.
! Resulting many small I/Os a major problem for parallel file systems.
! Metadata cache minimizes this during normal operation, but must still populate cache on file open, and flush it on file close.
! Problem if files are opened and closed often.
10/17/15 ICALEPCS 2015 34 www.hdfgroup.org Solution: Metadata Cache Image
! Store the contents of the metadata cache in a single block at file close, and then populate the cache with the stored entries on file open.
! If access pattern is similar over close and reopen, should save a significant number of small I/O operations.
! This solution is implemented in the metadata cache image feature.
10/17/15 ICALEPCS 2015 35 www.hdfgroup.org Metadata Cache Image
! To enable, set cache image FAPL property on file create or open: H5AC_cache_image_config_t cache_image_config = {H5AC__CURR_CACHE_IMAGE_CONFIG_VERSION, TRUE, 0}; fapl_id = H5Pcreate(H5P_FILE_ACCESS); H5Pset_libver_bounds(fapl_id, H5F_LIBVER_LATEST, H5F_LIBVER_LATEST); H5Pset_mdc_image_config(fapl_id, &cache_image_config);
! Then create or open file as usual.
10/17/15 ICALEPCS 2015 36 www.hdfgroup.org Metadata Cache Image
! Metadata cache image is read and deleted automatically on file open.
! Must set cache image FAPL property again if a new cache image is desired on file close.
! Earlier versions of HDF5 that don't understand the cache image will refuse to open the file.
! One can use a light-weight utility to remove caching info making file compatible with 1.8
! Prototype implementation showed order of magnitude speedup on parallel systems
10/17/15 ICALEPCS 2015 37 www.hdfgroup.org Performance imporvemnts DATA AGGREGATION AND PAGE BUFFERING
10/17/15 ICALEPCS 2015 38 www.hdfgroup.org Page buffering/ Data aggregation
Aggregate and align metadata and small data, perform I/O in aligned pages
10/17/15 39 www.hdfgroup.org Data and Metadata Aggregators
The new aggregators pack small raw data and metadata allocations into aligned blocks which work with the page buffer.
Data Metadata
HDF5 File
Small alloca ons
10/17/15 ICALEPCS 2015 40 www.hdfgroup.org HDF5 Page Buffering
Page buffer contains MD pages (L2 cache)
Metadata blocks are aligned
HDF5 File
Metadata blocks are mul ples of 64K
10/17/15 41 www.hdfgroup.org HDF5 Parallel IMPROVEMENTS FOR PARALLEL ACCESS
10/17/15 ICALEPCS 2015 42 www.hdfgroup.org Problems We Solved for PHDF5
• Slowness on opening and closing HDF5 files " Metadata Cache Optimizations - Avoiding the Metadata Read Storm - Collective Metadata Writes " Avoid Truncate Feature • Writing/reading multiple variables " Collective I/O on multiple datasets or Multi-Dataset I/O • I/O on selections bigger than 2GB with MPICH 3.1.4 • Page Buffering " Page Buffering - a layer under the VFD to capture small I/Os and cache them for larger paged size I/Os.
10/17/15 ICALEPCS 2015 43 www.hdfgroup.org Metadata reads with CGNS and netCDF-4
CGNS reads on Blue Gene, GPFS
netCDF-4 reads on Cray XE6 , GPFS
10/17/15 ICALEPCS 2015 44 www.hdfgroup.org Collective I/O on multiple datasets
• Two new rou nes H5Dread_mul () and H5Dwrite_mul ()
The plot shows the performance difference between using a single H5Dwrite() mul ple mes and using H5Dwrite_mul () once on 30 chunked datasets on Cray XE-6 with Lustre file system (hopper).
10/17/15 ICALEPCS 2015 45 www.hdfgroup.org HDF5 1.10.0 BACKWARD/FORWARD COMPATIBILITY ISSUES
10/17/15 ICALEPCS 2015 46 www.hdfgroup.org Backward/Forward compatibility issues
• HDF5 1.10.0 will always read files created by the earlier versions • HDF5 1.10.0 by default will create files that can be read by HDF5 1.8.* • HDF5 1.10.0 will create files incompatible with 1.8 version if new features are used • Tools to “downgrade” the file created by HDF5 1.10.0 " h5format_convert (SWMR files; doesn’t rewrite raw data) " h5repack (VDS, SWMR and other; does rewrite data)
10/17/15 47 www.hdfgroup.org Examples EXPLORING NEW DIRECTIONS
10/17/15 ICALEPCS 2015 48 www.hdfgroup.org HDF5 ODBC Driver
• Open DataBase Connectivity (ODBC) • Industry standard middleware API for accessing database management sys. • All analytics apps. have an ODBC client • HiFive – ODBC driver for HDF5 • Windows, [Linux, MacOS X] • Client & Client/Serve • Accessing HDF5 files from Excel & R Thanks to Gerd Heber, THG 10/17/15 ICALEPCS 2015 49 www.hdfgroup.org HDF5 for the Web
• Can I access HDF5 files remotely? • API? My (mobile) client speaks HTTP! • What is a file system? Who uses files anymore? • Cloud computing w/ HDF5
Thanks to John Readey, THG
10/17/15 ICALEPCS 2015 50 www.hdfgroup.org Emerging Trends in Exascale I/O
• Characteristics of Exascale Application I/O • Application I/O will be object-oriented, not file- based • Application I/O will be asynchronous • Applications responsible for managing I/O conflicts • Applications use transactional I/O model • The HDF Group has been working with Intel and others on the Fast Forward Project to investigate and contribute to those trends
10/17/15 ICALEPCS 2015 51 www.hdfgroup.org HDF5 role in the Fast Forward Storage Stack
• Object storage • Virtual Object Layer (VOL) • Data Integrity/ Fault Tolerance • Transaction • End-to-end checksums • Data Analysis Extensions • Query/View/Index APIs • Analysis Shipping • AIO (some prototyping was done in the past)
10/17/15 ICALEPCS 2015 52 www.hdfgroup.org HDF5 as an interface to non-HDF5 storage
h ps://wiki.hpdd.intel.com/display/PUB/Fast+Forward+Storage+and+IO+Program+Documents 10/17/15 ICALEPCS 2015 53 www.hdfgroup.org HDF5 as an interface to non-HDF5 storage
• Different File Formats plugins:
10/17/15 ICALEPCS 2015 54 www.hdfgroup.org Features we are investigating DATA INDEXING
10/17/15 ICALEPCS 2015 55 www.hdfgroup.org Indexing and HDF5 • New APIs for indexing and querying of both structure and contents of HDF5 file • H5Q API defines query to apply to a file Create/combine queries (OR, AND) • Basic operators supported (≤ , ≥ ,=, ≠ ) on either dataset/attribute values, link/attribute names • HDF5V API retrieves data • HDF5X API adds third-party indexing plugins
56 10/17/15 www.hdfgroup.org Example: Combined query
10/17/15 57 www.hdfgroup.org The HDF Group
Thank You!
Questions?
10/17/15 ICALEPCS 2015 58 www.hdfgroup.org