175-WP-001-002 An HDF-EOS and Data Formatting Primer for the ECS Project White Paper March 2001 Prepared Under Contract NAS5-60000 RESPONSIBLE ENGINEER E. M. Taaheri /s/ 3-1-2001 David Wynne, Abe Taaheri Date EOSDIS Core System Project SUBMITTED BY Larry Klein /s/ 2-28-2001 Darryl Arrington, Manager Toolkit Date Larry Klein, Manager Toolkit EOSDIS Core System Project Raytheon Company Upper Marlboro, Maryland This page intentionally left blank. Abstract This document is a release of a primer manual for using the Hierarchical Data Format (HDF) and extensions (HDF-EOS) in Earth Observing System (EOS), Version 1 and later. We confine our discussion in this document to HDF V4 and the versions of HDF-EOS (V2), which are based on that version of HDF. A new version of HDF (V5) has been released. Additional material for HDF5 and versions of HDF-EOS, based on HDF5 will be provided in a separate document. Comments and suggestions should be sent to: Larry Klein Emergent Information Technologies 9315 Largo Dr. West, Suite 250 Largo, MD 20774 USA Email: [email protected] Phone: (301) 925-0764 Fax: (301) 925-0321 Keywords: HDF, HDF-EOS, Swath, Grid, Point, Data Formats, Metadata, Standard Data Products, Disk Formats, Browse iii 175-WP-001-002 This page intentionally left blank. iv 175-WP-001-002 Table of Contents Abstract 1. Introduction 1.1 Purpose of this Document.......................................................................................... 1-1 1.2 Organization of this Document .................................................................................. 1-1 1.3 HDF and EOS: Introduction ...................................................................................... 1-1 1.4 HDF and EOS: General Philosophy........................................................................... 1-3 2. An Introduction to HDF 2.1 HDF Concepts........................................................................................................... 2-1 2.2 Overview of the NCSA HDF Library ........................................................................ 2-2 2.2.1 The Scientific Dataset Interfaces (DFSD, SD)................................................ 2-4 2.2.2 The Vdata Interfaces (VS, VSQ, VF)............................................................. 2-5 2.2.3 Vgroup interface (V)...................................................................................... 2-6 2.3 Overview of the HDF Disk Format............................................................................ 2-7 2.3.1 The HDF File Header..................................................................................... 2-7 2.3.2 The HDF Directory........................................................................................ 2-7 2.3.3 HDF Data Descriptors.................................................................................... 2-8 2.3.4 HDF Groups ..................................................................................................2-11 2.3.5 HDF Scientific Datasets.................................................................................2-14 2.3.6 HDF Vdatas ...................................................................................................2-16 2.3.7 HDF Extended Tags.......................................................................................2-17 2.4 HDF Examples ..........................................................................................................2-18 2.4.1 8-bit Raster Image Output ..............................................................................2-18 2.4.2 8-bit Raster Image with Palette Output...........................................................2-19 2.4.3 8-bit Raster Image with Palette Input .............................................................2-19 v 175-WP-001-002 2.4.6 Scientific Data Set Output..............................................................................2-20 2.4.7 Scientific Data Set Input ................................................................................2-21 3. HDF-EOS V2 3.1 Introduction............................................................................................................... 3-1 3.2 HDF and HDF-EOS File Formats.............................................................................. 3-1 3.2.1 Overview ....................................................................................................... 3-1 3.2.2 Structure of an HDF-EOS File ....................................................................... 3-2 3.3 HDF-EOS Library Functionality................................................................................ 3-5 3.3.1 Point Data Interface ....................................................................................... 3-6 3.3.2 Swath Interface ..............................................................................................3-10 3.3.3 Grid Interface.................................................................................................3-27 4. Related Topics 4.1 Introduction............................................................................................................... 4-1 4.2 ECS Metadata and the Science Data Processing Toolkit ............................................ 4-1 4.3 ECS Browse Specification......................................................................................... 4-6 4.3.1 Overview ....................................................................................................... 4-6 4.3.2 Browse Package Guidelines ........................................................................... 4-7 4.4 EOSView: An HDF-EOS ‘Browse Tool’.................................................................. 4-9 4.4.1 Introduction ................................................................................................... 4-9 4.4.2 EOSView Features........................................................................................4-10 List of Figures 2-1. HDF call to write an 8-bit image..................................................................................... 2-2 2-2. The HDF physical file format supports three levels of interaction ................................... 2-3 2-3. A Typical Vdata.............................................................................................................. 2-6 2-4. Organizational Levels for Dataset Shown in Table 2-8.................................................. 2-13 2-5. Example SDS................................................................................................................ 2-14 3-1. A Simple Point Data Example......................................................................................... 3-6 3-2. Recording Points Over Time........................................................................................... 3-7 vi 175-WP-001-002 3-3. Point Data from a Moving Platform ................................................................................ 3-7 3.4. A Typical Satellite Swath: Scanning Instrument............................................................ 3-10 3-5. A Swath Derived from a Profiling Instrument............................................................... 3-11 3-6. A “Normal” Dimension Map......................................................................................... 3-13 3-7. A “Backwards” Dimension Map ................................................................................... 3-13 3-8. HDF Objects Created by Program: SetupSwath.c.......................................................... 3-18 3-9. HDF Objects Created by Program: DefineField.c........................................................... 3-21 3-10. A Data Field in a Mercator-projected Grid .................................................................. 3-27 3-11. A Data Field in an Interrupted Goode’s Homolosine-Projected Grid ........................... 3-28 3-12. HDF Objects Created by Program: SetupGrid.c .......................................................... 3-34 4.1. Browse Package Data Objects ......................................................................................... 4-7 4-2. A Browse Data Package.................................................................................................. 4-8 4-3. EOSView Main Window .............................................................................................. 4-10 4-4. EOSView File Contents Window.................................................................................. 4-10 4-5. EOSView Image Display Window................................................................................ 4-11 4-6. EOSView Help Display ................................................................................................ 4-12 4-7. EOSView Table Display ............................................................................................... 4-13 4-8. EOSView Text Display................................................................................................. 4-13 List of Tables 2-1. The HDF Interfaces......................................................................................................... 2-4 2-2. Organization of the beginning of an HDF file ................................................................. 2-8 2-3. HDF data descriptor layout ............................................................................................. 2-8 2-4. Ranges of possible tag values.........................................................................................