The International Maritime Meteorological Archive (IMMA) Format
Total Page：16
File Type：pdf, Size：1020Kb
The International Maritime Meteorological Archive (IMMA) Format Shawn R. Smith1, Eric Freeman2,3, Sandra J. Lubker4, Scott D. Woodruff2,5, Steven J. Worley6, William E. Angel2, Dave I. Berry7, Philip Brohan8, Zaihua Ji6, Elizabeth C. Kent7 1Center for Ocean-Atmospheric Prediction Studies (COAPS), Florida State University, Tallahassee, USA 2 NOAA National Centers for Environmental Information (NCEI), Asheville, USA 3STG. Inc., Asheville, USA 4 Formerly NOAA Earth System Research Laboratory (ESRL), Boulder, USA 5 Cooperative Institute for Research in Environmental Sciences (CIRES), Boulder, USA 6National Center for Atmospheric Research (NCAR), Boulder, USA 7National Oceanography Centre (NOC), Southampton, UK 8Met Office, Exeter, UK (DRAFT) Updated Report (15 September 2016) 1 Table of Contents Introduction ........................................................................................................... 3 Background ........................................................................................................... 4 Original IMMA0 format structure ........................................................................... 6 Original IMMA0 format implementation ................................................................. 7 Format Modifications Resulting in IMMA1 ............................................................. 9 References ......................................................................................................... 12 Supplement A: Existing Formats and Codes ...................................................... 17 Early historical logbook formats ....................................................................... 17 WMO International Maritime Meteorological (IMM) formats ............................ 18 Alphanumeric telecommunication codes ......................................................... 19 WMO Table-Driven Code Forms (BUFR/CREX) ............................................. 20 Historical Sea Surface Temperature (HSST) Data Project formats ................. 22 Dual-record digitization formats ....................................................................... 22 ICOADS Long Marine Report (LMR) formats .................................................. 23 NOAA National Climatic Data Center (NCDC) TD-11 formats ........................ 23 Supplement B: Comparison of IMMA with WMO’s SHIP Code and IMM Formats ............................................................................................................................ 24 Supplement C. Record Types ............................................................................. 31 Supplement D. Field Configurations ................................................................... 47 Core (C0) ........................................................................................................ 48 ICOADS (Icoads) attm (C1) ............................................................................. 62 IMMT-5/FM 13 (Immt) attm (C5) ...................................................................... 76 Model quality control (Mod-qc) attm (C6) ........................................................ 84 Ship metadata (Meta-vos) attm (C7) ............................................................... 86 Near-surface oceanographic data (Nocn) attm (C8) ........................................ 92 Edited Cloud Report (Ecr) attm (C9) ............................................................... 94 Reanalysis QC/feedback (Rean-qc) attm (C95) .............................................. 98 ICOADS Value-Added Database (Ivad) attm (C96) ....................................... 100 Error (Error) attm (C97) ................................................................................. 103 Unique report identifier (Uida) attm (C98) ..................................................... 104 Supplemental data (Suppl) attm (C99) .......................................................... 105 Supplement E: ICOADS Release 3.0 IMMA Status .......................................... 107 Supplement F: Proposed IMMA Attachments ................................................... 109 2 Introduction With continuing recognition of the importance of upgrading and maximizing the data available for analyses of the climate record (e.g., Trenberth et al. 2002), efforts intensified over the last 15 years to digitize additional historical ship data (and metadata) that exist in many national logbook collections (Diaz and Woodruff 1999, Woodruff et al. 2004). Efforts over that period have focused on data during major gaps in the existing record, such as the two world wars, and adding 19th century and earlier data (e.g., Elms et al. 1993, Manabe 1999, García-Herrera et al. 2005, Woodruff et al. 2005, Brohan et al. 2009, Wilkinson et al. 2011, Kaspar et al. 2015). Many of these rescue and digitization activities have been organized and facilitated under the international Atmospheric Circulation Reconstructions over the Earth (ACRE) initiative (Allan et al. 2011). At present, however, there is no effective—and formally internationally approved— format for exchange of keyed historical data, or for exchange of more modern (e.g., non ship) data/metadata types that are also rescued or made newly available. The format needs flexibility to preserve crucial original data elements and metadata. This will help facilitate analyses of data biases and discontinuities arising from changes in instrumentation and observing practices. In addition to the exchange of newly digitized data, the format should be useful for reformatting, exchanging, and archiving existing national digital marine datasets, including contemporary marine data and metadata. Moreover, the format should be expandable, to meet new requirements that are not presently anticipated, but also simple enough that it is practical to implement by Member countries. The International Maritime Meteorological Archive (IMMA) format described herein meets these requirements. This document describes a new version of the IMMA format, IMMA1, that has been implemented for Release 3.0 (R3.0; Freeman et al. 2016) of the International Comprehensive Ocean-Atmosphere Data Set (ICOADS; Woodruff et al. 2011). The initial version of IMMA, i.e., IMMA0, was used to distribute and archive Release 2.5 (R2.5) ICOADS observational data (see http://icoads.noaa.gov/e-doc/imma/R2.5- imma.pdf). IMMA0 has also been used for the Climatological Database for the World’s Ocean (CLIWOC) project (García-Herrera et al. 2005), as an alternative output from the JCOMM Global Collecting Centres (GCCs), and is helping meet requirements for managing data from the JCOMM Voluntary Observing Ship (VOS) Scheme, including from its enhanced class of VOS Climate (VOSClim) ships. Following its introduction to the Subgroup on Marine Climatology (JCOMM 2000), development of the initial IMMA0 version has been reviewed periodically by JCOMM’s successor group, the Expert Team on Marine Climatology (ETMC) (JCOMM 2007, Woodruff 2007, JCOMM 2010, JCOMM 2012a, JCOMM 2015b, JCOMM 2016a). Moreover, the Second, Third, and Fourth “CLIMAR” Workshops on Advances in Marine Climatology (Parker et al. 2004, Charpentier et al. 2008, JCOMM 2015a), and the First, Second, Third, and Fourth “MARCDAT” Workshops on Advances in the Use of Historical Marine Climate Data (Diaz et al. 2002, Kent et al. 2007b, JCOMM 2016b), have all discussed the format, and regularly endorsed continued usage and expansion of the format. 3 The Background section of this document (together with Supps. A-B) describes the evolution of meteorological codes, and a variety of existing formats used for exchange and archival of marine data. This material also discusses strengths and weaknesses in these formats that helped define the requirements for the IMMA format. The format structure and technical options considered for format implementation for the original IMMA0 format are then discussed followed by an introduction to the current IMMA format (hereafter IMMA1). Supps. C-E document in detail the IMMA1 format as presently implemented for ICOADS, and lastly Supp. F looks to the future by outlining additional storage structures that appear desirable, in subsequent IMMA versions, to further advance the format to meet marine climate data stewardship objectives. Background International agreement to systematically record weather observations in ships’ logbooks was reached at the 1853 Maritime Conference held at Brussels (Maury 1854, JCOMM 2004b), but large quantities of earlier ship logbook records (largely pre-instrumental) are available extending back to about 1600 (Diaz and Woodruff 1999, García-Herrera et al. 2005). Around 1951, WMO took over from its predecessor International Meteorological Organization (IMO) the VOS Scheme (WMO 1973), including an early International Maritime Meteorological (IMM) punched card format (Yoshida 2004, WMO 1952). The international exchange of digitized logbook data in IMM formats was further formalized by WMO (1963) Resolution 35 (Cg-IV). However, maritime nations had earlier programs to digitize historical ship logbook data, and copies of many of the available digital collections of historical logbook data were exchanged (e.g., on punched cards in national formats; Verploegh 1966) through bilateral agreements (Woodruff et al. 2005). Many of these historical (plus real-time) data sources were compiled into global collections such as the Comprehensive Ocean- Atmosphere Data Set (COADS) (Slutz et al. 1985, hereafter Release 1; Woodruff et al. 1987), thus making