1 Proposed IMMA Revisions
Total Page:16
File Type:pdf, Size:1020Kb
Proposed IMMA Revisions Revised DRAFT, 19 August 2015 (v21) Scott Woodruff, Dave Berry, Philip Brohan, Eric Freeman, Zaihua Ji, Elizabeth Kent, Sandy Lubker, Shawn Smith, Steve Worley et al. Introduction Format changes—to yield IMMA version 1—are marked below in blue within Supplement C of the current IMMA (version 0) documentation. The abbreviated version of the documentation (http://icoads.noaa.gov/e-doc/imma/R2.5-imma_short.pdf) includes Supp. D, providing details on individual field configurations; whereas the complete documentation (http://icoads.noaa.gov/e-doc/imma/R2.5-imma.pdf) also provides additional background in its main text and in Supps. A-B, including more detailed comparisons between IMMA and international exchange (IMMT/IMMPC) formats. Detailed background/discussion notes for the Core and attachments (attms) appear following each table below. When significant changes or additions to attm content were needed (i.e. for Immt, Mod-qc, and Meta-vos) a revised attm was created with a new table number and new attachment ID (ATTI), but the old configuration was retained as a deprecated attm. The Nocn, Ecr, Rean-qc, Ivad, Error, and Uida attms are new, and the Auto, Nocq, Alt-qc, Track, and Hist attms are proposed (with many details of the proposed attms to be finalized; including any new field abbreviations, which need to be checked for uniqueness to avoid possible overlaps with already defined field abbreviations). The following items discuss aspects of the planned changes in greater detail: 1. Abbreviated structural element names in italics: A new IMMA documentation feature— to enhance communication—is that the Core and attms (e.g. Icoads) are all given abbreviated names in italics, but with only the first letter in uppercase, to distinguish them from the fully capitalized IMMA field abbreviations. 2. Switch to attm-internal field numbering: E.g. within the Icoads attm, the fields are now numbered 1-51 rather than 49-99. Otherwise documentation maintenance was becoming challenging, and in conjunction with the IVAD project this is viewed as a more flexible approach. However, the revised Fortran program to read/write IMMA1 ({rwimma1}1) still utilizes a linear field numbering approach for assigning array storage across the Core and all attms. 3. Operational and deprecated attms: Rather than change (i.e. add/subtract, or modify, fields) attms, a new attm version is created, with tables for the deprecated attms tentatively retained only in the last-version format documentation (and noting that by itself the item 2 field numbering switch is not considered such a change). The {rwimma1} program is able to read both the operational and deprecated attms, and the format remains fully backward compatible. Related also to item 2, the new Rean-qc, Ivad, and Error attms include fields for input component number (i.e. 0=Core or ATTI) (ICNR/ICNI/ICNE) and field number within the input component number (FNR/FNI/FNE). 1 In this document, {braces} are used to mark the names of Fortran software, with {rwimma1} referring specifically to http://icoads.noaa.gov/software/rwimma1. With limited modifications, this program can also write IMMA1. In terms of processing new attms, the current program will process the Nocn, Ecr, Rean-qc, Ivad, Error, and Uida attms, including processing of Subsidiary records. 1 4. Additional software maintenance considerations: To make translation software adapted from {rwimma} (e.g. existing adaptations of {rwimma1} used to translate data from other formats into IMMA) more robust over the longer term, usage of field abbreviations (e.g. FTRUE(SST)) rather than hard-coded storage array locations (e.g. FTRUE(35)) appears prudent with the advantage e.g. W2 is defined in both Tables C2 and C5. 5. Switch to multi-record “linked-report” approach: Rather than modifying the Icoads attm as originally proposed to include UID and associated release-tracking information etc., those fields are placed in a short new Uida attm (see Table C98), which appears both in the Main and (any optional) Subsidiary records, linking them all together, e.g.: Main IMMA record: Core + Icoads + Immt + Mod-qc + Meta-vos + Nocn + Uida + Suppl Subsidiary IMMA record: Uida + Rean-qc + Rean-qc + Rean-qc … + Rean-qc Subsidiary IMMA record: Uida + Ivad + Ivad + Ivad … + Ivad Subsidiary IMMA record: Uida + Error + Error + Error … + Error Each complete multi-record construct (i.e. Main plus Subsidiary records) is referred to below as a “linked-report.” Alternatively, information such as ship metadata (Meta-vos) attms, or the proposed alternative QC (Alt-qc) attms, might be conveyed separately back to ICOADS in a file containing only Subsidiary records, i.e. to be blended with (or possibly into fields in other attms, in the case of Alt-qc) the Main records before provision to users: Subsidiary IMMA record: Uida + Meta-vos Subsidiary IMMA record: Uida + Alt-qc + Alt-qc + Alt-qc … + Alt-qc Such Subsidiary-only records thus would not ordinarily be provided directly to users. The way this is being implemented is still anticipated to be fully backward compatible. The Core has not changed, but {rwimma1} checks the first few characters of each record to determine whether it is a Subsidiary record (i.e. starts with “9815”). 6. Software constraints on linked-reports and attm composition: Processing by {rwimma1} of Main+Subsidiary records (linked-reports), and of the Ivad and Error attms, presently is being implemented with these fairly loose constraints: (i) Both Main and Subsidiary records are allowed by {rwimma1} to contain multiples of any attachment in any order; except the Suppl attm, if present, must be the last Main attm. (ii) In the event of repeating (except Ivad and Error) attms, the last attm takes precedence and overwrites information from all previous (e.g. Subsidiary attms take precedence over any repeating Main attms). (iii) Subsidiary records must each begin with a Uida attm, followed by zero or more attms of any type except Suppl. (iv) The maximum number of Ivad and Error attms within a linked-report is set at 100 each. In addition, while by definition of UID two (or more) Main records should never appear with the same UID, no check is envisioned as feasible at the present time for UID uniqueness. 7. Status of new-field content: In the revised attms (Immt, Mod-qc, and Meta-vos) newly defined fields generally will not be populated (i.e. in the prototype Release 2.5 dataset), with the exception of field MDS in the Meta-vos attm (see also related discussion in Annex C). In the new Uida attm, UID is set as discussed below Table C98, and the Release number fields are set as RN1=2, RN2=5, RN3=1, i.e. this is considered R2.5.1, to distinguish it from the original R2.5. 2 Table 1 provides a summary of the development status of the IMMA components, and Table 2 outlines different record structures currently available with Release 2.5 (R2.5), the prototype Release 2.5.1 (R2.5.1), or under development for Release 3.0 (R3.0). These Annexes provide additional background information: • Annex A: Implementation/Unresolved Issues • Annex B: Development of the Unique Report ID (UID); and Intra-record Release No. (RN) Tracking • Annex C: Reprocessing Notes for Recovering Missing Field Configurations, Etc. • Annex D: QC Flag Discussion • Annex E: Edited Cloud Report (Ecr) attm • Annex F: Author Reference Code (ARC) Registration and Storage • Annex G: Discussion of ICOADS Future IMMA/IVAD Archival Structure NOTE: strikethrough in Suppl. C below indicates updates are still needed, or marks material that has been moved elsewhere. Table 1. Status of IMMA components and related information. Shaded components are deprecated. Component Abbrev. Status Reference Length information (char.) Core Core operational Table C0 108 ICOADS attm Icoads operational Table C1 65 IMMT-2/FM 13 attm (none) deprecated Table C2 (sep. doc.) 76 Model quality control attm (none) deprecated Table C3 (sep. doc.) 66 Ship metadata attm (none) deprecated Table C4 (sep. doc.) 67 IMMT-5/FM 13 attm Immt operational Table C5 94 Model quality control attm Mod-qc operational Table C6 68 Ship metadata attm Meta-vos operational Table C7 58 Near-surface oceano. attm Nocn operational Table C8 102 Edited cloud report attm Ecr operational Table C9 – Annex E 32 Reanalyses QC/feedback attm Rean-qc operational Table C95 61 IVAD attm Ivad operational Table C96 53 Error attm Error operational Table C97 variable Unique report ID attm Uida operational Table C98 15 Supplemental data attm Suppl operational Table C99 variable Automated instrument. attm Auto proposed Table CP1 41 Near-surface oceano. QC attm Nocq proposed Table CP2 28 Alternative QC attm Alt-qc proposed Table CP3 TBD Platform tracking attm Track proposed Table CP4 TBD Historical attm Hist proposed Table CP5 TBD Buoy metadata attm Meta-buoy proposed TBD TBD Daily observational (Daily) attm Daily proposed TBD TBD 3 Table 2. IMMA record structures currently operational for R2.5, R2.5.1, and the “linked-report” Main + Subsidiary (optional) record structure being implemented for R3.0. Inclusion of the attm count (ATTC) field in the Core, and of the attm ID (ATTI) and attm data length (ATTL) fields at the beginning of each attm, enables computer parsing of the records. Thus additional variations on these basic record types are implemented by inclusion or omission of attms, and new attms can be defined in the future as needed for new data or metadata requirements.1 Release Record structure Len. (char.)2 R2.5 Core + Icoads + Immt2 + Mod-qc3 + Meta-vos3 [+ Suppl] 372 R2.5.1 Core + Icoads + Immt + Mod-qc + Meta-vos + Uida [+ Suppl] 408 R3.0: Main Core + Icoads + Immt + Mod-qc + Meta-vos + Nocn + Ecr + Uida [+ Suppl] 544 Subsid.