Near Real Time Altimeter Validation System User Handbook
Shailen D. Desai Melissa Soriano Bruce J. Haines with contributions from
John Lillibridge (NOAA) Olivier Thepaut (EUMETSAT) Timothy Patterson (EUMETSAT)
Version 2.5 May 1, 2012
Jet Propulsion Laboratory California Institute of Technology
JPL D-35859
Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Copyright Notice
Copyright 2007, by the California Institute of Technology. ALL RIGHTS RESERVED. United States Government Sponsorship (National Oceanic and Atmospheric Administration) acknowledged. Any commercial use must be negotiated with the Office of Technology Transfer at the California Institute of Technology.
Export Notice
The Near Real Time Altimeter Validation System software package is subject to U.S. export control laws and regulations. (22 C. F. R. 120-130 and 15 C. F. R. 730-774). To the extent that the software is subject to U.S. export control laws and regulations, the recipient has the responsibility to obtain export licenses or other export authority as may be required before exporting such information to foreign countries or providing access to foreign nationals.
i Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012
TABLE OF CONTENTS
1. Introduction ...... 1 2. System Requirements and Installation ...... 2 2.1 System Requirements ...... 2 2.2 NRTAVS Software Package ...... 2 2.3 Installation ...... 4 3. The NRTAVS Programs ...... 6 3.1 nrtavs_quicksetup.py ...... 6 3.2 nrtavs_main.py ...... 7 3.3 nrtavs_report.py ...... 9 3.4 nrtavs_computestats ...... 11 3.5 NRTAVS Data Files ...... 12 3.5.1 Statistics File Format ...... 12 3.5.2 Map File Format ...... 14 3.6 Log Messsages ...... 15 3.7 Date, Time, Units, and Data Windows used by the NRTAVS ...... 16 3.8 Altimeter Data File Types ...... 16 3.8.1 Currently Supported Data File Types ...... 16 3.8.2 Modification of the NRTAVS to Support New Data File Types ...... 17 3.9 Parameter Types ...... 17 3.9.1 Currently Supported Parameter Types ...... 17 3.9.2 Modification of the NRTAVS to Support New Parameter Types ...... 19 4. Main Configuration File ...... 20 4.1 Description ...... 20 4.2 Example ...... 23 5. Parameter Configuration File ...... 25 5.1 Description ...... 25 5.2 Example ...... 39 5.3 Data Windows for Plots ...... 41 6. Example Plots ...... 42 7. References ...... 50
ii Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 LIST OF TABLES
Table 1. Directory structure for the installed version of the NRTAVS...... 5 Table 2. Format of Statistics file...... 14 Table 3. Format of Map file...... 15 Table 4. Altimeter Data File Types Supported by the NRTAVS...... 16 Table 5. Parameter Types Supported by the NRTAVS...... 17 Table 6. Keywords for Main Configuration File...... 20 Table 7. General Keywords for Parameter Configuration Files...... 25 Table 8. Data Editing Flag Keywords in Parameter Configuration Files...... 26 Table 9. Plot and Table Generation Keywords in Parameter Configuration Files...... 31
LIST OF FIGURES
Figure 1. Flow chart showing processing steps in nrtavs_main.py. Thick arrows indicate data flow, and gray boxes indicate data directories. Flow begins at upper left of Figure in the box indicating “Start up”. Not explicitly shown is the capability for nrtavs_report.py to send PDF versions of the report to the website...... 7 Figure 2. Example of ssha Mean Plot...... 43 Figure 3. Example of ssha Sigma Plot...... 43 Figure 4. Example of ssha Data Points Plot (with log scale) ...... 44 Figure 5. Example of ssha One Cycle Per Revolution Amplitude Plot...... 44 Figure 6. Example of ssha Histogram Plot...... 45 Figure 7. Example of ssha Map Plot ...... 45 Figure 8. Example of ssha Edit Map Plot...... 46 Figure 9. Example of iono_ku Histogram Plot with Histogram_Plot_Asc_Des = ON...... 47 Figure 10. Example of iono_ku Map Plot with Map_Plot_Asc_Des = ON...... 48 Figure 11. Example of iono_ku Edit Map Plot with Edit_Map_Plot_Asc_Des = ON...... 49
iii Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012
1. Introduction
The Near Real Time Altimeter Validation System (NRTAVS) software package is an automated quality assurance system for satellite altimeter data products. It has been developed at the Jet Propulsion Laboratory (JPL), California Institute of Technology through sponsorship by the National Oceanic and Atmospheric Administration (NOAA). Version 2.1 onwards of the NRTAVS applied modifications provided by EUMETSAT (Jason-1 Operational Sensor Data Record capability), and NOAA (syslogging, encrypted password for remote web host, 64-bit platform capability, and first version of software to read Jason-1 NetCDF data products).
The system was developed to perform quality assurance on the near-real-time satellite altimeter data products from the Jason-2/Ocean Surface Topography Mission (OSTM). However the design of the system allows it to be extended to other satellite altimeter data products, including longer latency products, given an appropriate data product reader. For example, during development this system was extensively tested with the Jason-1 Interim Geophysical Data Records (IGDRs). Version 2.1 of the NRTAVS has also been tested using the original Jason-1 IGDRs (version b and c), the Jason-1 Operational Sensor Data Records, NetCDF Jason-1 IGDRs (version b and c). Version 2.1 has also been tested with the first post-launch versions of the NetCDF Jason-2 OGDR data products, which correspond to version 2.1 of the Jason-2 User Products Specification [Dumont et al, 2008a]. NRTAVS version 2.3 has been tested with sample data products that correspond to version 2.3 of the Jason-2 User Products Specification [Dumont et al, 2008b]. NRTAVS version 2.4 has been tested with operational version “C” products from the Jason-1 (OSDRs and IGDRs) and Jason-2 (OGDRs and IGDRs) missions, as well as pre-launch sample products for the SARAL/AltiKa mission (OGDR and IGDR). Version 2.5 includes enhancements provided by EUMETSAT to process two additional parameters (inverse barometer and SSHA from the product) and a new reader for OGDR-BUFR files. NOAA provided enhancements including password retrieval via gnome-keyring-query and the use of the logfile prefix globally in the PDF reports names. NOAA also upgraded the underlying Python to version 2.7.2, and upgraded several python modules (matplotlib, basemap, numpy) in going from NRTAVS version 2.4 to 2.5.
The NRTAVS is composed of three python programs, one C program, and various python libraries. It has been developed and tested on a Linux operating system.
1 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012
2. System Requirements and Installation
2.1 System Requirements The NRTAVS has been developed and tested extensively (over 1 year) on a RedHat Linux operating system 32-bit platform, and on a RedHat Linux operation system 64-bit platform.
The NRTAVS requires the availability of the following five packages on the computer where the NRTAVS is installed. Proper installation requires these packages be available prior to running the installation scripts. − GNU Compiler Collection (GCC), version 3.2.3 or newer, to compile the Python package that is provided with the NRTAVS installation package, and to compile the one C program that is part of the NRTAVS. − pdflatex, an open source program that directly converts latex files into PDF documents (see http://www.tug.org/applications/pdftex). This program is only required to generate hard copy reports of the statistics computed by the NRTAVS. If hardcopy reports are not required, then pdflatex is also not required. − The NetCDF library and associated netcdf.h header file. − gnome-keyring-devel, to retrieve unencrypted passwords from the user’s GNOME keyring. − The ECMWF BUFR library to support the OGDR-BUFR reader: http://www.ecmwf.int/products/data/software/download/bufr.html
Further information on the use of the GNOME keyring for NRTAVS password management is provided in Appendix A.
Instructions for installation of the BUFR library are included with the description of EUMETSAT provided enhancements in Appendix B. Note that NRTAVS 2.5 internally handles the environment variable and stack size increase needed for the BUFR reader, so those manual steps do NOT need to be executed with this version.
WARNING: Experience has also shown that correct compilation of the matplotlib python package requires that the following packages be installed prior to installation of the matplotlib package, and therefore the complete NRTAVS installation package (e.g., http://matplotlib.sourceforge.net/installing.html): − freetype − libpng − zlib − freetype-devel − libpng-devel − zlib-devel
2.2 NRTAVS Software Package The NRTAVS software package is provided in a tarred and gzipped package nrtavs-v.v.tar.gz, where v.v identifies the version number of the package. The NRTAVS version 2.5 package contains the following modules developed at JPL, with enhancements from NOAA and EUMETSAT:
2 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012
− nrtavs_main-0.2.5 This module contains the three main Python programs for the NRTAVS: o nrtavs_main.py This is the main program that generates statistics, plots, and web sites containing the plots and tables of statistics. It directly executes the C program nrtavs_computestats, and optionally has the capability to directly execute the program nrtavs_report.py. o nrtavs_report.py This is the main program that generates hardcopy reports of the statistics, and optionally posts these hardcopy reports on a web site and/or distributes the report via e-mail. It can be executed stand alone, and can also be directly executed by the nrtavs_main.py program. However, with either approach it requires the output from the statistical analysis performed by nrtavs_main.py. o nrtavs_quicksetup.py This program can be used to setup an operations directory, namely a directory where all computations can be performed by nrtavs_main.py. The operations directory is set up from the example input decks and plot logos that are provided with the installation package. Note that this script creates a suggested configuration for the operations directory which the user can choose to ignore.
− nrtavs_computestats-0.2.5 This module contains the C program that nrtavs_main.py uses to compute statistics from the satellite altimeter data products. It can be executed stand alone, but is automatically executed by the nrtavs_main.py program.
− nrtavs_library-0.2.5 This module contains various NRTAVS Python library functions, and templates for the web site generation capability. These include: o nrtavs_library.py This file contains the primary library functions for nrtavs_main.py and nrtavs_report.py o nrtavs_plotlib.py This file contains the plotting functions for nrtavs_main.py and nrtavs_report.py. o nrtavs_htmllib.py This file contains the web site generation functions for nrtavs_main.py. o nrtavs_reportlib.py This file contains library functions for nrtavs_report.py. o commonNOAA.css This is a web site style sheet for a NOAA web site, and is required by nrtavs_main.py. o nrtavs.css This is a web site style sheet for the NRTAVS system, and is required by nrtavs_main.py o templateNOAA.xhtml (Provided by NOAA for version 2.0 and modified by JPL) This is a template for NOAA website html files, and is required by nrtavs_main.py o ostm_banner.jpg (Provided by NOAA for version 2.0) Sample banner for the web site. o spacer.gif (Provided by NOAA for version 2.0) Sample banner for the web site.
− nrtavs_examples-0.2.5 This module contains examples of input configuration files for the NRTAVS as well as sample png files with logos for the plots.
3 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012
The NRTAVS requires the availability of the open source Python compiler and some associated open source Python packages. To ensure consistent performance of the NRTAVS on the host (installation) computer, the version of Python and required open source Python packages that were used during development and testing are also provided in the NRTAVS installation package. The open source packages that were not developed at JPL, NOAA or EUMETSAT, but which are provided in the NRTAVS installation package include:
− Python 2.7.2 − basemap-1.0.2 − elementtree-1.2.6 − matplotlib-1.1.0 − numpy-1.6.1 − PyXML-0.8.4 − xyaptu-1.0.0
The basemap-1.0.2 module is installed as a toolkit within matplotlib and includes geos-3.3.1, which is compiled as part of the module installation. The full and high resolution coastline data have been removed from the basemap module to conserve space, since they are not needed to plot the global NRTAVS maps.
Use of the NRTAVS with these versions of Python and its open source packages is highly recommended.
2.3 Installation It is best to install and run the NRTAVS as a non-root user. The following steps should be used to install the NRTAVS, where
Installation logs for Python and the various python modules can be found in the directories:
The install.sh script will install the full NRTAVS package including Python and the associated open source Python modules in the directory
4 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012
Table 1. Directory structure for the installed version of the NRTAVS. (Assumes that the installation directory is
5 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 3. The NRTAVS Programs 3.1 nrtavs_quicksetup.py The purpose of the program nrtavs_quicksetup.py is to set up a suggested directory structure where all processing of the data will occur, and to set up suggested input decks for the NRTAVS that are based on the example decks that are provided with the NRTAVS installation package. The suggested input decks should be considered to be a starting point for the user to modify. A usage statement for the program is returned by executing the program without any input options, for example: Below is the usage statement for the program. USAGE nrtavs_quicksetup.py -o nrtavs_operations_directory [-w web_host_dir_path] [-p web_host_http_path] [[-e error_email1] [-e error_email2] ...] [-h] [-H] [--help] Optional input arguments are identified by square brackets, []. All other input arguments are required by the program. More detailed information about the program and its input options are returned by executing the program with any one of the three equivalent help options, “-h”, “-H”, or “--help", for example: Only one input flag, “-o” is required by the program. This flag identifies the target directory where all processing of the data will occur, say 1. Create the directory The user can then edit the main deck or parameter specific decks in the directory 6 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 3.2 nrtavs_main.py Altimeter local_data_holding_directory local_data_processed_directory Data Files ntavs_main.py Start up Loop over each altimeter data file Compute statistics Generate plots Move altimeter file nrtavs_computestats and tables to processed area Found N Files Log from Statistics Map Plots statistics File File Tables Processed List files in computation (/statfiles) (/mapfiles) (/plotfiles) all N files in No holding area. (/computestats) holding area? local_statistics_directory No Files Yes Update website First wake up First wake up Wake up No after report hour No No Exit in UTC day? interval > 0 in UTC day? Yes Yes Yes Remove Statistics Generate hard Sleep until next file entries and Map copy report. wake up time. files outside of window. nrtavs_report.py Figure 1. Flow chart showing processing steps in nrtavs_main.py. Thick arrows indicate data flow, and gray boxes indicate data directories. Flow begins at upper left of Figure in the box indicating “Start up”. Not explicitly shown is the capability for nrtavs_report.py to send PDF versions of the report to the website. The purpose of the program nrtavs_main.py is to perform a statistical analysis of parameters within satellite altimeter data files, plot these statistics, present these plots and tables of statistics on web sites, and optionally execute the program nrtavs_report.py to generate hardcopy reports of the statistics. The nrtavs_main.py Python program is invoked from the command line. It requires the version of Python and associated Python modules that are provided with the NRTAVS installation package. The processing steps in nrtavs_main.py are illustrated in Figure 1. Each data processing cycle that is executed by nrtavs_main.py involves the following steps: 1. Program starts up, or wakes up after a user-defined interval (wakeup_interval_seconds). 2. Query a data holding directory for satellite altimeter data files to process. 3. For each satellite altimeter data file found in the holding area: a. Use the C program nrtavs_computestats to perform a statistical analysis of each altimetric parameter that is specified by the user. 7 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 b. Generate plots, maps, and tables of the statistics. c. Move the processed altimeter data file from the holding directory to the processed directory. 4. Generate html files to populate a web site with the plots, maps, and tables of statistics, and copy or ftp the html files, plots, maps, and tables to the physical location of the web site. 5. Once per day, clean up the database of statistics and parameter values, retaining only those within a user-defined window. 6. Once per day, optionally execute the program nrtavs_report.py to generate a hard copy report of the statistics that is optionally e-mailed to the distribution list specified by the list of error_email keywords in the main input deck, or posted to the a reports subdirectory of the web site specified by the web_host_http_path keyword in the main input deck. 7. Sleep until the next wake up time if wakeup_interval_seconds > 0, otherwise exit the program. A usage statement for the program is returned by executing the program without any input options, for example: Below is the usage statement for the program. USAGE nrtavs_main.py -i main_configuration_file [-w web_host_password] [-h] [-H] [--help] Optional input arguments are identified by square brackets, []. All other input arguments are required by the program. More detailed information about the program and its input options are returned by executing the program with any one of the three equivalent help options, “-h”, “-H”, or “--help", for example: Only one input flag, “-i” is required by the program. This flag identifies the main configuration file that contains all of the configuration information needed by the program to process the satellite altimeter data. The program is ideally executed in the directory where all processing of the satellite altimeter data will be performed, referred to here as The program can also be executed in the background as follows: cd The optional input argument –w can be used if the web host server is not the same computer on which the NRTAVS is being executed (web_host_machine keyword in main configuration file is not “LOCAL”). This option allows the user to provide the true password for the ftp connection to the web host server, and overrides the alternative means of providing the web host server password, which is by using the web_host_password in the main configuration file. Note the web_host password is an encrypted password that must be manually verified by the user on startup if the –w option is not used. NRTAVS version 2.5 allows extraction of the unencrypted password from the user’s GNOME keyring, by literally specifying: 8 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 ‘-w gnome-keyring’. This allows automated startup without having the cleartext password appear in process listings such as ‘ps –ef’. Instructions for setting up the GNOME keyring are given in Appendix A. The nrtavs_main.py program can be executed either as a process that completes a single data processing cycle and then exits, or as a sleeper process that runs continuously and completes one data processing cycle at some user-defined time interval. If the program is executed as a sleeper process then it should be executed in the background, as described above. The value of the parameter wakeup_interval_seconds in the input main configuration file (see Table 6) determines if the program is executed as a sleeper process or not. A value of 0 executes the program with a single data processing cycle, while any value greater than 0 executes the program as a sleeper process. The execution of the nrtavs_main.py process can be manipulated through four possible kill signals. Namely, given the process id of the running program, The three possible values of 3.3 nrtavs_report.py The purpose of the program nrtavs_report.py is to generate a hard copy PDF report of the statistics of all satellite altimeter data files within a user-defined window. The nrtavs_report.py Python program is invoked from the command line. It requires the version of Python and associated Python modules that are provided with the NRTAVS installation package. This program can also be automatically executed by the program nrtavs_main.py. The report generated is optionally distributed via e-mail to the list of addresses specified by the error_e_mail keyword in the main input deck, and/or to a subdirectory “reports” of the web site specified by the web_host_http_path keyword in the main input deck. If the website option is used and the web host server is remote, a password for the server is required. This password is provided to the program using the –w option, providing the password via standard input (stdin), or using the web_host_password keyword in the main input deck, in that order of priority. A usage statement for the program is returned by executing the program without any input options, for example: Below is the usage statement for the program. USAGE nrtavs_report.py -i main_configuration_file [-s report_window_start_time] [-e report_window_end_time] [-l log_file_name] [-w web_host_password] [--noclean] [-h] [-H] [--help] 9 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Optional input arguments are identified by square brackets, []. All other input arguments are required by the program. More detailed information about the program and its input options are returned by executing the program with any one of the three equivalent help options, “-h”, “-H”, or “--help", for example: Only one input flag, “-i” is required by the program. This flag identifies the main configuration file that contains all of the configuration information that the program needs to generate the report. This configuration file is usually identical to that used by nrtavs_main.py. This is because the program nrtavs_report.py requires output generated by the program nrtavs_main.py. Using the same main configuration file therefore ensures that identical data directory structure is specified for the two programs. The optional input argument -e specifies the UTC date of the end of the window of data to include in the report and is provided as a quoted string “YYYY-MM-DD HH:MM:SS”. If not specified, its value is set to the current time. The optional input argument -s specifies the UTC date of the start of the window of data to include in the report and is also provided as a quoted string “YYYY-MM-DD HH:MM:SS”. If not specified then its value is set to report_window_days before the value of the -e option (of its default value). The parameter report_window_days is required in the input main configuration file (see Table 6). The optional input argument –l specifies the name of the file where all log messages from the program will be sent. If not specified it is set to nrtavs_report_YYYYMMDD_HHMMSS.log where the date is derived from the value of the -e option (or its default value if not specified). Note, specifying “-l stdout” sends all log messages from the program to standard output. The optional input argument –w can be used if the web host server is not the same computer on which the NRTAVS is being executed (web_host_machine keyword in main configuration file is not “LOCAL”). This option allows the user to provide the true password for the ftp connection to the web host server, and overrides the alternative means of providing the web host server password, which is by providing the password through standard input (stdin) or using the web_host_password in the main configuration file. Note the web_host password is an encrypted password that must be manually verified by the user on startup if the –w option is not used. As with nrtavs_main.py, the optional argument ‘-w gnome-keyring’ allows the unencrypted password to be retrieved from the user’s GNOME keyring. The optional input flag --noclean when specified instructs the program to keep the work directory where the report was generated. The default removes the work directory after saving the PDF version of the generated report. Unlike nrtavs_main.py, the program nrtavs_report.py is configured to ignore each of the following kill signals: 1. INT (or 2) 2. USR1 (or 10) 3. USR2 (or 12) 10 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 When nrtavs_report.py is automatically executed by nrtavs_main.py it will set the end time of the data window to the 00:00:00 of the current day, and the start time of the window to report_window_days before that time. 3.4 nrtavs_computestats The purpose of the C program nrtavs_computestats is to perform a statistical analysis of individual parameters within satellite altimeter data files. Normally, this program is automatically executed by the nrtavs_main.py program. However, it can also be executed as a stand alone program. More details on the possible input options to this program are documented at the top of the program file The program performs the following functions: 1. Use the specific data product reader to read an individual satellite altimeter data file, and convert those data into the generic internal altimetric data structure used by nrtavs_computestats.c. 2. Perform a statistical analysis on a single user-defined parameter in the data file, given a list of user-defined flags for data editing. 3. Append the computed statistics for the specified parameter to a Statistics file. 4. Generate a Map file containing the values of the user-defined parameter and various flags for the specified parameter. The values in this file are used to generate maps of the parameter. Refer to Section 3.5 for more details on Statistics and Map file. The C program nrtavs_computestats utilizes the following C source and header files: − nrtavs_computestats.c − nrtavs_computestats.h − jason_ncgdr_reader.c (Modified version of that provided by NOAA) − jason_ncgdr_reader.h (Modified version of that provided by NOAA) − jason_igdrreader.c − jason_igdrreader.h − jason_osdrreader.c (Small modifications to version provided by EUMETSAT) − jason_osdrreader.h (Small modifications to version provided by EUMETSAT) − jason_bufrreader.c (Provided by EUMETSAT) − jason_bufrreader.h (Provided by EUMETSAT) − saral_ncgdr_reader.c − saral_ncgdr_reader.h The files jason_ncgdr_reader.c and jason_ncgdr_reader.h define the data readers for the version b and c NetCDF Jason-1 Interim and definitive Geophysical Data Records (IGDRs and GDRs), and the NetCDF Jason-2 Operational Geophysical Data Records (OGDRs), IGDRs, and GDRs. The reader for the version b Jason-1 NetCDF I/GDRs is the function named jason_ncgdrv1_reader, for the version c Jason-1 I/GDRs and the postlaunch Jason-2 O/I/GDRs are the functions named jason_ncgdrv21_reader and jason_ncgdrv23_reader. The Jason-2 data product reader, jason_ncgdrv21_reader, is based on version 2.1 of the Jason-2 User Products Specification [Dumont et al., 2008a] and has been tested with the first post-launch Jason-2 OGDRs. The Jason-2 data product reader, jason_ncgdrv23_reader, is based on version 2.3 of the Jason-2 User Products Specification [Dumont et al., 2008b] and has been tested with the sample post-launch Jason-2 OGDRs, and operational version “C” OGDRs and IGDRs. The three functions, jason_ncgdrv23_reader, jason_ncgdrv21_reader, and jason_ncgdrv1 are identical except for some changes to parameter names. 11 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 The files jason_igdrreader.c and jason_igdrreader.h define the data reader for the original versions of the Jason-1 Interim and definitive Geophysical Data Records (IGDRs and GDRs). The Jason-1 data reader is provided with the NRTAVS installation package because the original Jason-1 IGDRs were originally used to develop and test the system. The files jason_osdreader.c and jason_osdrreader.h define the data reader for the Jason-1 Operational Sensor Data Records (OSDRs). The files jason_bufrreader.c and jason_bufrreader.h define the data reader for the Jason-2 Operational Geophysical Data Records in BUFR format (OGDR-BUFR). The files saral_ncgdr_reader.c and saral_ncgdr_reader.h define the data reader for the SARAL/AltiKa OGDR and IGDR products. They have been tested on pre-launch samples of these two products. Data files with a different format require the addition of data readers for their particular format in to this C program. The data readers are required to translate the data into a generic internal data structure that is adopted by the main C program nrtavs_computestats.c. The jason_ncgdr_reader.c, jason_igdrreader.c, jason_osdrreader.c, and saral_ncgdr_reader.c functions serve as examples for how this translation should be performed. 3.5 NRTAVS Data Files There are two types of data files that are generated by nrtavs_computestats, and therefore by the execution of the higher-level program nrtavs_main.py, which executes nrtavs_computestats. They are referred to here as the Statistics and Map files. Both of these data files are required by the programs nrtavs_main.py and nrtavs_report.py to generate plots and tables of the statistics. Note that nrtavs_report.py does not execute the C program nrtavs_computestats and therefore does not generate the Statistics and Map files. As such, the location of those files, as generated by nrtavs_main.py, must be defined in the configuration file that is input to nrtavs_report.py. Separate Statistics and Map files are generated for each of the parameters that are processed. Each parameter-specific Statistics file contains a table of statistics from all of the altimeter data files that are processed. So a new entry is appended to the Statistics file as each altimeter data file is processed. In contrast, a separate parameter-specific Map file is generated for each altimeter data file that is processed. The main program nrtavs_main.py performs a daily clean up of the Statistics and Maps files from all parameters so that data from only a user-defined time window is kept online, while older data are removed. Note that SI units are always used in the Statistics and Map files. 3.5.1 Statistics File Format The Statistics file is a text file. Each line of the statistics file contains statistics for a particular altimeter data file. There are 24 entries in each line as shown in Table 2. Two sets of mean and standard deviation values are computed by nrtavs_computestats and provided on the Statistics file. The first set is computed from the available data after editing with the user-specified flags, while the second set is computed after an iterative 3 standard deviation (3-sigma) outlier detection and editing of the data already edited by the user-specified flags. Plots and tables provided on the web site and reports use the 3-sigma edited mean and standard deviation values. 12 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Entries 20-22 provide amplitude coefficients of a sinusoid that is fit to the data within a single altimeter data file. The period of this sinusoid is specified in the parameter specific configuration files (see keyword One_Cycle_Per_Rev_Period_Seconds in Table 7). This sinusoid can only be computed for the two sea surface height anomaly parameters (ssha and ssha_wetmod). While the keyword One_Cycle_Per_Rev_Period_Seconds can be assigned an arbitrary positive value, the sinusoid is intended to capture the signal at a period corresponding to one orbit revolution of the satellite, namely at a frequency of one cycle per revolution (1 cpr). The user should set the keyword accordingly (e.g. to 6746 seconds for Jason-1 and OSTM). The first column of the Statistics file defines an NRTAVS date string that corresponds to the start time and duration (in seconds) of the particular altimeter data file whose statistics are defined in that line of the file. This string has the format YYYYMMDDHHMMSS_SSSSS, where the first part of the string, YYYYMMDDHHMMSS, corresponds to the start date of the altimeter data file, and the second part of the string, SSSSS, corresponds to the duration of data (end time – start time) in the altimeter data file in seconds. This field is particularly important to the NRTAVS because the name of all other files generated by the NRTAVS, including the Map files, tables and plots, that correspond to a particular altimeter data file contain this string. As such, the NRTAVS is highly dependent on the availability of the parameter specific Statistics files. By definition the following equivalencies between entries in the Statistics file should always be true. 1. Total number of data points in mean (no 3 sigma edit) + Total number of data points with defaulted latitude or longitude + Total number of flagged values + Total number of unflagged defaulted values = Total number of data points in file So Entry 8 + Entry 13 + Entry 15 + Entry 17 = Entry 12 2. Total number of data points in mean (after 3 sigma edit) + Total number of data points with defaulted latitude or longitude + Total number of flagged values + Total number of unflagged defaulted values + Total number of data points exclued because of 3-sigma edit = Total number of data points in file So Entry 11 + Entry 13 + Entry 15 + Entry 17 + Entry 19 = Entry 12 13 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Table 2. Format of Statistics file. Entry Field Type 1 NRTAVS date field (YYYYMMDDHHMMSS_SSSSS) character 2 Name of altimeter data file character 3 Start time of data file (in J2000 seconds) double 4 End time of data file (in J2000 seconds) double 5 Median double 6 Mean (no 3 sigma edit) double 7 Standard Deviation (no 3-sigma edit) double 8 Total number of data points used in mean (no 3-sigma edit) long 9 Mean (after 3-sigma edit) double 10 Sigma (after 3-sigma edit) double 11 Total number of data points used in mean (after 3-sigma edit) long 12 Total number of data points in file long 13 Total number of data points with defaulted latitude or longitude long 14 Total number of data points over ocean long 15 Total number of flagged values in file long 16 Total number of flagged values over ocean long 17 Total number of unflagged defaulted values in file long 18 Total number of unflagged defaulted values over ocean long 19 Total number of data points excluded because of 3 sigma edit long 20 Bias term for 1 cpr sinusoid – Non-zero for ssha parameter only double 21 Cosine amplitude term for 1 cpr sinusoid – Non-zero for ssha parameter only double 22 Sine amplitude term for 1 cpr sinusoid – Non-zero for ssha parameter only double 23 Latency of altimeter data file in hours (Create time – First Measurement time) double 24 Absolute Revolution Number of first measurement time in file (from header of file)1 long 3.5.2 Map File Format The Map file is a binary file. Each Map file contains all data from a specific parameter in a particular altimeter data file. The name of the Map file indicates the parameter and the altimeter data file that it represents. For example the file ssha_20060707222029_03371_map.bin contains all of the data from the parameter “ssha” that was found in the altimeter data file that has an NRTAVS data string “20060707222029_03371” (entry 1 from Statistics file). Each record in the Map file corresponds to one record in the altimeter data file that is being processed. Each record in the Map file contains 9 entries as shown in Table 3. Outliers (as determined from 3-sigma editing) can be determined from entries 6, 7, and 8. In particular, the data point is an outlier (3-sigma edited) when Entry 8 = 1 and Entry 7 = 0 and Entry 6 = 0. Likewise, the data point has an unflagged default value when Entry 6 = 0 and Entry 7 = 1. The five flags are determined as follows: 1. The Land/Ocean flag is determined from the land/ocean flag in the altimeter data file. 2. The Flagged Data Point flag is set to 1 (flagged) if any one of the altimeter data file flags that the user specifies is encountered (see Table 8). 1 The revolution number shown in the report and website statistics table is offset to agree with NOAA’s pass numbering scheme (Sequence-Of-Events file) based on the input data filename. For example, if the filename contains “JA2” or “JASON2” the revolution number is offset by +177 to agree with the SOE. 14 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 3. The Default flag is set to 1 (defaulted) if the parameter value is found to have its default value. 4. The Edit flag is set to 1 (edited) if the parameter value is outside the 3 standard deviation window about the mean for that file. 5. The Pass Type flag is crudely defined from a difference between the latitude of neighboring data points. Table 3. Format of Map file. Entry Field Type 1 Time (in J2000 seconds) double 2 Longitude (degrees) double 3 Latitude (degrees) double 4 Parameter value scaled to SI units double 5 Land/Ocean Flag (1 = land, 0 = ocean) Signed char 6 Flagged Data Point Flag (1 = flagged, 0 = not flagged) Signed char 7 Default Flag (1 = defaulted, 0 = no defaulted) Signed char 8 Edit Flag (1 = edited because defaulted, flagged or outlier, 0 = not edited) Signed char 9 Pass Type Flag (0=unknown, 1 = ascending pass, 2 = descending pass) Signed char 3.6 Log Messsages All log messages from nrtavs_main.py and nrtavs_report.py are preceded by a time stamp with the format YYYY-MM-DD HH:MM:SS. Typically, messages are segregated into three categories: normal processing messages, WARNING messages, and ERROR messages. The distinction between WARNING and ERROR messages is that the program will terminate after an ERROR message, but continue after a WARNING message. In nrtavs_main.py the name of a log file is determined from two parameters specified in the input main configuration file, local_log_directory and log_filename_prefix (see Table 6). A separate log file is defined for each processing day and then named: local_log_directory/log_filename_prefix_YYYYMMDD.log. where YYYYMMDD is the date when the system most recently woke up. For example, if nrtavs_main.py is executed as a sleeper process, a new log file will be created for each day that the process is running. All messages are sent to standard output until the main configuration file has been successfully read and the name of the log file has been determined from the logfile keywords in the configuration file. From that point on, all log messages are sent to the specified log file. There is one exception to this rule. While the program is sleeping, the log file is closed and reset to standard output until it wakes up again. So if any of the kill signals are received while the program is sleeping then a log of that received signal will be sent to standard output. Any ERROR or WARNING log messages will also be sent to an e-mail distribution list that is specified in the main configuration file by the parameter error_e_mail. In nrtavs_report.py all log messages are sent to a log file, that is either determined from the input option – l, or determined by the program itself. If the –l option is not provided to nrtavs_report.py then the log file is named: log_filename_prefix_report_YYYYMMDD_HHMMSS.log where YYYYMMDD_HHMMSS is the date of the end time of the window of data that is being used to generate the report. The only e-mails generated by nrtavs_report.py are an e-mail indicating that an error was encountered while creating the PDF report, or an e-mail that contains the PDF report itself or a link to the report on website. These two e-mails are sent to the distribution list specified in the main configuration file by the parameter error_e_mail. 15 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 All of the three main programs, nrtavs_main.py, nrtavs_report.py, and nrtavs_computestats use the syslog standard to forward log messages to the syslog daemon or server of the computer on which the NRTAVS is being executed. Versions of NRTAVS prior to 2.5 used the syslog ‘facility’ LOG_LOCAL2. In version 2.5 support for dual-satellite logging (Jason-2 + Jason-3) is supported by setting environment variable ‘NRTAVS_SYSLOG_FACILITY’ to ‘LOCAL3’ so that a Jason-3 instance of NRTAVS will log to LOG_LOCAL3, while the Jason-2 instance will continue to log to the default LOG_LOCAL2. 3.7 Date, Time, Units, and Data Windows used by the NRTAVS Dates used by the NRTAVS are UTC dates. Time tags of altimeter data (in the Statistics file and the Map file) are in seconds past J2000 (Jan 1, 2000, 12:00:00). Altimeter parameter values are in SI units. Data windows are used throughout the NRTAVS. The NRTAVS treats each altimeter data file as a single entity with epoch defined by the end time of that data file. In practice, this means that if an NRTAVS data window starts at t1 and ends at t2, and an altimeter data file starts at P1 and ends at P2, then if t1 <= P2 <= t2 all of the data from that altimeter data file is considered to lie within the specified data window even if P1 < t1. Furthermore, when plotting statistics from a particular data file (e.g. mean, standard deviation), the statistic is assigned to the epoch P2. 3.8 Altimeter Data File Types 3.8.1 Currently Supported Data File Types The NRTAVS currently supports the altimeter data file types shown in Table 4. Note that the data file types shown in this table are required to be identical in the two programs nrtavs_main.py and nrtavs_computestats.c. The supported file types are defined in the nrtavs_data_file_types tuple in the nrtavs_library.py file, and in the initfiletypes function in nrtavs_computestats.c. A data file reader for each file type must be part of the C program nrtavs_computestats. The same reader is currently being used for the original format Jason-1 IGDRs and GDRs, since they have identical format. Table 4. Altimeter Data File Types Supported by the NRTAVS. File Type Index in Data File Type Description nrtavs_computestats 1 jason_nc_gdr Jason-2/OSTM NetCDF Operational, Interim and definitive Geophysical Data Records, based on version 2.3 of Jason-2 User Products Specification [Dumont et al., 2008b]. 2 jason_igdr Original Jason-1 Interim Geophysical Data Record 3 jason_gdr Original Jason-1 Geophysical Data Record 4 jason_osdr Jason-1 Operational Sensor Data Record 5 jason_nc_gdrv1 Version b of NetCDF Jason-1 Interim and definitive Geophysical Data Records. [Sicard and Dumont, 2008] 6 jason_nc_gdrv21 Jason-2/OSTM NetCDF Operational, Interim and definitive Geophysical Data Records, based on version 2.1 of Jason-2 User Products Specification [Dumont et al., 2008a]. Also for version c of Jason-1 NetCDF I/GDRs. 7 saral_nc_gdr SARAL/AltiKa NetCDF Operational, Interim, and definitive Geophysical Data Records, based upon pre-launch sample products. 8 jason_bufr Jason-2 OGDR in BUFR format 16 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 3.8.2 Modification of the NRTAVS to Support New Data File Types Executing the NRTAVS on altimeter data files that are not listed in Table 4 requires modification to the following programs. 1. nrtavs_computestats a. Change the value of NUM_FILE_TYPES in nrtavs_computestats.h. It is currently set to 8. b. Add a data file type index to nrtavs_computestats.h. This index definition is required in 1c below. c. Modify the initfiletypes function in nrtavs_computestats.c to add a new file type and appropriately set its index value. d. Modify nrtavs_computestats.c to call the appropriate data reader according to the data file type index. Currently the functions jason_ncgdrv23_reader, jason_igdrreader, jason_osdrreader, jason_ncgdrv1_reader, jason_ncgdrv21_reader, saral_ncgdr_reader, and jason_bufrreader are called when file type indices 1-8, respectively, are encountered. e. Write a new data file reader with a corresponding header file if necessary that reads the data files and converts the data into the generic internal structure that is expected by nrtavs_computestats.c. The jason_ncgdr_reader.c, jason_igdrreader.c, jason_osdrreader.c, and saral_ncgdr_reader.c functions serve as examples for this conversion. Note that jason_ncgdr_reader.c contains data readers for the three file types, jason_nc_gdr, jason_nc_gdrv1, and jason_nc_gdrv21. f. Modify the Makefile to add the new data file reader and corresponding header file. 2. nrtavs_library.py a. Modify the nrtavs_data_file_types tuple in nrtavs_library.py. This list should be identical to that used in 1c above. 3.9 Parameter Types 3.9.1 Currently Supported Parameter Types The NRTAVS currently supports the list of parameters shown in Table 5. The user defines which of these parameters should be evaluated by the NRTAVS. The user-defined list can range from any one to all of these parameters. The NRTAVS independently processes each of these parameters. More specifically, the program nrtavs_main.py executes the C program nrtavs_computestats once for each parameter in a single data file, and separate Statistics and Map files are generated for each parameter. Table 5. Parameter Types Supported by the NRTAVS. Parameter Parameter Type Description Type Index 1 range_ku Primary Channel altimeter range 2 ssha Sea Surface Height Anomaly (using radiometer wet troposphere delay) 3 ssha_wetmod Sea Surface Height Anomaly (using model wet troposphere delay) 4 wind_alt Altimeter Wind Speed 5 swh_ku Primary Channel Significant Wave Height 6 sigma0_ku Primary Channel Backscatter Coefficient 7 iono_ku Primary Channel Ionosphere Path Delay 8 att_ku Square of Attitude from Primary Channel Waveforms 9 tb_chan1 18.7 GHz Brightness Temperature 10 tb_chan2 23.8 GHz Brightness Temperature 11 tb_chan3 34.0 GHz Brightness Temperature 17 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 12 wet_rad Radiometer Wet Troposphere Path Delay 13 wet_mod Model Wet Troposphere Path Delay 14 dry Dry Troposphere Path Delay 15 ssb_ku Primary Channel Sea State Bias 16 wet_dif Wet Path Delay Difference (JMR-Model) 17 iono_ku_dif Primary Channel Ionosphere Path Delay Difference (Altimeter-Model or Altimeter-DORIS) 18 wind_dif Wind Speed Difference (Altimeter-JMR) 19 surf_type Altimeter Surface Type 20 rain_flag Rain Flag 21 ice_flag Ice Flag 22 alt_echo_type Altimeter Echo Type 23 alt_qual_flag Alimeter Quality Flag 24 alt_corr_flag Altimeter Instrument Correction Flag 25 alt_state_flag Altimeter State Flag 26 rad_surf_type Radiometer Surface Type 27 rad_qual_flag Radiometer Quality Flag 28 rad_state_flag Radiometer State Flag 29 rad_interp_flag Radiometer Brightness Temperature Interpolation Flag 30 geo_interp_flag Geophysical Parameter Interpolation Flag 31 orb_state_flag Orbit State Flag 32 Inv_bar_corr Inverse Barometer height correction 33 ssha_prod Sea Surface Height Anomaly directly from variable ‘ssha’ in product In the Table above, “Primary Channel” refers to the Ku-band channel for Jason-1 and Jason-2/OSTM, and the only altimeter channel (Ka) for SARAL/AltiKa. Note that parameters 19-31 are exactly the same as the flags described below in Table 8 that are used to edit the data. For data file types jason_nc_gdr, jason_nc_gdrv1, jason_nc_gdrv21, jason_igdr, jason_gdr, and jason_bufr all of these parameter types are available. For data type jason_osdr, the parameter types ssha, ssha_wetmod, wet_mod, dry, ssb_ku, wet_dif, iono_ku_dif, rain_flag, ice_flag, alt_corr_flag, rad_interp_flag, and geo_interp_flag are not available. The Ku band ionosphere path delay (iono_ku) is not explicitly provided on the Jason-1 OSDR but is computed by the jason_osdrreader function using the available total electron content. The radiometer wet path delay (wet_rad) is not explicitly provided on the Jason-1 OSDR but is computed by the jason_osdrreader function using the available radiometer vapor and liquid content. The flags associated with rain_flag, ice_flag, alt_corr_flag, rad_interp_flag, and geo_interp_flag are not available on the Jason-1 OSDR. For data file type saral_nc_gdr the parameters tb_chan3, iono_ku_dif, wind_dif, alt_state_flag, rad_state_flag and rad_interp_flag are not available. 18 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 3.9.2 Modification of the NRTAVS to Support New Parameter Types Executing the NRTAVS on parameters other than those listed in Table 5 requires modification to the following programs. 1. nrtavs_computestats a. Change the value of NUM_VAR_TYPES in nrtavs_computestats.h. It is currently set to 33. b. Add a new parameter type index to nrtavs_computestats.h. This index definition is required in 1c below. c. Modify the initvartypes function in nrtavs_computestats.c to add the new parameter type and appropriately set its index value. d. Edit the data file readers (e.g. jason_igdrreader.c, jason_ncgdr_reader.c, and jason_osdrreader.c) to appropriately extract the parameter from the altimeter data files. e. 2. nrtavs_library.py a. Modify the nrtavs_parameter_types tuple in nrtavs_library.py. This list should be identical to the list in 1c above. 19 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 4. Main Configuration File 4.1 Description The main configuration file, also referred to as the main deck, contains global configuration information relevant to all parameters. This includes a list of parameter-specific configuration files, or subdecks. Refer to Section 5 for more details about the parameter-specific configuration files. At least one subdeck must be specified in the main deck or the nrtavs_main.py and nrtavs_report.py programs will terminate execution. The decks are read once at program startup, and nrtavs_main.py will read the decks again only if it receives the signal USR2 sent by the command, kill –USR2 The comment character, ‘#”, may be used at any point in the main configuration file. When the comment character is used, the rest of the line after the comment character is ignored. All configuration information contained in the main configuration file is described using keyword/value pairs. Each line should contain a keyword and a corresponding value separated by a space or tab. All keywords are case insensitive and values are case sensitive. Sample main decks, (e.g. *_main_deck), can be found in the examples subdirectory of the directory where the NRTAVS is installed, Table 6 provides a complete list of the allowable keywords in the main configuration file, and also indicates if each keyword is required or optional. Note that all directory paths are either absolute paths, or relative to the directory in which the programs nrtavs_main.py or nrtavs_report.py are executed. Table 6. Keywords for Main Configuration File. Keyword Required/Optional Description local_data_holding_directory Required Specifies the directory that contains the altimeter data files to be processed. If this directory does not exist it is created. local_data_processed_directory Required Specifies the directory where the altimeter data files will be moved after they have been processed. If this directory does not exist it is created. local_log_directory Required Specifies the directory that will contain the log files for nrtavs_main.py. One log file is created per day, with the following naming convention: 20 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 log_filename_prefix Required Specifies the prefix of the name of the log file for nrtavs_main.py. If set to stdout then all logs from nrtavs_main.py are sent to standard output. Beginning with NRTAVS 2.5 this prefix is also used in the PDF report filename and logfile. statistics_window_days Required Defines the window (in days) of statistics and data that will be kept in the local_statistics_directory. The window is measured with respect to the epoch of the last data point on the last available data file. All data files with last data point outside of this window will be removed during once per day clean up. report_window_days Required Defines the window (in days) of statistics that will be incorporated in the report. See section 3.3 for more details about how this window is applied. If set to 0, nrtavs_main.py will not automatically execute nrtavs_report.py. daily_report_generate_hour Required Hour of the day at which nrtavs_main.py will automatically execute nrtavs_report.py to generate a report of statistics from the last report_window_days. report_attach_to_email Optional Default is OFF. If OFF report is not attached to e- mail, and e-mail indicates successful report generation. If ON report is attached to e-mail as soon as available. E-mail sent to addresses provided by error_e_mail keyword. report_websave_window_days Optional Default is 0. If 0 reports are not sent to website. If > 0 then reports from previous specified number of days are saved in a reports subdirectory of the path specified by web_host_http_path, e.g. web_host_http_path/reports. To save all reports at website set value to very large number. If > 0, a link to the reports web area is provided on the web pages. wakeup_interval_seconds Required Specifies the interval (in seconds) from when nrtavs_main.py wakes up to when it wakes up again. If the wake up interval is set to 0 then nrtavs_main.py will not sleep and wake up again. Instead it will terminate immediately after one data processing cycle. data_file_type Required Specifies the type of data file used as input. The value must be one of the values from Table 4. data_file_min_age_seconds Required Specifies minimum age of an input altimeter data file (in seconds) in order for it to be processed and included in the statistics calculations. Age is defined as current time – time stamp of the file. If the file is not old enough it is left in the holding directory until the time it becomes old enough to process. data_file_max_gap_seconds Required Specifies maximum gap between successive files processed in seconds. Gap is computed as the difference between the start time of the file being processed and the end time of the previously processed file. If the gap is exceeded an alert is sent to the e-mail distribution list. This parameter 21 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 is also used to identify file regressions. Regressions are computed as the difference between the start time of the file being processed and the start time of the previously processed file. If the regression is less than -data_file_max_gap_seconds an alert is sent to the e-mail distribution list. left_logo Required Specifies the path of the logo to place on the bottom-left corner of each plot. “None” may be used to specify no logo. Note, logos are not placed on plots in the hard copy report. right_logo Required Specifies the path of the logo to place on the bottom-right corner of each plot. “None” may be used to specify no logo. Note, logos are not placed on plots in the hard copy report. x_axis_plot_pad Required Specifies the pad to apply to the right side of the x axis of the statistics time series plots, and left and right sides of the x axis of the histogram plots, to ensure that the extreme data point does not lie exactly on the axis. This pad is specified as a percentage of the total x-axis range. web_host_machine Required Specifies the name of the computer to transfer plots, tables, html, and report files. This computer will act as the web host. If “LOCAL” is specified, the current computer is the web host and all plots, tables html, and reports are transferred using copy rather than ftp. web_host_login Required unless Specifies the login username to use for ftp login web_host_machine = for the transfer of the plots, tables, html, and LOCAL report files to the web site. web_host_password Required unless Specifies the password to use for ftp login for web_host_machine = transfer of the plots, tables, html, and report files LOCAL to the web site. This value is the encrypted version of the true password and the user will be prompted by nrtavs_main.py and nrtavs_report.py for the true password, which must then correspond to the encrypted password. If –w option is used in nrtavs_main.py and nrtavs_report.py to provide the true password then this keyword is ignored and the user is not prompted for the true password. Similarly, if the password is provided through standard input to nrtavs_report.py then this keyword is similarly ignored. web_host_dir_path Optional. If this Specifies the base directory on the web host to keyword is not transfer files to. Sub-directories will be created in specified statistics and this base directory for each parameter. plots will be generated but no plots, tables, or html pages will be posted to the web host. web_host_http_path Optional. If this Specifies the http path that corresponds to the keyword is not web host directory, web_host_dir_path. specified statistics and plots will be generated 22 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 but no plots, tables, or html pages will be posted to the web host. error_e_mail Optional Specifies recipients to receive e-mails regarding any errors or warnings that occur. This keyword may be used multiple times to generate a list of recipients. If this keyword is not used, the recipient list is empty and errors and warnings are output to the log file only. parameter_deck Required Specifies the name of a parameter-specific sub deck. This keyword must be used at least once in the main configuration file, namely at least one sub deck must be included. This keyword may be used multiple times to specify a list of parameter- specific sub decks. Note that the web sites and hard copy report display statistics from parameters in the same order that they are specified in the main configuration file. 4.2 Example A copy of the example main deck (main_deck) from the NRTAVS installation package is provided below. # # General configuration information # local_data_holding_directory holding/ local_data_processed_directory processed/ local_log_directory logfiles/ local_statistics_directory stats/ local_report_directory reports/ log_filename_prefix nrtavs statistics_window_days 20 report_window_days 10 daily_report_generate_hour 6 report_attach_to_email OFF report_websave_window_days 20 wakeup_interval_seconds 60 data_file_type jason_igdr data_file_min_age_seconds 360 data_file_max_gap_seconds 10 left_logo /logos/logo-noaa.png right_logo /logos/logo-jpl.png x_axis_plot_pad 0.02 web_host_machine LOCAL web_host_login None web_host_password None web_host_dir_path /web_directory/ web_host_http_path http://machine.noaa.gov/web_directory/ error_e_mail [email protected] error_e_mail [email protected] # # List of parameter-specific decks # 23 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 # Sea Surface Height Anomaly parameter_deck decks/ssha_input_deck # Ku-Band Significant Wave Height parameter_deck decks/swh_ku_input_deck # Altimeter Wind Speed parameter_deck decks/wind_alt_input_deck # Wind Speed Difference (Altimeter-JMR) parameter_deck decks/wind_dif_input_deck # Ku-Band Backscatter Coefficient parameter_deck decks/sigma0_ku_input_deck # Dry Troposphere Path Delay parameter_deck decks/dry_input_deck # Radiometer Wet Troposphere Path Delay parameter_deck decks/wet_rad_input_deck # Wet Path Delay Difference (JMR-Model) parameter_deck decks/wet_dif_input_deck # 18.7 GHz Brightness Temperature parameter_deck decks/tb_187_input_deck # 23.8 GHz Brightness Temperature parameter_deck decks/tb_238_input_deck # 34.0 GHz Brightness Temperature parameter_deck decks/tb_340_input_deck # Ku-Band Ionosphere Path Delay parameter_deck decks/iono_ku_input_deck # Ku-Band Ionosphere Path Delay Difference parameter_deck decks/iono_ku_dif_input_deck # Ku-Band Sea State Bias parameter_deck decks/ssb_ku_input_deck # Attitude from waveforms parameter_deck decks/att_ku_input_deck 24 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 5. Parameter Configuration File 5.1 Description Each parameter specific configuration file, sometimes referred to as a parameter subdeck, is associated with a particular parameter and contains all associated processing, plotting and tabulating information. Example parameter subdecks are also provided in the examples directory of the installation directory. The comment character, ‘#”, may be used at any point in the parameter-specific configuration file. When the comment character is used, the rest of the line after the comment character is ignored. All configuration information contained in the parameter-specific configuration file is described using keyword/value pairs. Each line should contain a keyword and a corresponding value separated by a space or tab. All keywords are case insensitive, and values are case sensitive. Tables 7, 8, and 9 provide a complete list of the keywords that can be specified in the parameter-specific configuration files, and also indicate if each keyword is required or optional. Table 7 provides list of the some general keywords in the parameter specific configuration file. Table 7. General Keywords for Parameter Configuration Files. Statistics Keyword Required/Optional Description Data_Evaluation_Parameter Required Specifies parameter name that will be evaluated by nrtavs_computestats executable. The value must be one of the values from Table 5. Local_Parameter_Name Optional Specifies a user specific alternative name for the parameter being evaluated. This name will be used to name the subdirectories, data files, plots, and tables corresponding to data for the associated Data_Evaluation_Parameter. This name will also be used in the web site drop down list for that parameter. If not specified then it will be set to the value of Data_Evaluation_Parameter. Variable Required Specifies variable long name. This name is used as the title of the parameter’s web site index web page and in the figure and table captions for the parameter’s plot and table in the hard copy report. Max_Num_Iter_3Sigma Optional Specifies the maximum number of iterations to use in the iterative 3-Sigma outlier editing when computing the statistics (e.g. mean and standard deviation) in nrtavs_computestats. The default value is 10 iterations. One_Cycle_Per_Rev_Period_Seconds Optional Specifies the period (in seconds) of the sinusoid that will be estimated by nrtavs_computestats. This sinusoid should typically have a period of one orbit revolution. This keyword is only used if Data_Evaluation_Parameter = ssha or ssha_wetmod and ignored for all other parameters. If set to 0 the sinusoid will not be estimated. The default value is 0. Max_Num_Iter_One_Cycle_Per_Rev Optional Specifies the maximum number of iterations to use for 3-Sigma outlier editing during the estimation of the one cycle per revolution 25 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 sinusoid. The default value is 10. This keyword is only used if Data_Evaluation_Parameter = ssha or ssha_wetmod and ignored otherwise. Table 8 provides a list of flags that can be specified in the parameter-specific configuration files to edit the data prior to the computation of statistics. The value of each of these flag keywords must be an 8 character string, denoted herein BBBBBBBB. Each digit of this string corresponds to the value of each of the 8 bits of the 1 byte integer flags that are used in the generic NRTAVS altimeter data structure. Bit 0 is the least significant (or rightmost) bit, and bit 7 is the most significant (or leftmost) bit. The user sets the value of each bit of that flag that must be satisfied in order for the corresponding data record in the altimeter data file to be included in the statistics computation. If the data record does not satisfy any one of those criteria, the Flagged Data Point flag in the Map file is set to 1 (see Section 3.5.2) and the record is excluded from the statistics computation. If B = 2, the data record’s flag bit is completely ignored and has no impact on data editing. If B = 0 or B = 1, then the data record’s corresponding flag bit must have the specified value of 0 or 1 in order for it to be included in the computation of the statistics. The generic data structure for the NRTAVS has defined these flags as described in Table 8 below. The descriptions provided in Table 8 use parameter names that are specifically used in: − the NetCDF header and the Jason-2 User Products specification, Sicard and Dumont [2008] for the data file type jason_nc_gdrv1, Dumont et al. [2008a] for jason_nc_gdrv21 ,and Dumont et al. [2008b] for jason_nc_gdr . − the Jason-1 I/GDR User Handbook [Picot et al., 2006] for data file types jason_gdr, jason_igdr, and jason_osdr. WARNING: The use of orbit_state_flag_diode from the Jason-2 NetCDF OGDRs to populate the Orbit_State_Flag in the NRTAVS has not been tested in version 2.0 of the NRTAVS because the OGDRs were not yet available. Users are advised to test this functionality when the OGDRs become available. As an example, nrtavs_computestats defines the surface type to be 0 for open oceans and semi-enclosed seas, 1 for enclosed seas or lakes, 2 for continental ice, and 3 for land. So to edit all data over continental ice and land, Surface_Type would be set to 22222202, which means that data included in the statistics must have bit 1 equal to 0. Table 8. Data Editing Flag Keywords in Parameter Configuration Files. Statistics Flag Keyword Required/Optional Description Surface_Type Optional Specifies surface type. The default value is 22222222. Data File Type: jason_nc_gdr, jason_nc_gdrv1, jason_nc_gdrv21, jason_gdr, jason_igdr, jason_osdr, jason_bufr and saral_nc_gdr. Set to value of surface_type. Possible integer values of this flag are 0 for open oceans and semi-enclosed seas, 1 for enclosed seas or lakes, 2 for continental ice, 3 for land. Rain_Flag Optional Specifies rain or no rain at the measurement location. The default value is 22222222. Data File Type: jason_nc_gdr, jason_nc_gdrv1, jason_nc_gdrv21, jason_gdr, jason_igdr, jason_bufr and 26 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 saral_nc_gdr. Set to value of rain_flag. Possible integer values of this flag are 1 for rain, and 0 for no rain. Data File Type: jason_osdr All 8 bits set are spares, and set to 0. Ice_Flag Optional Specifies ice or no ice at the measurement location. The default value is 22222222. Data File Type: jason_nc_gdr, jason_nc_gdrv1, jason_nc_gdrv21, jason_gdr, jason_igdr, jason_bufr and saral_ncgdr Set to value of ice_flag. Possible integer values of this flag are 1 for ice, and 0 for no ice. Data File Type: jason_osdr All 8 bits are spares, and set to 0. Altimeter_Echo_Type Optional Specifies the echo type detected by the altimeter. The default value is 22222222. Data File Type: jason_nc_gdr, jason_nc_gdrv1, jason_nc_gdrv21, jason_gdr, jason_igdr, jason_osdr, jason_bufr and saral_nc_gdr Set to value of alt_echo_type. Possible integer values are 0 for ocean-like and 1 for non ocean- like. Altimeter_Quality_Flag Optional Specifies the quality of the altimeter data. The default value is 22222222. Data File Type: jason_nc_gdr, jason_bufr and jason_nc_gdrv21 Bit 0: qual_alt_1hz_range_ku Bit 1: qual_alt_1hz_range_c Bit 2: qual_alt_1hz_swh_ku Bit 3: qual_alt_1hz_swh_c Bit 4: qual_alt_1hz_sig0_ku Bit 5: qual_alt_1hz_sig0_c Bit 6: qual_alt_1hz_off_nadir_angle_wf_ku Bit 7: set to 0 (qual_alt_1hz_off_nadir_angle_pf not meaningful, and not provided in version ‘d’) Data File Type: jason_nc_gdrv1 Bit 0: qual_1Hz_alt_Ku_range Bit 1: qual_1Hz_alt_C_range Bit 2: qual_1Hz_alt_Ku_SWH Bit 3: qual_1Hz_alt_C_SWH Bit 4: qual_1Hz_alt_Ku_Sig0 Bit 5: qual_1Hz_alt_C_Sig0 Bit 6: qual_1Hz_alt_off_nadir_angle_Ku Bit 7: qual_1Hz_alt_off_nadir_angle_ptf Data File Type: jason_gdr, jason_igdr and jason_osdr Set to value of qual_1hz_alt_data. Data File Type: saral_nc_gdr Bit 0: qual_alt_1hz_range Bit 1: Spare, set to 0. Bit 2: qual_alt_1hz_swh Bit 3: Spare, set to 0. Bit 4: qual_alt_1hz_sig0 Bit 5: Spare, set to 0. 27 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Bit 6: qual_alt_1hz_off_nadir_angle_wf Bit 7: qual_alt_1hz_off_nadir_angle_pf Altimeter_Instrument_Correction_Fl Optional Specifies the quality of the altimeter ag instrumental corrections. The default value is 22222222. Data File Type: jason_nc_gdr, jason_bufr and jason_nc_gdrv21 Bit 0: qual_inst_corr_1hz_range_ku Bit 1: qual_inst_corr_1hz_range_c Bit 2: qual_inst_corr_1hz_swh_ku Bit 3: qual_inst_corr_1hz_swh_c Bit 4: qual_inst_corr_1hz_sig0_ku Bit 5: qual_inst_corr_1hz_sig0_c Bit 6: Spare, set to 0. Bit 7: Spare, set to 0. Data File Type: jason_nc_gdrv1 Bit 0: qual_1Hz_inst_corr_Ku_range Bit 1: qual_1Hz_inst_corr_C_range Bit 2: qual_1Hz_inst_corr_Ku_SWH Bit 3: qual_1Hz_inst_corr_C_SWH Bit 4: qual_1Hz_inst_corr_Ku_Sig0 Bit 5: qual_1Hz_inst_corr_C_Sig0 Bit 6: Spare, set to 0. Bit 7: Spare, set to 0. Data_File Type: jason_gdr and jason_igdr Set to value of qual_1hz_alt_instr_corr. Data File Type: jason_osdr All 8 bits are spares, and set to 0. Data File Type: saral_nc_gdr Bit 0: qual_inst_corr_1hz_range Bit 1: Spare, set to 0. Bit 2: qual_inst_corr_1hz_swh Bit 3: Spare, set to 0. Bit 4: qual_inst_corr_1hz_sig0 Bit 5: Spare, set to 0. Bit 6: Spare, set to 0. Bit 7: Spare, set to 0. Altimeter_State_Flag Optional Specifies the state of the altimeter. The default value is 22222222. Data File Type: jason_nc_gdr, jason_bufr and jason_nc_gdrv21 Bit 0: Spare, set to 0 Bit 1: alt_state_flag_oper Bit 2: Spare, set to 0 Bit 3: Spare, set to 0 Bit 4: alt_state_flag_c_band Bit 5: alt_state_flag_band_seq Bit 6: alt_state_flag_ku_band_status Bit 7: alt_state_flag_c_band_status Data File Type: jason_nc_gdrv1 Bit 0: Spare, set to 0 Bit 1: alt_state_flag_oper Bit 2: Bit 0 of alt_state_flag_acquisition_mode_20Hz Bit 3: Bit 1 of alt_state_flag_acquisition_mode_20Hz 28 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Bit 4: alt_state_flag_C_band Bit 5: alt_state_flag_band_seq Bit 6: alt_state_flag_Ku_band_status Bit 7: alt_state_flag_C_band_status Data File Type: jason_gdr, jason_igdr, and jason_osdr Set to value of alt_state_flag. Data File Type: saral_nc_gdr All 8 bits are spares, and set to 0. Radiometer_Surface_Type Optional Specifies the radiometer surface type. The default value is 22222222. Data_File_Type: jason_nc_gdr, jason_nc_gdrv1, jason_nc_gdrv21, jason_gdr, jason_igdr, jason_osdr, jason_bufr and saral_nc_gdr. Set to value of rad_surf_type. Possible integer values of this flag are 0 for ocean, and 1 for land. Radiometer_Quality_Flag Optional Specifies the quality of the radiometer data. The default value is 22222222. Data File Type: jason_nc_gdr, jason_bufr and jason_nc_gdrv21 Bit 0: qual_rad_1hz_tb187 Bit 1: qual_rad_1hz_tb238 Bit 2: qual_rad_1hz_tb340 Bits 3-8, Spares, set to 0. Data File Type: jason_nc_gdrv1 Bit 0: qual_1Hz_rad_tb187 Bit 1: qual_1Hz_rad_tb238 Bit 2: qual_1Hz_rad_tb340 Bits 3-8, Spares, set to 0. Data File Type: jason_gdr, jason_igdr, and jason_osdr Set to value of qual_1hz_rad_data. Bits 0, 1, and 2 indicate the quality (1 = bad, 0= good) of three radiometer channels from lowest to highest frequency, respectively. Data File Type: saral_nc_gdr Bit 0: qual_rad_1hz_tb_k Bit 1: qual_rad_1hz_tb_ka Bits 2-8, Spares, set to 0. Radiometer_State_Flag Optional Specifies the state of the radiometer. The default value is 22222222. Data File Type: jason_nc_gdr, jason_bufr and jason_nc_gdrv21 Set to value of rad_state_flag_oper. Data File Type: jason_nc_gdrv1 and saral_nc_gdr. All 8 bits set are spares, and set to 0. Data File Type: jason_gdr, jason_igdr, and jason_osdr Set to value of rad_state_flag. Brightness_Temperature_Interpolatio Optional Specifies the radiometer brightness n_Flag temperatures interpolation. The default value is 22222222. Data File Type: jason_nc_gdr, jason_bufr 29 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 and jason_nc_gdrv21 Set to value of interp_flag_tb. Possible integer values of this flag are 0 for interpolation with no gap, 1 for interpolation with gap, 2 for extrapolation, and 3 for failure of interpolation and extrapolation. Data File Type: jason_nc_gdrv1, jason_gdr, and jason_igdr Set to value of tb_interp_flag. Possible integer values of this flag are 0 for interpolation with no gap, 1 for interpolation with gap, 2 for extrapolation, and 3 for failure of interpolation and extrapolation. Data File Type: jason_osdr and saral_nc_gdr. All 8 bits are spares, and set to 0. Geophysical_Parameter_Interpolatio Optional Specifies the geophysical parameter n_Flag interpolation. The default value is 22222222. Data File Type: jason_nc_gdr, jason_bufr and saral_nc_gdr. Bit 0: interp_flag_mean_sea_surface Bit 1: interp_flag_ocean_tide_sol1 Bit 2: interp_flag_ocean_tide_sol2 Bit 3: interp_flag_meteo Bit 4: Bit 0 of ecmwf_meteo_map_avail Bit 5 Bit 1 of ecmwf_meteo_map_avail Bit 6: interp_flag_mdt Bit 7: Spare, set to 0 Data File Type: jason_nc_gdrv21 and jason_nc_gdrv1. Bit 0: interp_flag_mss Bit 1: interp_flag_ocean_tide_sol1 Bit 2: interp_flag_ocean_tide_sol2 Bit 3: interp_flag_meteo Bit 4: Bit 0 of ecmwf_meteo_map_avail Bit 5 Bit 1 of ecmwf_meteo_map_avail Bit 6: interp_flag_mdt Bit 7: Spare, set to 0 Data File Type: jason_gdr and jason_igdr Set to value of interp_flag. Data File Type: jason_osdr All 8 bits are spares, and set to 0. Orbit_State_Flag Optional Specifies the orbit state. The default value is 22222222. Data File Type: jason_nc_gdr, jason_bufr, jason_nc_gdrv21, jason_nc_gdrv1, and saral_nc_gdr. If product_type is “OGDR”, set to value of orbit_state_flag_diode, otherwise set to value of orbit_state_flag_rest. Data File Type: jason_gdr, jason_igdr, and jason_osdr Set to value of orb_state_flag. 30 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Table 9 provides a list of the parameter-specific configuration file keywords and supported values that are used to generate plots and tables. Table 9. Plot and Table Generation Keywords in Parameter Configuration Files. Plot Keyword Required/Optional Description Mean_Plot_Switch Optional Specifies whether to generate the Mean/Median plot and transfer the plot to the web host or include it in the hard copy report. Possible values are ON or OFF. The default value is OFF. Mean_Plot_Title Required if Specifies the title of the Mean/Median plot. Mean_Plot_Switch = ON. Ignored if Mean_Plot_Switch = OFF Mean_Plot_X_Min Required if Specifies the X axis (time scale) minimum of Mean_Plot_Switch = the Mean/Median plot. ON. Ignored if Mean_Plot_Switch = OFF Mean_Plot_X_Max Required if Specifies the X axis (time scale) maximum of Mean_Plot_Switch = the Mean/Median plot. ON. Ignored if Mean_Plot_Switch = OFF Mean_Plot_X_Unit Required if Specifies the X axis (time scale) units of the Mean_Plot_Switch = Mean/Median plot. Possible values are Hours ON. Ignored if or Days. Mean_Plot_Switch = OFF Mean_Plot_Y_Min Required if Specifies the Y axis minimum of the Mean_Plot_Switch = Mean/Median plot. ON. Ignored if Mean_Plot_Switch = OFF Mean_Plot_Y_Max Required if Specifies the Y axis maximum of the Mean_Plot_Switch = Mean/Median plot. ON. Ignored if Mean_Plot_Switch = OFF Mean_Plot_Y_Title Required if Specifies the label for the Y axis of the Mean_Plot_Switch = Mean/Median Plot. ON. Ignored if Mean_Plot_Switch = OFF Mean_Plot_Buffer_Max Optional Specifies the maximum number of plots stored in the web page plot buffer for the Mean/Median plot. The default value is 10. Mean_Plot_X_Scale Optional Specifies type of scale to use for X axis of Mean/Median plot; possible values are linear or log. The default value is linear. Mean_Plot_Y_Scale Optional Specifies type of scale to use for Y axis of Mean/Median plot; possible values are linear or log. The default value is linear. 31 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Mean_Plot_Val_Max_Thresh Optional Specifies maximum threshold of mean value in plot, which when exceeded, for the file being processed, will generate an alert to the e-mail distribution list. Mean_Plot_Val_Min_Thresh Optional Specifies minimum threshold of mean value in plot, which when exceeded, for the file being processed, will generate an alert to the e-mail distribution list. Sigma_Plot_Switch Optional Specifies whether to generate the Standard Deviation plot and transfer the plot to the web host or include it in the hard copy report. Possible values are ON or OFF. The default value is OFF. Sigma_Plot_Title Required if Specifies the title of the Standard Deviation Sigma_Plot_Switch = plot. ON. Ignored if Sigma_Plot_Switch = OFF Sigma_Plot_X_Min Required if Specifies the X axis (time scale) minimum of Sigma_Plot_Switch = the Standard Deviation plot. ON. Ignored if Sigma_Plot_Switch = OFF Sigma_Plot_X_Max Required if Specifies the X axis (time scale) maximum of Sigma_Plot_Switch = the Standard Deviation plot. ON. Ignored if Sigma_Plot_Switch = OFF Sigma_Plot_X_Unit Required if Specifies the X axis (time scale) units of the Sigma_Plot_Switch = Standard Deviation plot. Possible values are ON. Ignored if Hours or Days. Sigma_Plot_Switch = OFF Sigma_Plot_Y_Min Required if Specifies the Y axis minimum of the Standard Sigma_Plot_Switch = Deviation plot. ON. Ignored if Sigma_Plot_Switch = OFF Sigma_Plot_Y_Max Required if Specifies the Y axis maximum of the Standard Sigma_Plot_Switch = Deviation plot. ON. Ignored if Sigma_Plot_Switch = OFF Sigma_Plot_Y_Title Required if Specifies the label for the Y axis of Standard Sigma_Plot_Switch = Deviation Plot. ON. Ignored if Sigma_Plot_Switch = OFF Sigma_Plot_Buffer_Max Optional Specifies the maximum number of plots stored in the web page plot buffer for Standard Deviation plot. The default value is 10. Sigma_Plot_X_Scale Optional Specifies type of scale to use for X axis of Standard Deviation plot; possible values are linear or log. The default value is linear 32 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Sigma_Plot_Y_Scale Optional Specifies type of scale to use for Y axis of Standard Deviation plot; possible values are linear or log. The default value is linear. Sigma_Plot_Val_Max_Thresh Optional Specifies maximum threshold of sigma value in plot, which when exceeded, for the file being processed, will generate an alert to the e-mail distribution list. Sigma_Plot_Val_Min_Thresh Optional Specifies minimum threshold of sigma value in plot, which when exceeded, for the file being processed, will generate an alert to the e-mail distribution list. Data_Points_Plot_Switch Optional Specifies whether to generate the Data Points plot and transfer the plot to the web host or include it in the hard copy reports. Possible values are ON or OFF. The default value is OFF. Data_Points_Plot_Title Required if Specifies the title of the Data Points plot. Data_Points_Plot_Swit ch = ON. Ignored if Data_Points_Plot_Swit ch = OFF Data_Points_Plot_X_Min Required if Specifies the X axis (time scale) minimum of Data_Points_Plot_Swit the Data Points plot. ch = ON. Ignored if Data_Points_Plot_Swit ch = OFF Data_Points_Plot_X_Max Required if Specifies the X axis (time scale) maximum of Data_Points_Plot_Swit the Data Points plot. ch = ON. Ignored if Data_Points_Plot_Swit ch = OFF Data_Points_Plot_X_Unit Required if Specifies the X axis (time scale) units of the Data_Points_Plot_Swit Data Points plot. Possible values are Hours or ch = ON. Ignored if Days. Data_Points_Plot_Swit ch = OFF Data_Points_Plot_Y_Min Required if Specifies the Y axis minimum of the Data Data_Points_Plot_Swit Points plot. ch = ON. Ignored if Data_Points_Plot_Swit ch = OFF Data_Points_Plot_Y_Max Required if Specifies the Y axis maximum of the Data Data_Points_Plot_Swit Points plot. ch = ON. Ignored if Data_Points_Plot_Swit ch = OFF Data_Points_Plot_Y_Title Required if Specifies the label for the Y axis of Data Data_Points_Plot_Swit Points Plot. ch = ON. Ignored if Data_Points_Plot_Swit ch = OFF Data_Points_Plot_Buffer_Max Optional Specifies the maximum number of plots stored in the web page plot buffer for Data Points plot. The default value is 10. Data_Points_Plot_X_Scale Optional Specifies type of scale to use for X axis of 33 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Data Points plot; possible values are linear or log. The default value is linear. Data_Points_Plot_Y_Scale Optional Specifies type of scale to use for Y axis of Data Points plot; possible values are linear or log. The default value is linear. Data_Points_Plot_Val_Max_Thresh Optional Specifies maximum threshold of flagged percentage value in plot, which when exceeded, for the file being processed, will generate an alert to the e-mail distribution list. Data_Points_Plot_Val_Min_Thresh Optional Specifies minimum threshold of flagged percentage value in plot, which when exceeded, for the file being processed, will generate an alert to the e-mail distribution list. Histogram_Plot_Switch Optional Specifies whether to generate the Histogram plot and transfer the plot to the web host or include it in the hard copy reports. Possible values are ON or OFF. The default value is OFF. Histogram_Plot_Title Required if Specifies the title of the Histogram plot. Histogram_Plot_Switch = ON. Ignored if Histogram_Plot_Switch = OFF Histogram_Plot_X_Min Required if Specifies the X axis minimum of the Histogram_Plot_Switch Histogram plot. Any data values lower than = ON. Ignored if this minimum add towards the histogram Histogram_Plot_Switch count at this minimum. = OFF Histogram_Plot_X_Max Required if Specifies the X axis maximum of the Histogram_Plot_Switch Histogram plot. Any data values higher than = ON. Ignored if this maximum add towards the histogram Histogram_Plot_Switch count at this maximum. = OFF Histogram_Plot_X_Title Required if Specifies the label for the X axis of Histogram Histogram_Plot_Switch Plot. = ON. Ignored if Histogram_Plot_Switch = OFF Histogram_Plot_Stepsize Required if Specifies the step size (discretization) interval Histogram_Plot_Switch of the histogram. = ON. Ignored if Histogram_Plot_Switch = OFF Histogram_Plot_Y_Min Required if Specifies the Y axis minimum of the Histogram_Plot_Switch Histogram plot. = ON. Ignored if Histogram_Plot_Switch = OFF Histogram_Plot_Y_Max Required if Specifies the Y axis maximum of the Histogram_Plot_Switch Histogram plot. = ON. Ignored if Histogram_Plot_Switch = OFF Histogram_Plot_Y_Title Required if Specifies the label for the Y axis of Histogram Histogram_Plot_Switch Plot. 34 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 = ON. Ignored if Histogram_Plot_Switch = OFF Histogram_Plot_Window Required if Specifies the time span (or window) of data to Histogram_Plot_Switch include in the Histogram plot. All points = ON. Ignored if within this window contribute to the Histogram_Plot_Switch histogram. For the web site plots, the same = OFF figure also shows a histogram of the data from the last file processed. Histogram_Plot_Window_Unit Required if Specifies the units for the time span (or Histogram_Plot_Switch window) of data to include in the Histogram = ON. Ignored if plot. Possible values are Hours or Days. Histogram_Plot_Switch = OFF Histogram_Plot_Buffer_Max Optional Specifies the maximum number of plots stored in the web page plot buffer for Histogram plot. The default value is 10. Histogram_Plot_X_Scale Optional Specifies type of scale to use for X axis of Histogram plot; possible values are linear or log. The default value is linear. Histogram_Plot_Y_Scale Optional Specifies type of scale to use for Y axis of Histogram plot; possible values are linear or log. The default value is linear. Histogram_Plot_Asc_Des Optional Possible values are ON and OFF. Default is OFF. If OFF a single histogram is generated using data from ascending and descending passes. If ON, a single figure with two histograms is generated with the first histogram using data from ascending passes and the second histogram from descending passes. Map_Plot_Switch Optional Specifies whether to generate the Map plot and transfer the plot to the web host or include it in the hard copy reports. Possible values are ON or OFF. The default value is OFF. Map_Plot_Title Required if Specifies the title of the Map plot. Map_Plot_Switch = ON. Ignored if Map_Plot_Switch = OFF Map_Plot_Lon_Min Required if Specifies the longitude axis minimum of the Map_Plot_Switch = Map plot. ON. Ignored if Map_Plot_Switch = OFF Map_Plot_Lon_Max Required if Specifies the longitude axis maximum of the Map_Plot_Switch = Map plot. ON. Ignored if Map_Plot_Switch = OFF Map_Plot_Lat_Min Required if Specifies the latitude axis minimum of the Map_Plot_Switch = Map Plot. ON. Ignored if Map_Plot_Switch = OFF 35 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Map_Plot_Lat_Max Required if Specifies the latitude axis maximum of the Map_Plot_Switch = Map Plot. ON. Ignored if Map_Plot_Switch = OFF Map_Plot_Color_Title Required if Specifies the title of the Map Plot color scale. Map_Plot_Switch = ON. Ignored if Map_Plot_Switch = OFF Map_Plot_Window Required if Specifies the time span (or window) of data to Map_Plot_Switch = include in the Map plot. All points within this ON. Ignored if window are plotted in the Map plot. Map_Plot_Switch = OFF Map_Plot_Window_Unit Required if Specifies the units for the time span (or Map_Plot_Switch = window) of data to include in the Map plot. ON. Ignored if Possible values are Hours or Days. Map_Plot_Switch = OFF Map_Plot_Color_Min Required if Specifies the minimum value used in the Map Map_Plot_Switch = plot color scale. ON. Ignored if Map_Plot_Switch = OFF Map_Plot_Color_Max Required if Specifies the maximum value used in the Map Map_Plot_Switch = plot color scale. ON. Ignored if Map_Plot_Switch = OFF Map_Plot_Buffer_Max Optional Specifies the maximum number of plots stored in the web page plot buffer for Map Plot. The default value is 10. Map_Plot_Color_Scale Optional Specifies type of scale to use for Map plot color scale; possible values are linear, reverse, or log. The default value is linear. The value reverse provides a color scale that is in the opposite order to that provided by the value linear. Map_Plot_Asc_Des Optional Possible values are ON and OFF. Default is OFF. If OFF a single map is generated using data from ascending and descending passes. If ON, a single figure with two maps is generated with the first map using data from ascending passes and the second map from descending passes. Edit_Map_Plot_Switch Optional Specifies whether to generate the Edit Map plot and transfer the plot to the web host or include it in the hard copy reports. Possible values are ON or OFF. The default value is OFF. Edit_Map_Plot_Title Required if Specifies the title of the Edit Map plot. Edit_Map_Plot_Switch = ON. Ignored if Edit_Map_Plot_Switch 36 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 = OFF Edit_Map_Plot_Lon_Min Required if Specifies the longitude axis minimum of the Edit_Map_Plot_Switch Edit Map plot. = ON. Ignored if Edit_Map_Plot_Switch = OFF Edit_Map_Plot_Lon_Max Required if Specifies the longitude axis maximum of the Edit_Map_Plot_Switch Edit Map plot. = ON. Ignored if Edit_Map_Plot_Switch = OFF Edit_Map_Plot_Lat_Min Required if Specifies the latitude axis minimum of the Edit_Map_Plot_Switch Edit Map Plot. = ON. Ignored if Edit_Map_Plot_Switch = OFF Edit_Map_Plot_Lat_Max Required if Specifies the latitude axis maximum of the Edit_Map_Plot_Switch Edit Map Plot. = ON. Ignored if Edit_Map_Plot_Switch = OFF Edit_Map_Plot_Window Required if Specifies the time span (or window) of data to Edit_Map_Plot_Switch include in the Edit Map plot. All points = ON. Ignored if within this window are plotted in the Edit Edit_Map_Plot_Switch Map plot. = OFF Edit_Map_Plot_Window_Unit Required if Specifies the units for the time span (or Edit_Map_Plot_Switch window) of data to include in the Edit Map = ON. Ignored if plot. Possible values are Hours or Days. Edit_Map_Plot_Switch = OFF Edit_Map_Plot_Buffer_Max Optional Specifies the maximum number of plots stored in the web page plot buffer for Edit Map Plot. The default value is 10. Edit_Map_Plot_Asc_Des Optional Possible values are ON and OFF. Default is OFF. If OFF a single map is generated using data from ascending and descending passes. If ON, a single figure with two maps is generated with the first map using data from ascending passes and the second map from descending passes. One_Cycle_Per_Rev_Plot_Switch Optional Specifies whether to generate the one cycle per revolution amplitude plot and transfer the plot to the web host or include it in the hard copy reports. Possible values are ON or OFF. This keyword is forced to OFF unless Data_Evaluation_Parameter = ssha. The default value is OFF. Onc_Cycle_Per_Rev_Plot_Title Required if Specifies the title of the one cycle per One_Cycle_Per_Rev revolution amplitude plot. _Plot_Switch = ON. Ignored if One_Cycle_Per_Rev _Plot_Switch = OFF One_Cycle_Per_Rev_Plot_X_Min Required if Specifies the X axis (time scale) minimum of 37 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 One_Cycle_Per_Rev the one cycle per revolution amplitude plot. _Plot_Switch = ON. Ignored if One_Cycle_Per_Rev _Plot_Switch = OFF One_Cycle_Per_Rev_Plot_X_Max Required if Specifies the X axis (time scale) maximum of One_Cycle_Per_Rev the one cycle per revolution amplitude plot. _Plot_Switch = ON. Ignored if One_Cycle_Per_Rev _Plot_Switch = OFF One_Cycle_Per_Rev_Plot_X_Unit Required if Specifies the X axis (time scale) units of the One_Cycle_Per_Rev one cycle per revolution amplitude plot. _Plot_Switch = ON. Possible values are Hours or Days. Ignored if One_Cycle_Per_Rev _Plot_Switch = OFF One_Cycle_Per_Rev_Plot_Y_Min Required if Specifies the Y axis minimum of the one One_Cycle_Per_Rev cycle per revolution amplitude plot. _Plot_Switch = ON. Ignored if One_Cycle_Per_Rev _Plot_Switch = OFF One_Cycle_Per_Rev_Plot_Y_Max Required if Specifies the Y axis maximum of the one One_Cycle_Per_Rev cycle per revolution amplitude plot. _Plot_Switch = ON. Ignored if One_Cycle_Per_Rev _Plot_Switch = OFF One_Cycle_Per_Rev_Plot_Y_Title Required if Specifies the label for the Y axis of one cycle One_Cycle_Per_Rev per revolution amplitude plot. _Plot_Switch = ON. Ignored if One_Cycle_Per_Rev_P lot_Switch = OFF One_Cycle_Per_Rev_Plot_Buffer_M Optional Specifies the maximum number of plots ax stored in the plot buffer for one cycle per revolution amplitude plot. The default value is 10. One_Cycle_Per_Rev_Plot_X_Scale Optional Specifies type of scale to use for X axis of one cycle per revolution amplitude Plot; possible values are linear or log. The default value is linear. One_Cycle_Per_Rev_Plot_Y_Scale Optional Specifies type of scale to use for Y axis of one cycle per revolution amplitude Plot; possible values are linear or log. The default value is linear. One_Cycle_Per_Rev_Plot_Val_Max Optional Specifies maximum threshold of one cycle per _Thresh revolution value in plot, which when exceeded, for the file being processed, will generate an alert to the e-mail distribution list. One_Cycle_Per_Rev_Plot_Val_Min Optional Specifies minimum threshold of one cycle per _Thresh revolution value in plot, which when exceeded, for the file being processed, will generate an alert to the e-mail distribution list. 38 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Statistics_Table_Switch Optional Specifies whether to generate the table of statistics on web page and transfer the table to the web host, or generate the table on the hard copy reports. Possible values are ON or OFF. The default value is OFF. Statistics_Table_Title Required if Specifies the title of the Statistics Table. Statistics_Table_Switch = ON. Ignored if Statistics_Table_Switch = OFF Statistics_Table_Window Required if Specifies the time span (or window) of data to Statistics_Table_Switch include in the Statistics Table. All points = ON. Ignored if within this window are included in the Statistics_Table_Switch Statistics Table. = OFF Statistics_Table_Window_Unit Required if Specifies the units for the time span (or Statistics_Table_Switch window) of data to include in the Statistics = ON. Ignored if Table. Possible values are Hours or Days. Statistics_Table_Switch = OFF Statistics_Table_Precision Required if Specifies the precision to use for table entries Statistics_Table_Switch in the Statistics table. = ON. Ignored if Statistics_Table_Switch = OFF Note: When any thresholds are exceeded, incuding data file gaps or regressions, the e-mail alerts occur as oe alert per file, with the e-mail describing all thresholds that were exceeded. 5.2 Example A copy of the example sea surface height anomaly deck (ssha_deck) from the NRTAVS installation package is provided below. Variable Sea Surface Height Anomaly Data_Evaluation_Parameter ssha # # Value of flag bit fields that must be passed to include measurement in the statistics # 2 indicates that the value of the bit will be ignored # Surface_Type 22222200 Rain_Flag 22222220 Ice_Flag 22222220 Altimeter_Echo_Type 22222220 Altimeter_Quality_Flag 00000000 Altimeter_Instrument_Correction_Flag 00000000 Altimeter_State_Flag 22222222 Radiometer_Surface_Type 22222220 Radiometer_Quality_Flag 22222000 Radiometer_State_Flag 22222222 Brightness_Temperature_Interpolation_Flag 22222222 Geophysical_Parameter_Interpolation_Flag 22222222 Orbit_State_Flag 00000011 # Variables for One Cycle Per Revolution (ocpr) One_Cycle_Per_Rev_Period_Seconds 6745.756 39 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 # Variables for Plot of Mean Mean_Plot_Switch ON Mean_Plot_Title Sea Surface Height Anomaly Mean_Plot_X_Min -3 Mean_Plot_X_Max 0 Mean_Plot_X_Unit Days Mean_Plot_Y_Min 0.0 Mean_Plot_Y_Max 0.3 Mean_Plot_Y_Title Mean/Median (m) Mean_Plot_Buffer_Max 25 Mean_Plot_Val_Max_Thresh 0.3 Mean_Plot_Val_Min_Thresh -0.3 # Variables for Plot of Standard Deviation Sigma_Plot_Switch ON Sigma_Plot_Title Sea Surface Height Anomaly Sigma_Plot_X_Min -3 Sigma_Plot_X_Max 0 Sigma_Plot_X_Unit Days Sigma_Plot_Y_Min 0 Sigma_Plot_Y_Max 0.2 Sigma_Plot_Y_Title Standard Deviation (m) Sigma_Plot_Buffer_Max 25 Sigma_Plot_Val_Max_Thresh 0.1 Sigma_Plot_Val_Min_Thresh 0.01 # Variables for Map Plot Map_Plot_Switch ON Map_Plot_Title Sea Surface Height Anomaly Map_Plot_Color_Scale linear Map_Plot_Color_Min -0.2 Map_Plot_Color_Max 0.3 Map_Plot_Color_Title meters Map_Plot_Window 3 Map_Plot_Window_Unit Days Map_Plot_Lon_Min 0 Map_Plot_Lon_Max 360 Map_Plot_Lat_Min -90 Map_Plot_Lat_Max 90 Map_Plot_Buffer_Max 25 # Variables for Edit Map Plot Edit_Map_Plot_Switch ON Edit_Map_Plot_Title Sea Surface Height Anomaly Edit_Map_Plot_Window 3 Edit_Map_Plot_Window_Unit Days Edit_Map_Plot_Lon_Min 0 Edit_Map_Plot_Lon_Max 360 Edit_Map_Plot_Lat_Min -90 Edit_Map_Plot_Lat_Max 90 Edit_Map_Plot_Buffer_Max 25 # Variables for Data Points Plot Data_Points_Plot_Switch ON Data_Points_Plot_Title Sea Surface Height Anomaly Data_Points_Plot_X_Min -3 Data_Points_Plot_X_Max 0 Data_Points_Plot_X_Unit Days Data_Points_Plot_Y_Min 0.1 Data_Points_Plot_Y_Max 100 40 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Data_Points_Plot_Y_Title Percentage of Over Ocean Data Edited Data_Points_Plot_Y_Scale log Data_Points_Plot_Buffer_Max 25 Data_Points_Plot_Val_Max_Thresh 90 Data_Points_Plot_Val_Min_Thresh 10 # Variables for Histogram Plot Histogram_Plot_Switch ON Histogram_Plot_Title Sea Surface Height Anomaly Histogram_Plot_X_Min -0.8 Histogram_Plot_X_Max 0.8 Histogram_Plot_X_Title Sea Surface Height Anomaly (m) Histogram_Plot_Stepsize 0.005 Histogram_Plot_Y_Min 0.01 Histogram_Plot_Y_Max 100.0 Histogram_Plot_Y_Title Percentage Occurence Histogram_Plot_Y_Scale log Histogram_Plot_Window 3 Histogram_Plot_Window_Unit Days Histogram_Plot_Buffer_Max 25 # Variables for One Cycle Per Revolution Plot One_Cycle_Per_Rev_Plot_Switch ON One_Cycle_Per_Rev_Plot_Title One Cycle Per Revolution Amplitude One_Cycle_Per_Rev_Plot_X_Min -3 One_Cycle_Per_Rev_Plot_X_Max 0 One_Cycle_Per_Rev_Plot_X_Unit Days One_Cycle_Per_Rev_Plot_Y_Min 0.0 One_Cycle_Per_Rev_Plot_Y_Max 0.50 One_Cycle_Per_Rev_Plot_Y_Title One Cycle Per Rev Amplitude (m) One_Cycle_Per_Rev_Plot_Buffer_Max 25 One_Cycle_Per_Rev_Plot_Val_Max_Thresh 0.1 One_Cycle_Per_Rev_Plot_Val_Min_Thresh 0.0 # Variables for Web Page Statistics Page Statistics_Table_Switch ON Statistics_Table_Window 3 Statistics_Table_Window_Unit Days Statistics_Table_Precision 0.001 Statistics_Table_Title Sea Surface Height Anomaly Statistics, Units (m) 5.3 Data Windows for Plots Data windows for plots displayed on the web site are treated in a different manner than those on the hard copy reports. For the web site plots, all data windows and the time scale on the linear plots are defined with respect to the last available data point from all processed altimeter data files. As such, the X_Min keywords in the parameter-specific keywords are typically a negative value, and the X_Max keywords are usually 0. The data window is then effectively X_Max – X_Min. In contrast, the start and end time of the data windows for all of the plots in the hard copy reports is explicitly defined by the input arguments -s and -e, and their associated default values. The time scale on the linear plots is then defined with respect to the start time of the data window. 41 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 6. Example Plots The example ssha parameter specific configuration file that is shown in Section 5.2 generates the plots shown in Figure 2-8 when applied to the Cycle 90 Jason-2 OGDRs. Note that the title of each plot contains the name of the individual file that was processed to create that plot. The Mean Plot, as shown in Figure 2, plots the mean of each input data file in read, and the median in blue. The last altimeter data file processed is indicated by the square symbol, and all other files are shown as circles. The symbols on the Sigma_Plot, Data_Points_Plot, and One_Cycle_Per_Rev_Plot plots are treated similarly. These different symbols are only used on web site plots, and not on the plots in the hard copy reports. Note that although Mean_Plot_X_Max was set to 0, the x-axis actually extends to +0.03days. This enables clear view of the last data point, and the axis extension is determined to be Mean_Plot_X_Max + (Mean_Plot_X_Max-Mean_Plot_X_Min)*x_axis_plot_pad, where x_axis_plot_pad is defined in the main configuration files (in this case x_axis_plot_pad was set to 0.01). The x-axis on the Sigma_Plot, Data_Points_Plot, and One_Cycle_Per_Rev_Plot plots is similarly extended. This extension is only performed on the web site plots, and not on the plots in the hard copy reports. In this histogram plot the left and rightmost x-axes are also similarly padded in the plots for the web site and the hard copy to ensure that the histogram at the extremes is viewable. Note that the two extremes of the histogram actually include a count of the number of data points less than or equal to than these extremes. The data points plot in Figure 4 provides an example of the use of a log scale on the y-axis. In this example only one file has one default value of sea surface height anomaly, which is then also apparent in the map of edited data shown in Figure 8. Figures 7 and 8 show the two types of map plots that are generated by the NRTAVS. Note that the ground track of the last altimeter data file process is indicated on these two plots by the solid black line. This line is only shown on the web site plots, and no on the plots in the hard copy reports. Figures 9, 10, and 11 show a similar hisogram and maps for the Ku-band ionosphere correction. These three examples show the capability to separate the historam and maps by ascending and descending passes using the Histogram_Plot_Asc_Dec, Map_Plot_Asc_Des, and Edit_Map_Plot_Asc_Des keywords in the parameter configuration files. 42 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Figure 2. Example of ssha Mean Plot. Figure 3. Example of ssha Sigma Plot. 43 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Figure 4. Example of ssha Data Points Plot (with log scale) Figure 5. Example of ssha One Cycle Per Revolution Amplitude Plot. 44 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Figure 6. Example of ssha Histogram Plot. Figure 7. Example of ssha Map Plot 45 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Figure 8. Example of ssha Edit Map Plot. 46 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Figure 9. Example of iono_ku Histogram Plot with Histogram_Plot_Asc_Des = ON. 47 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Figure 10. Example of iono_ku Map Plot with Map_Plot_Asc_Des = ON. 48 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 Figure 11. Example of iono_ku Edit Map Plot with Edit_Map_Plot_Asc_Des = ON. 49 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 7. References Dumont, J. P., O. Lauret, and P. Sicard, SALP Products Specification – Volume 1: Jason-2 User Products, SALP-ST-M-EA-15704-CN, Issue 2.1, 2008a. Dumont, J. P., O. Lauret, and P. Sicard, SALP Products Specification – Volume 1: Jason-2 User Products, SALP-ST-M-EA-15704-CN, Issue 2.3, 2008b. Picot, N., K. Case, S. Desai, and P. Vincent, 2006, AVISO and PODAAC User Handbook. IGDR and GDR Jason Products, SMM-MU-M5-OP-13184-CN (AVISO), JPL D-21352 (PODAAC), Edition 3.0. Sicard, P., and J. P. Dumont, SALP Products Specification – Volume 1: Jason-2 User Products, SALP-ST-M-EA-15704-CN, Issue 2.0, 2008. 50 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 APPENDIX A – Instructions for using GNOME keyring for NRTAVS password management Background The Jason-2/OSTM Near Real-Time Altimetry Validation System (NRTAVS) runs on the ESPC File Manager (FM) machines and transmits its web page output to the Jason-2 Web Server (WS) machines via ftp. Initially the user account AND password for this ftp transfer were stored in cleartext in the NRTAVS ‘main_deck’ configuration file on FM. To avoid the cleartext password we developed a workaround in which the ‘main_deck’ configuration file on FM contained the ENCRYPTED password for the ftp login (with user ‘nrtavs’) into the ftp server running on WS. When NRTAVS is started the ESPC operator must enter the cleartext password (which isn’t echoed to the screen) and NRTAVS checks that it yields the same encrypted password as found in the ‘main_deck’. If the encrypted passwords match, the cleartext password is correct an will ensure a successful ftp login when required by NRTAVS. During ESPC functional acceptance testing several issues arose with the NRTAVS startup procedure. Specifically, if NRTAVS was started in a separate terminal or xterm window it was impossible to background the job. In that case, if that window was inadvertently closed then the entire NRTAVS system would stop without warning. The following is an attempt to both keep the secure, encrypted password procedure described above, but to also avoid the need for an interactive login or other manual intervention. The GNOME keyring The GNOME Desktop environment which is standard on the FM machines’ Red Hat Enterprise Linux operating system includes a means of storing tokens in a password protected ‘keyring’. A public domain utility ‘gnome- keyring-query’, available from the GENTOO Linux project, allows for command line access to store & retrieve secrets from the keyring. This is ideal for our purposes, as it allows scripts to interact with the gnome-keyring for storing and retrieving the password needed by NRTAVS to ftp to WS. To add a secret (password in our case) to the keyring: echo -n “secret_password” | gnome-keyring-query set nrtavs The success can be confirmed by the converse, or with the keyring manager GUI: gnome-keyring-query get nrtavs Using the Python ‘commands’ module within NRTAVS (which is written in Python) it is possible to execute this ‘get nrtavs’ command to retrieve the cleartext password from the keyring. It remains in memory while NRTAVS is running so that the system can successfully ftp into the WS. Initial Installation The following one-time steps are required to implement the gnome-keyring in NRTAVS: 1. compile & install the gnome-keyring-query program: a. use Red Hat Network to install package: gnome-keyring-devel-0.4.0-1.2.EL4.x86_64.rpm b. tar -xvf gnome-keyring-query.tar c. cd gnome-keyring-query d. make e. sudo make install f. make clean 2. create a default keyring (disk based vs. the memory based ‘session’ keyring): a. launch the Applications > System Tools > Keyring Manager GUI from the RedHat menu bar b. type Control-O to open the Keyring Manager window 51 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 c. Click on the ‘New’ button d. name the new keyring ‘default’ e. enter the login password for the operator’s File Manager account as the Keyring password f. select this ‘default’ keyring and click on the ‘Set as Default’ button g. close both Keyring Manager windows 3. store the NRTAVS password: a. use script ‘nrtavs_password.sh” from the gnome-keyring-query directory to enter the password for user ‘nrtavs’ on the Web Server machine b. the script will hide (no echo) the user’s input during password entry c. remember - this is the password for account ‘nrtavs’ on WS, which is what NRTAVS uses to ftp it’s web pages to WS. This is NOT the password for the operator on FM. d. if desired, check that the password is correct with the Keyring GUI or more simply with ‘gnome- keyring-query get nrtavs’ 4. Update NRTAVS to use the gnome-keyring a. from inside the gnome-keyring-query directory: sudo cp nrtavs_main.py /home/ServiceWrapper/NRTAVS/SRC/local_directory/bin b. kill/stop previous instances of NRTAVS 5. Modify the ServiceWrapper scripts that launch NRTAVS: a. cd /usr/sbin b. sudo vi MakePrimary (etc). c. change ‘xterm -e nrtavs &” to ‘nrtavs &’ & exit vi Password updates Whenever a new password is required for account ‘nrtavs’ on WS, two steps must be taken to ensure that NRTAVS will properly utilize the new password: 1. Copy the encrypted password for user ‘nrtavs’ on WS into the main_deck configuration on FM: a. On WS, to view the encrypted password: sudo grep nrtavs /etc/shadow | cut -f2 -d: b. you’ll see something like: $1$k9enqI/J$I8YViPn1IHgpdMQ7EF.43/ c. On FM ‘vi /home/ServiceWrapper/NRTAVS/SRC/local_directory/decks/main_deck’ change the value of ‘web_host_password’ to the encrypted password in step 1b) above 2. Store the new cleartext password in the gnome-keyring by running ‘nrtavs_password.sh’. Routine Operations Whenever the master ‘MakePrimary’ or ‘MakeBackup’ scripts are executed, or nrtavs is launched directly via ‘/usr/sbin/nrtavs’, the operator will not be asked to enter the NRTAVS password for the Web Server. However (and this is desirable) if the gnome-keyring is locked the user WILL be prompted via a separate dialog box to enter the operator’s keyring password. To simplify operations the gnome-keyring password should be the same as the operator’s login password on File Manager. Once the operator unlocks the gnome-keyring with their login password, NRTAVS will retrieve the WS nrtavs password and continue executing in the background. If the NRTAVS password stored in the gnome-keyring does NOT match the encypted password in the NRTAVS ‘main_deck’ then NRTAVS will exit with an error, which will be logged in the composite syslogs by the ServiceWrapper module. The operator should confirm that NRTAVS is running successfully via ‘ps -ef | grep -I ntravs’. If it is not running, the operator should enter ‘/usr/sbin/nrtavs’ to try to restart NRTAVS. If this fails again the passwords in ‘main_deck’ and the gnome-keyring should be checked for consistency. 52 Near Real Time Altimeter Validation Version 2.5 System Handbook May 1, 2012 APPENDIX B – NRTAVS 2.4 Enhancements from EUMETSAT, including OGDR-BUFR 53 EUMETSAT Additions to NRTAVS EUMETSAT Doc.No. : EUM/OPS/TEN/11/3166 Eumetsat-Allee 1, D-64295 Darmstadt, Germany Tel: +49 6151 807-7 Issue : v1B Fax: +49 6151 807 555 Date : 8 May 2012 http://www.eumetsat.int WBS : © EUMETSAT The copyright of this document is the property of EUMETSAT. EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS Table of Contents 1 Introduction ...... 4 1.1 Purpose and Scope ...... 4 1.2 Reference Documents ...... 4 2 NRTAVS NetCDF READER ...... 5 2.1 Introduction ...... 5 2.2 Overview of Modifications to Current NRTAVS Codebase ...... 5 2.3 Installation of NRTAVS ...... 6 3 NRTAVS BUFR READER ...... 7 3.1 Introduction ...... 7 3.2 Overview of Modifications to Current NRTAVS Codebase ...... 7 3.3 Installation of NRTAVS BUFR Reader ...... 7 3.4 Differences in BUFR vs netCDF reports ...... 8 3.4.1 Flattening of median values curve ...... 9 3.4.2 Additional default values in the BUFR monitoring ...... 9 3.5 Troubleshooting ...... 10 3.6 Updating jason_buffreader for BUFR format changes ...... 10 3.7 Adding new monitoring parameters to jason_bufrreader ...... 10 Page 3 of 11 EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS 1 INTRODUCTION 1.1 Purpose and Scope The purpose of this document is to document the additional functionality and parameter definitions added to the Near Real Time Altimeter Validation System (NRTAVS) software by the team at EUMETSAT. 1.2 Reference Documents [Ref1] WMO-No. 306, Manual on Code - International Codes Volume I.2, 2010 edition [Ref2] Near Real Time Altimeter Validation System User Handbook, JPL D-35859, Version 2.4, Dec 16 2010 [Ref3] TDCF tables extracted from the Manual on Codes, Volume I.2, FM 94 BUFR edition 4, Version 16, 4 May 2011 Page 4 of 11 EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS 2 NRTAVS NETCDF READER 2.1 Introduction This section details the changes to NRTAVS to allow reading and processing of new parameters in JASON NetCDF products. 2.2 Overview of Modifications to Current NRTAVS Codebase The codebase used was NRTAVS release 0.2.4 The code that has been added or modified is located in the following directories: • $NRTAVS_ROOT/modules/src/nrtavs_computestats-0.2.4.2/ for the C code, • $NRTAVS_ROOT/modules/src/nrtavs_library-0.2.4.2/ for the Python code, • $NRTAVS_ROOT/modules/src/nrtavs_examples-0.2.4.2/ for the deck files, • $NRTAVS_ROOT/modules/src/nrtavs_main-0.2.4.2/ for the quick setup, where $NRTAVS_ROOT is the location of the nrtavs-0.2.4.2 code. Routine Change Location Description jason_ncgdr_reader.c Modified nrtavs_computestats New code to read the parameters inv_bar_corr and ssha_prod nrtavs_computestats.c Modified nrtavs_computestats New code to process the parameters inv_bar_corr and ssha_prod nrtavs_computestats.h Modified nrtavs_computestats New code to declare the parameters inv_bar_corr and ssha_prod nrtavs_library.py Modified nrtavs_library Add inv_bar_corr and ssha_prod parameter types att_ku_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation dry_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation inv_bar_corr_deck Added nrtavs_examples Inverted Barometer Height Correction, new parameter iono_ku_dif_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation iono_ku_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation main_deck Modified nrtavs_examples Visualisation and/or threshold adaptation sigma0_ku_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation Page 5 of 11 EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS Routine Change Location Description ssb_ku_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation ssha_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation ssha_prod_input_deck Added nrtavs_examples Sea Surface Height Anomaly (Product Values), new parameter ssha_wetmod_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation swh_ku_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation tb_187_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation tb_238_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation tb_340_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation wet_dif_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation wet_mod_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation wet_rad_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation wind_alt_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation wind_dif_input_deck Modified nrtavs_examples Visualisation and/or threshold adaptation nrtavs_quicksetup.py Modified nrtavs_main In order to correctly generate the main_deck 2.3 Installation of NRTAVS See next section, as it is mandatory to install the BUFR library with the delivered package, as well as it is mandatory too to install the NetCDF library for the previous versions. Page 6 of 11 EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS 3 NRTAVS BUFR READER 3.1 Introduction This section details the changes and additions to NRTAVS to allow reading and processing of JASON BUFR products. 3.2 Overview of Modifications to Current NRTAVS Codebase The codebase used was NRTAVS release 0.2.4 The code that has been added or modified is located in the following directories: • $NRTAVS_ROOT/modules/src/nrtavs_computestats-0.2.4.2/ for the C code, • $NRTAVS_ROOT/modules/src/nrtavs_library-0.2.4.2/ for the Python code, where $NRTAVS_ROOT is the location of the nrtavs-0.2.4.2 code. Routine Change Location Description jason_bufrreader.c New code nrtavs_computestats New code to read a Jason 2 BUFR product jason_bufrreader.h New code nrtavs_computestats Include file for Jason_bufrreader.h nrtavs_computestats.c Modified nrtavs_computestats Add call to jason_bufrreader nrtavs_computestats.h Modified nrtavs_computestats Add parameters for BUFR products Makefile Modified nrtavs_computestats Add Jason_bufrreader compilation nrtavs_library.py Modified nrtavs_library Add jason_bufr data file type The code has been tested on our 32-bit SUSE Linux development machine and on the operational 64-bit Red Hat Linux machine. The BUFR library should be compiled with the correct 32-bit or 64-bit compilation option select to match the architecture of the target machine and the NRTAVS compilation will pick up the correct BUFR library. 3.3 Installation of NRTAVS BUFR Reader 1. If they are not installed on your computer, install the GNU compilers for C and Fortran from http://gcc.gnu.org/. These will be needed to compile the NRTAVS and BUFR libraries. Please ensure that the C and Fortran compilers are the same release to avoid any problems due to mis-matched shared library versions. 2. Download and install the ECMWF BUFR encoding/decoding software from: http://www.ecmwf.int/products/data/software/bufr.html following the instructions in the provided documentation. When installing on the target machine, you will need to select the “compile as 64-bit” option when prompted. Install the libbufrR64.a BUFR Page 7 of 11 EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS library to your chosen BUFR_INSTALL directory (either the default /usr/local/lib or your chosen directory e.g. /home/myname/bufr_000387/) 3. Unpack the NRTAVS package to your chosen NRTAVS_UNPACK directory. 4. If you have not installed the BUFR library in the default location at /usr/local/lib, then the nrtavs_computestats Makefile will locate the BUFR library using the $BUFRHOME environment variable which should be set to point to your BUFR_INSTALL directory e.g. $ export BUFRHOME = /home/myname/bufr_000387/ 5. Compile and install the tool to your chosen $NRTAVS_INSTALL directory following the instructions in the NRTAVS user guide. The software should be compiled using the GNU compilers installed in Step 1. If running on a 64-bit machine, the Makefile will use the 64-bit BUFR library (libbufrR64.a). On a 32-bit machine, it will use the 32-bit version (libbufr.a). 6. Environment variable $BUFR_TABLES should be set to the directory where the BUFR tables are located (remembering the final slash) e.g. $ export BUFR_TABLES=/home/myname/bufr_000387/bufr_tables/ 7. If you wish to suppress diagnostic output from the BUFR decoding routines, environment variable $PRINT_TABLE_NAMES should be set to false e.g. $ export PRINT_TABLE_NAMES=false 8. Set the stack memory to unlimited. If using csh/tcsh you can set the see the stack memory settings using $ limit And set the stack to unlimited using $ unlimit If using bash you can set the see the stack memory settings using $ limit -s And set the stack to unlimited using $ limit –sS unlimited 9. Navigate to the $NRTAVS_WORKING/decks directory and edit the main_deck file to set the data_file_type to jason_bufr and verify that the web configuration parameters are set correctly. 10. You should now be ready to run NRTAVS using the Jason BUFR files 3.4 Differences in BUFR vs netCDF reports When comparing the NRTAVS reports produced by analysing the BUFR to those produced by analysing the netCDF products, the reports are on the whole comparable, although a number of small discrepancies have been noted. These seem to be a result of differences between the BUFR and netCDF products rather than due to NRTAVS software. Page 8 of 11 EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS 3.4.1 Flattening of median values curve The Ku-Band waveform attitude plots show markedly different median values between BUFR and NETCDF. The BUFR values are stored to a lower precision and because the bulk of the values tend towards the lower end of the range, the median curve has been smoothed out. 3.4.2 Additional default values in the BUFR monitoring A number of the BUFR reports have more default values indicated than their netCDF counterparts. This is where the value has been set to a default (or flag) value rather than a meaningful physical value. There are two reasons for this. For the altimeter wind speed, the netCDF files have cases where the values are negative (ranging from roughly -0.25 to -0.01). The BUFR format does not cater for negative wind speeds and the BUFR encoder therefore sets these to its default (missing data) value in these cases, resulting in the higher number of default values for all monitored parameters that use the altimeter wind speed. The other reason for the larger number of default values appears to be when the BUFR field does not have sufficient range to encode the netCDF values. This is shown by the Radiometer Wet Tropospheric Path Delay (wet_rad) which is encoded as BUFR 0 25 164. This field has a scale of 4 and a reference value of -5000 and a width of 13 bits. This means values less than - 0.5 cannot be encoded and are set to default in the BUFR product. The netCDF does have a number of values below -0.5, resulting in the higher number of default values for all monitored parameters that use the wet_rad parameter. 3.5 Software License for jason_bufrreader code The jason_bufrreader.c and jason_bufrreader.h code have been released under the EUMETSAT freeware license as follows: Freeware Licence Disclaimer ‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐‐ The user acknowledges and shall at all times respect EUMETSAT's intellectual property rights in the software. EUMETSAT shall at all times retain such intellectual property rights in the software and in all copies thereof regardless of form. The user agrees when using the software in any recognisable form to name EUMETSAT as the source by including "(c) (year) EUMETSAT". The user of the software may provide feedback, report problems and suggest enhancements to the software to EUMETSAT. In addition, the user shall grant to EUMETSAT unrestricted use of this information. Page 9 of 11 EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS Neither EUMETSAT nor its Member States are liable for the usefulness or proper functioning of software, nor do they accept any liability for any consequences, whether direct or indirect, of any use of software or for any results related to the use of software or for any right or claims by third parties related to all or any part of software or its use. Where the source code is made available to users this is done without any warranty, and EUMETSAT will not provide any support for its use and customisation. 3.6 Troubleshooting Problem Possible Solution Product is processed very quickly and no nrtavs_computestats may be crashing out due files are produced to stack size set too low. Set stack size higher (see installation instructions). BUFR Tables cannot be read Check you have defined the $BUFR_TABLES environment variable with the path to the directory where the BUFR tables are stored. Make sure the path is terminated with a directory separator (‘/’) BUFR routines producing unwanted output Turn off the additional BUFR information by to screen setting $PRINT_TABLE_NAMES to false. (see installation instructions). 3.7 Updating jason_buffreader for BUFR format changes The jason_buffreader code is designed, as far as possible, to be independent of the BUFR format used to store the Jason data, so the reader is immune to the BUFR descriptors being re-ordered or additional descriptors inserted into the format rather than appended at the end. It achieves this by locating the required data using the BUFR data descriptor FXY value rather than locating data by using fixed offsets into the BUFR dataset. This is done through the lookup_element_index function which is passed the FXY number code of the required data element and returns the offset of the data value within the expanded BUFR values array, thus avoiding any hard-coded offset values. For the cases where FXY values are used repeatedly, i.e. channel numbers and brightness temperatures, the function lookup_element_indices is used instead and an array of index values is returned, one for each occurrence of the FXY code. In this case, the program does need to know the ordering of the data, but as this is usually ascending channel number, it is relatively easy to select the correct data value. 3.8 Adding new monitoring parameters to jason_bufrreader Page 10 of 11 EUM/OPS/TEN/11/3166 v1B, 08 May 2012 EUMETSAT Additions to NRTAVS Additional parameters can be added by following the coding template for existing monitored parameters. Where the parameter to be monitored is constructed from the values of a number of fields within the BUFR product, then care should be taken to assure that if any field returns a default value then the returned parameter value is set to the default flag value. Page 11 of 11