Mcidas J MANAGEMENT and DISPLAY of FOUR-DIMENSIONAL ENVIRONMENTAL DATA SETS USING Mcidas
Total Page:16
File Type:pdf, Size:1020Kb
MANAGEMENT AND DISPLAY OF FOUR-DIMENSIONAL ENVIRONMENTAL DATA SETS USING McIDAS J MANAGEMENT AND DISPLAY OF FOUR-DIMENSIONAL ENVIRONMENTAL DATA SETS USING McIDAS A Final Report to National Aeronautics and Space Administration (NASA) prepared for George C. Marshall Space Flight Center (MSFC) Marshall Space Flight Center Alabama 35812 for Management and Display of Four-Dimensional ' Environmental Data Sets Using blcIDAS Contract _NAS8-36292 University of Wisconsin Account #144X669 for the period of 2/28/86 thru 8/31/90 submitted by William L. Hibbard David Santek Verner E. Suomi Space Science and Engineering Center at the University of Wisconsin - Madison 1225 West Dayton Street Madison, Wisconsin 53706 (608) 262-0544 November 1990 v I. INTRODUCTION Over the past four years, great strides have been made in the areas of data management and display of 4-D meteorological data sets. When the current contract went into effect in early 1986, the displays consisted of wind trajectories and moisture mesh surfaces from upper air data over a topographical map. Now the software can produce multi-parameter displays of transparent surfaces, trajectories as streamers, density surfaces, 2-D contours and text, in stereo or mono (with color). Also, we have made the move from single image processing on the mainframe computer to real-time interaction on the Stardent computer. This report summarizes the work accomplished under this contract with details contained in the appendices. II. WORK SUMMARY The Statement of Work for this contract read as follows: TASK 1: Inventory of 4-dimensional data setr. Survey of available and planned 4-D meteorological data sources. Evaluate the data types for their impact on the data management and display system. TASK 2: Data management. Analyze the requirements for data base management generated by the 4-D data display system. Evaluate the suitability of the existing data base management procedures and file structures in light of the new requirements. Where needed, design and implement new data base management tools and file structures. TASK 3: Data integration. Assure the quality of the basic 4-D data sets. Investigate interpolation and extrapolation techniques for the 4-D data. Combine 4-D data from various sources to make a uniform and consistent data set for display purposes. TASK 4: Design data display softv_are to create abstract line graphic 3-D displays. Create realistic shaded 3-D displays. Develop animation routines for these displays in order to produce a dynamic 4-D presentation. TASK 5: Workstation design. Implement a prototype dynamic color stereo workstation. Produce a complete functional design specification based on interactive studies and user feedback. TASK 1: 4-D INVENTORY Inventory of 4-D data sources Over the course of the contract, many scientists and research institutions were queried about their use of or availability of 4-dimensional data sets. Primarily, the search was limited to meteorological data, though data from other earth sciences have been included. The effort relied on catalogs, journal articles, and referrals to track down the source of a particular data type. A questionnaire was designed to provide a concise format for the inventory: whom to contact, what kind of data is available, how to acquire (price, format), and any reference or catalog that would be useful. Naturally, even the most exhausted searches would not be complete, but also we ! excluded individuals that were not willing to provide their data to outside users. Appendix A contains the inventory for the 4-D data sets. Data sourqes u_ed Over the past three years, the 3-D software has been applied to data from 21 different sources. The development of a general 3-D grid structure and display program made this task manageable. Usually, the only new software that needed to be written, were programs to put the data in the MclDAS (such as a tape read program). For imagery, though, it was handled on an individual basis, since most of the effort was in the area of model output. The data types fall into three general categories: imagery, observations, and models. Imagery GOES images McGill volumetric radar NSSL Doppler radar NASA/MSFC volumetric radar LIDAR CT scan medical data Observations Rawinsonde balloon soundings VAS satellite soundings Wind Cave data Global topography Global soil and vegetation type HIS aircraft soundings Models NSSL kinematic microphysical cloud model NASA/MSFC LAMPS model Robert Schlesinger's 3-D cloud model UW/CIMSS 4-D data assimilation model NASA/GSFC GMASS model NMC global model RAMS model Warzyn Engineering hydrology model More information on these data sets can be found in Appendix A and in Hibbard (1989b). TASK 2: DATA MANAGEMENT Requirements The management of 4-D data sets requires more than one data structure for efficient access and organization. The different file structures are divided into categories similar to the way the 21 data sources were divided (under TASK I): V Images, unevenly spaced data (observations and trajectories), and regularly spaced data (2-D and 3-D grids from model output). Image data and 2-D grids are similar in structure (2-D matrices), with the differences being the source and volume of data, and the display. Remote sensors (satellite, radar, lidar) produce massive amounts of data spatially, temporally, and spectrally. Regularly spaced data (grids) are usually produced from numerical models. The output consists of many physical parameters at a sequence of time steps in the form of 2-D and 3-D grids. These data sets may also be large (as is imagery), but the access pattern requires a different structure. Observed data (rawinsonde and satellite soundings) and trajectories do not fit into a matrix format. No assumptions are made about the space or time distribution of the data. Evaluation of existing data manaRemenr The MclDAS provides three data structures for the storing of weather data: Image, Grid and MD (Meteorological Data) files. Images are stored as 2-D arrays as 1, 2, or 4 byte quantities per pixel. For multi- spectral data, interleaving of the pixels is done, thus adding a third dimension. Images of different times are kept in individual files. A separate file contains a directory entry for all images on the system so that all users have read access to all images on the system, but not necessarily write access. Since the naming convention is numeric, a time sequence of images would normally be stored in numeric order. This structure is very general, and is being used for the radar, lidar, and the resulting rendered 3-D graphics. The grid file structure provides for only 2-D grids. There are limitations in size, 10240 maximum number of grid points per grid, and 159 grids per grid file. Also, the data values at the grid points are stored as scaled integers. Since the naming of data sets on MclDAS is currently numeric, large numbers of 2-D grids are stored in successive grid files. The MD file structure is a key driven system for randomly spaced data. The layout is a matrix, with each cell containing the data at a particular time and location. It is usually organized so that each row corresponds to a time, and each column a location. This works well for surface hourly reports and upper air data. The data structure is organized well, but may be inefficient in some access modes. Trajectories could be stored in MD files, but since trajectories are distributed more non-uniformly, performance and storage would be degraded. New data structure and MclDAS commands The current MclDAS data structures did not provide an efficient means for storing and manipulating 3-D grids and trajectories. The 2-D grid structure was used as a model to develop a 3-D grid structure. The naming of utility commands for accessing and listing the data are similar to their 2-D counterparts. But there are some important differences between the two. 3-D grids are stored as floating point numbers and there is no size limitation. The maximum number of grids per grid file was increased to 319 grids; but for large 3-D grids (40,000 points), only about 200 grids can L,,,.w4 v be stored due to limitations in the MclDAS file system. Some of these changes have brought about discussions regarding changing the 2-D structure. A new type of file was designed for trajectories. They are stored as lists of X, Y, Z, T coordinates with a maximum of 1000 trajectories or 30,000 coordinates. Currently, there is one trajectory file that can store up to 10 trajectory sets. This has not been a problem since trajectories are usually a derived parameter and can be regenerated when needed from the original data of U, V, and W 3-D grids. Along with the new data structures, new commands for MclDAS were written. A concise and general package of programs were written to manage and display 3-D grids and trajectories. A summary of these can be found in Appendix C. Management of 4-D and 5-D data sets Organization of multi-parameter time sequences of 3-D grids is necessary for not only the user, but also for the software. To generate long time animation sequences, the grids should be ordered such that this task can proceed somewhat automatically. We order the grids by grouping all parameters for each time, with an integral number of groups per grid file if the data set spans several files. TASK 3: DATA INTEGRATION Data quality assurance Most of the data we have worked with are processed in some way. For the modelers, any anomalies that would appear in the 3-D images would be useful in detecting problems in the model.