Publications

1986

Earth Observing System, volume IIa: Data and Information System, Report of the EOS Data Panel

Raymond Arvidson Washington University

Frederick Billingsley Jet Propulsion Labortory

Robert Chase Woods Hole Oceanographic Institution

Pat Chavez Jr. United States Geological Service

Michael Devirian NASA Headquarters

See next page for additional authors

Follow this and additional works at: https://commons.erau.edu/publication

Part of the Atmospheric Sciences Commons, and the Meteorology Commons

Scholarly Commons Citation Arvidson, R., Billingsley, F., Chase, R., Chavez, P., Devirian, M., Mosher, F., & et al. (1986). Earth Observing System, volume IIa: Data and Information System, Report of the EOS Data Panel. , (). Retrieved from https://commons.erau.edu/publication/551

This Report is brought to you for free and open access by Scholarly Commons. It has been accepted for inclusion in Publications by an authorized administrator of Scholarly Commons. For more information, please contact [email protected]. Authors Raymond Arvidson, Frederick Billingsley, Robert Chase, Pat Chavez Jr., Michael Devirian, Frederick Mosher, and et al.

This report is available at Scholarly Commons: https://commons.erau.edu/publication/551 https://ntrs.nasa.gov/search.jsp?R=19860021622 2017-10-09T16:01:38+00:00Z

NASA-TM-87777 19860021622 Volumella EliATH []BSEAU~~IJ SYSTEr:1

LIBRARY COpy

LANGLEY RESEARCH CENTER LIBRARY, NASA HAMPTON. VIRGINIA

REP[]RT []F THE E[]S lJfiTfi Pfi~El NAS/\ National Aeronautics and Space Administration 1986 Technical Memorandum 87777 EARTH OBSERVING SYSTEM REPORTS

Volume I Science and Mission Requirements Working Group Report

Volume II From Pattern to Process: The Strategy of the Earth Observing System Science Steering Committee Report

Volume IIa Data and Information System Data Panel Report

Volume lIb MODIS Moderate-Resolution Imaging Spectrometer Instrument Panel Report

Volume lIe HIRIS & SISEX High-Resolution Imaging Spectrometry: Science Opportunities for the 19905 Instrument Panel Report

Volume lId LASA Lidar Atmospheric Sounder and Altimeter Instrument Panel Report

Volume lIe HMMR High-Resolution MuItifrequency Microwave Radiometer Instrument Panel Report

Volume IIf SAR Synthetic Aperture Radar Instrument Panel Report

Volume IIg LAWS Laser Atmos·pheric Wind Sounder Instrument Panel Report

Volume IIh AItimetric System Panel Report

ii DATA PANEL FOR THE EARTH OBSERVING SYSTEM

Raymond Arvidson, Washington University Frederick Billingsley, Jet Propulsion Laboratory Robert Chase, Woods Hole Oceanographic Institution Pat Chavez, Jr., USGS Flagstaff Michael Devirian, NASA Headquarters John Estes, University of California at Santa Barbara Gregory Hunolt, NOAA Satellite Data Services Division J. Charles Klose, Jet Propulsion Laboratory George Ludwig, Consultant Frederick Mosher, NOAA National Severe Storms Forecast Center William Rossow, Goddard Institute for Space Studies

iii

EXECUTIVE SUMMARY

The purpose of this report is to provide NASA their research. We envision many Eos researchers be­ with a rationale and recommendations for planning, ing supported under ongoing NASA Earth science implementing, and operating an Earth Observing programs associated both with Eos and with other System data and information system that can evolve global Earth science programs. to meet the Earth Observing System's needs in the We therefore recommend that the Eos data and 1990s. The Earth Observing System (Eos), defined information system serve both the Eos project and by the Eos Science and Mission Requirements Work­ the more general NASA Earth Science and Applica­ ing Group', consists of a suite of instruments in low tions program needs, providing needed continuity to Earth orbit acquiring measurements of the Earth's receive data from other sources (e.g., the Upper At­ atmosphere, surface, and interior; an information mospheric Research Satellite, the Ocean Topog­ system to support scientific research; and a vigorous raphy Experiment spacecraft) and from the Eos program of scientific research, stressing study of spacecraft, to deliver these data to temporary reposi­ global-scale processes that shape and influence the tories for conversion to physical units, and to sup­ Earth as a system. The Eos data and information sys­ port long-term archives where the data will be ac­ tem is conceived as a complete research information cessible by and available to the greater scientific system that would transcend the traditional mission community. data system, and include additional capabilities such Mission operations and standard data processing as maintaining long-term, time-series data bases and tasks should be tailored to acquire and produce providing access by Eos researchers to relevant non­ viable scientific data for a given instrument. The in­ Eos data. The Working Group recommends that the itial data should be housed in temporary mission or Eos data and information system be initiated now, instrument repositories for primary analyses. These with existing data, and that the system evolve into data should then be provided to data centers for one that can meet the intensive research and data long-term maintenance and access by scientific re­ needs that will exist when Eos spacecraft are return­ searchers. In a number of cases, subsets of Eos and ing data in the 1990s. relevant non-Eos data should be maintained at active data base sites to support specific research tasks (ac­ tive data bases are subsets of data that are being Eos SCIENTIFIC AND OPERATIONAL routinely used in research and are under direct con­ ENVIRONMENT trol of a given research group). The resultant, highly processed data sets and associated documentation Meeting the Earth science objectives of the should migrate back to the data centers once the 1990s, delineated by the Eos Science and Mission Re­ specific, chartered research tasks of an active data quirements Working Group, requires in many cases base have been completed. The core of the Eos data access to data from a number of instruments, from and information system should be an electronic in­ in situ observations, and to data that have been formation network, alIowing access to the fulI suite placed in time-series data bases that extend over a of Eos system capabilities. This network should be decade or more. In that sense, Eos as a program will flexible, providing access to mission operations, ar­ clearly transcend the traditional NASA flight­ chives, selected active data bases, and to, for exam­ project model, which stresses spacecraft observa­ ple, large mainframe computers for certain, very in­ tions over a selected, and usualIy relatively short, tensive, computational activities (e.g., modeling) period of time, and includes initial analyses ofthe ac­ needed to support Eos data analyses. quired data. Thus, major issues for Eos and its data There will be at least four major types of Eos data and information system include how Eos will oper­ and information system users: ate both as a flight project and as a long-term scien­ tific research program. 1. The first type includes instrument team We envision Eos instrument teams that will guide members and support personnel, associated instrument and algorithm development and will be with Eos instrument or mission operations involved in mission operations and scientific analy­ centers. They will need to monitor continualIy ses of the data. We also envision that consortia of a sampling of data in near-real time for quality researchers, many stressing multidisciplinary analy­ assurance, error detection, and instrument ses, will form and intensely work with Eos data for malfunction assessment. They should have the extended, although not indefinite, periods of time. capability to reconfigure observational se­ In some cases, these research groups will request that quences when malfunctions or special events special sets of observations from a variety of sources occur. We do not expect that alI personnel will be acquired to meet their research needs. Many Eos need to be resident at the centers, since the Eos researchers will need efficient access to relevant non­ information network is envisioned to provide re­ Eos data bases and to a variety of models to conduct mote, electronic access to the centers' functions.

v 2. A second group of users will consist of re­ remote electronic access to the variety of capabilities searchers, instrument team members or and services that Eos will provide. operations-oriented personnel who need We recommend that the Eos data and informa­ instrument-specific, near-real time, or even tion system of the 1990s be designed to accom­ real-time data processing, delivery, and modate a suite of required functions, including: display capabilities. In some cases (e.g., NOAA or DoD) large data volumes will likely I. Eos flight system functions, including the be required. recommended functions and characteristics of both remote sensing instrumentation and on­ 3. The third user group will consist ofresearchers board data systems. who will need to interrogate directories and Flight systems functions are presumably more catalogs of Eos and other relevant data on an instrument, geographic location, and time of closely akin to those dealt with in previous missions. There are, however, new requirements being placed acquisition basis. They will also need to order into this category, requirements consistent with both data, and in some cases they will request that evolving research methodologies and developing particular observational sequences be ac­ technologies. Specifically, given the large data quired with Eos instruments. volumes and rates envisioned, there will be a need for 4. A fourth group of researchers also will need to expert systems, automated command and control, interrogate directories and catalogs ofEos and transparency in command control, rapid response, pertinent non-Eos data but is distinguished onboard monitoring, significant onboard buffers, from the third group by a need to browse Eos etc. Technically, the flight systems aspects of Eos data visually via attributes, or preferably via may be quite challenging. expert systems (to find particular features, at­ tributes, or special cases). 2. User functions, which embrace the likely ways in which researchers, operations personnel, In many circumstances, ranging from pipe-line instrument scientists and teams, and other in­ data processing activities to researchers dealing with dividuals requiring access to Eos services will Eos-data archives, access to non-Eos data will be operate. crucial to meeting processing and analysis require­ ments. Thus, the Eos data and information system User functions as a characteristic grouping are should provide a directory to the catalogs of pertin­ unique to Eos. In general terms, we envision the ent non-Eos data and, in some cases, to the catalogs, users of Eos and its data and information system . data, and.processing algorithms, per se. playing a significantly greater role than in the past. Researchers should be tasked with the responsibility of ensuring the overall functional capability of the system, presumably in an oversight and advisory ca­ RECOMMENDATIONS FOR THE 19905 pacity. Scientists should be obligated to return to the The Eos data and information system must be archives reduced or derived data sets resulting from designed to meet the challenges of Eos mission oper­ their research. Thus, the communications of data ations, transparent data access, transmission, pro­ and information will be bidirectional and not merely cessing, a,nd maintenance, as well as those imposed a one-way conduit with the researcher at the end of by the needs of scientists for non-Eos data sets. The the system. system should also accommodate "operational" 3. Operational functions, including those in­ users (e.g., NOAA) if their activities do not detri­ herent in spacecraft and data and information mentally impact the conduct of Eos scientific tasks. system operation. If operational uses are to be made ofEos data, and if these users significantly affect the information sys- . Operational functions, like flight systems func­ tem, then those operational users or agencies should tions, will bear some resemblances to previous mis­ be expected to provide the system enhancements sions. Since we anticipate that Eos will operate dur­ needed to meet their specific requirements. ing the same timeframe as the Space Station, it is We delineate within this report a number of func­ likely that some subset of the operational re­ tions that an Eos data and information system of the quirements may be met by activities not under the 1990s should fulfill. These functions do not con­ direct control of Eos, the research project. As an ex­ strain physical location of personnel, data, or pro­ ample, the bidirectional communications link be­ cessing capabilities. In the 1990s, we expect that local t ween spacecraft and ground facilities will un­ processing capabilities, combined with modern net­ doubtedly be operated and managed by non-project work capabilities will allow a geographically distri­ personnel. There are a host of other operational re­ buted information system to become a reality. In quirements that will not be provided by other ac­ fact, the goal of the data and information system tivities and must be satisfied within the confines of should be to provide the scientific community with Eos per se. These requirements include quick-look

vi data production for quality assurance purposes, 7. Functional requirements for non-Eos data command management, quality assurance, access bases consistent with overall Eos scientific services, near-real-time processing, etc. objectives. 4. Eos information services functions, encom­ The principal function associated with non-Eos passing the suite of network services (both data bases is straight forward, transparent access. space- and ground-based) that are required. Ideally, this would provide the researcher with a Information network services functions are envi­ "one-stop shopping" capability. Through close col­ sioned to be far more extensive and comprehensive laboration of archival personnel and advanced net­ than those employed in previous experiments. Sys­ work capabilities, a researcher could explore data tem transparency while accessing geographically catalogs, browse data sets, select pertinent data, and dispersed, heterogeneous data archives will be a fun­ order these data, all from the same facilities that he damental requirement. We believe the primary goal customarily uses for similar services from Eos ar­ of the system should be remote, electronic access to chives. Although the major difficulties that will be the host of services and capabilities afforded by Eos. encountered are political, every effort should be These include mission and instrument operational made to provide these functional capabilities since planning; access to directory, catalogs, and data the success of Eos will be largely dependent upon a bases; a spectrum of communication services; man­ researcher's ability to utilize all relevant data in his agement of inter- and intra-system information work. flow; access security, etc. Provisions must be made to ensure maximum flexibility in data management Eos will be a unique space-research program; and communications network design, accommodat­ unlike many, the functional elements of its data and ing needs of future researchers. information system are highly interdependent. Re­ sponsibilities residing within one element or category 5. The functional requirements characteristic of may require contributions from one or more addi­ advanced data base management practices. tional elements for completion. Thus, for Eos to be The requirements for advanced data base man­ successfUl, all of the requirements and systems' at­ agement functions are probably more extensive than tributes delineated within this report (and briefly for the other functional groupings. They may be sub­ summarized in an appendix) must be provided. Fur­ divided into electronic directory and catalog, browse thermore, to ensure that the resultant system can file, data ordering, documentation, data archives, meet researcher's needs, the architectu~al design and standards requirements and functions. The should feature and utilize two fundamental prin­ prime purpose of incorporating extensive advanced ciples or contemporary design techniques through­ data base management capabilities is to maximize out. These principles are "layering" (the technique scientific efforts and exploit Eos capabilities to their of dividing and conquering that produces modular­ fullest. The requisite functions within this group are ity) and "standardization" (of formats and proto­ cols, which promotes data autonomy, hence trans­ designed to allow scientists to focus on research, rather than on the details of accessing and preparing parency). Together, they can be used to create a data data for analysis. and information system that is adaptively flexible, transparent to the user, and robust, providing the 6. Eos ~ata processing functions, tailored to foundation for diverse and evolving data processing meeting a spectrum of requirements imposed and archival needs of Eos researchers and opera­ by the four major types of Eos data and infor­ tional users. mation system users. Data processing requirements will be highly varied. They range from quick-look processing .for RECOMMENDATIONS FOR THE 19805 instrument and mission operational personnel, to The key to successfully implementing the recom­ routine preprocessing to Levels 0 and lA, to higher mendations for the 1990s rests squarely with the level data reductions, to mainframe modeling capa­ care, planning, and attention devoted to initiating an bilities. Near-real and real-time processing will be re­ Eos data and information system during the 1980s. quired for operational purposes and may be of value The requisite information system should be evolu­ for interactive browse activities. Processing capacity tionary, supporting ongoing and. near-term NASA and performance should be capable of flexible programs in the Earth sciences with existing data. enhancement to meet evolving needs defined by in­ We recommend that the Eos data and information strument teams. Data reduction, grid overlay, stan­ system be built upon the experience gained from dard projection data set production, and data set NASA's existing pilot data system projects (i.e., merger will be required by the majority of Eos Pilot climate, Pilot Ocean, and Pilot Land Data Sys­ researchers. Throughout these data processing ac­ tems) with a focus toward Eos data and information tivities, self-documenting software will be needed to system objectives. We further recommend close col­ produce inventory, catalog, and directory entries. laboration and coordination with the Unidata

vii initiative of the University Corporation for At­ developing detailed requirements for needed data mospheric Research (UCAR), and utilization of the sets, for the data base management and network en­ Global Resource Information System philosophy as vironment, and for evaluations of whether the evolv­ a unifying concept. ing information system adequately meets scientific The pilot projects and Unidata address discipline­ needs. In developing the information system, par­ unique problems. On the other hand, the Global ticular emphasis should be placed upon technology Resource Information System concept stresses ac­ associated with flexible and transparent access to cess to data from each of these pilots and from other dispersed, heterogeneous data bases, to advanced relevant NASA and non-NASA data bases. In es­ data base management techniques (including search sence, a major task that can and should begin now is capabilities employing expert systems), and to cost­ facilitating access to the geographically dispersed, effective electronic network approaches (including heterogeneous data bases that already exist within direct broadcast of data). these pilot projects and at a number of university and The final, and perhaps pivotal recommendation agency locations. of th is panel is to initiate the planning and implemen­ Development of the information system should tation of an evolutionary Eos data and information be done in close collaboration with scientists sup­ system without delay. A functional system providing ported under NASA's Earth Science and Applica­ the means through which Eos data will be fully ex­ tions Division programs. In doing so, researchers ploited cannot be built in a matter of a few years; it would use the network and access, for example, the must be encouraged and allowed to evolve in concert pilot data systems, thereby exercising the overall em­ with the ever increasing knowledge base of the Earth bryonic system and allowing NASA and its research sciences and with the requisite expertise to manage community to gain experience appropriate to estab­ this data base. There are two imperatives or guiding lishing a data and information system well suited to principles which should be closely followed through­ the Eos spacecraft era. We therefore recommend that the Earth Science and Applications Division out this evolutionary process. They are (i) involve the fund a limited number of multidisciplinary studies scientific research community at the outset and and discipline-oriented research teams (requiring throughout all subsequent activities, since the data multiple data sources) that can begin to address will be acquired, transmitted, and processed for specific Eos scientific objectives in biogeochemistry, scientific research, and (ii) provide a representative hydrology, and climatic research, as well as in group of active researchers with an oversight and re­ specific disciplines requiring multiple data sources. view responsibility, since the most successful ex­ These teams will provide scientific focus for develop­ amples of data base management involve user ing the Eos data and information system, including oversight.

viii PREFACE

The sciences, by their very nature, are evolutionary. The questions posed and information needed to address specific problems (as well as the problems themselves) change through time as we learn more about a particular phenomenon or process; this is scientific progress. Similarly, the computer and communications industries have been and continue to experience significant technological progress year after year. With this comes inevitable change: change that when capitalized upon will lead to better and more efficient ways of conducting scientific research. A report such as this, which deals with scientific data and information systems, should not constrain a future system to today's innovations but rather should leave as its legacy a firm set of guiding principles and recommendations that will stand and remain valid long after its authors have accepted new challenges and well beyond the time when a specific piece of hardware or software has become obsolete. It will provide an approach and methodology for designing and building a flexible data and information system that is consis­ tent with the evolving character of the sciences it seeks to serve. As well, its flexibility will allow new technological advances that loom on the horizon to be readily incorporated into its design. The Eos Data Panel has, I believe, succeeded in creating a report of this genre. Clearly, a report of this breadth must draw quite heavily upon the expertise, experience, and knowledge of a large quorum of dedicated professionals. On behalf of the Eos Data Panel, I would particularly like to thank those individuals who graciously gave of their time and talent to contribute to this report. AlphabeticalJy, they include:

Mark Abbott, Scripps Institution of Oceanography Dixon Butler, NASA Headquarters Jim Dodge, NASA Headquarters John Dutton, Pennsylvania State University Ed Greenberg, Jet Propulsion Laboratory Dick Hartle, NASA Goddard Space Right Center Ed Hurley, NASA Goddard Space Right Center Tom Karras, NASA Goddard Space Right Center Ron Muller, NASA Goddard Space Flight Center Rick Pomphrey, Jet Propulsion Laboratory Stan Sobieski, NASA Goddard Space Flight Center Jeff Star, University of California at Santa Barbara Mike Ward, NASA Goddard Space Right Center Jim Weiss, Jet Propulsion Laboratory

To the host of other individuals who reviewed and commented upon various drafts of this report, our sincere thanks.

Robert R.P. Chase Chairman, Eos Data Panel Woods Hole, Massachusetts October 1985

ix

CONTENTS EXECUTIVE SUMMARY ...... v Eos Scientific And Operational Environment ...... v Recommendations For The 1990s ...... vi Recommendations For The 1980s ...... ' ...... vii PREFACE ...... ~ ...... ix I. PURPOSE AND SCOPE OF REPORT ...... I II. Eos SCIENTIFIC AND OPERATIONAL ENVIRONMENT ...... 2 Eos Data Users ...... 2 Data Acquisition ...... 3 Data Processing ...... 5 Telecommunications ...... 9 Documentation ...... II Non-Eos Data And Models ...... 12 Standards ...... 13 Summary ...... 15 III. ARCHITECTURAL CONSIDERATIONS ...... 16 Design Principles ...... 16 Design Considerations ...... 17 Functional Architecture: Top Level ...... 18 Functional Architecture: Peer Layers ...... 22 IV. MANAGEMENT CONSIDERATIONS ...... 24 Longevity ...... 24 Synergistic Characteristics ...... 26 Data Volume And Rate ...... 27 Archival Data Sets ...... 27 Flexibility In Design: Future Users ...... 28 Economic Factors ...... 28 Interaction With Other Governmental Entities ...... 29 Rewards And Punishment: A Personnel Dilemma ...... 29 Key Issues ...... 30 V. RECOMMENDATIONS AND CONCLUSION ...... 31 Operational And Commercial Users ...... 31 Functional Elements: The 1990s ...... 31 Information System Evolution: The 1980s ...... 32 Conclusion ...... 33 REFERENCES ...... , ...... 34 APPENDIX I: RESEARCH SCENARIOS ...... 35 Climatic Research ...... 35 Climatic Monitoring ...... 37 Land Surface Climatology ...... 37 Geologic Mapping...... 39 Meteorological Uses ...... 40 Vegetation Biomass ...... 41 Temporal Information Extraction ...... 42 APPENDIX II: REQUIREMENTS SYNOPSIS ...... 44 Flight Systems Functions ...... 44 User Functions ...... 44 Operational Functions ...... 45 Information Services Functions ...... 45 Advanced Data Base Management Functions ...... 46 Data Processing Functions ...... 48 Non-Eos Data Bases Functions ...... 48 Management Considerations ...... 49

xi

I. PURPOSE AND SCOPE OF REPORT·

The Earth Observing System Science and Mis­ turn, established a number of panels, one of which is sion Requirements Working Group was formed as the Eos Data Panel. an ad hoc advisory body to NASA's Earth Science The Data Panel was charged with examining an and Applications Division and was charged with Earth Observing System data and information sys­ evaluating: (i) the potential for increasing our tem in more detail. This report, resulting from understanding of the Earth and how it works by deliberations by the Eos Data Panel, is intended to utilizing systematic, multi-instrument observations provide guidance to NASA in developing the req­ from low Earth orbits, (ii) the role of in situ uisite Eos data and information system to meet the measurements, and (iii) requirements for programs needs of the Eos research community throughout the of theory and data analysis to support scientific in­ next decade, including the era when Eos instruments terpretations of these data. The Working Group will be acquiring data. delineated a number of high-priority, global-scale The Eos Data Panel began its deliberarions by creating realistic scientific research scenarios (Ap­ Earth science questions that should be addressed pendix I) that closely parallel the scientific objectives during the next decade. They then considered set forth in the Eos Science and Mission Require­ synergistic groupings of instruments to best meet ments Working Group Report. The Panel then re­ these scientific objectives, the role of in situ viewed the current state of and likely developments measurements, and the need for a computation and within the computer and telecommunications indus­ data management environment that would facilitate tries. This information, together with the lessons of research in an Eos era'. history (embodied to a great extent within the Na­ The Working Group pointed out that the data tional Academy of Science Committee on Data Man­ variety, complexity, and volume to be produced by agement and Computation reports), forms the basis Eos instruments would present major challenges in for developing the 'd·ata and information system's mission operations, data transmission, processing, scientific and operational environment (Chapter II). and long-term maintenance of data that will ulti­ Based upon the requirements, attributes, and mately constitute a portion of the time-series data characteristics of this environment, information bases. In fact, the Working Group considered that system design and architectural considerations Eos as a whole should be thought of as an informa­ (Chapter III) were determined. tion system, whose development should begin now, Throughout this process, the Panel encountered with existing data. Through use of sound manage­ many features that make Eos unique among flight ment principles and appropriate technologies, this projects. These unique features must be given information system could evolve to meet the needs of careful thought and consideration by NASA man­ Eos during the 1990s. agement (Chapter IV) if Eos is to be successfully im­ The Working Group was succeeded by an Eos plemented. Finally, the Panel summarized its major Science Steering Committee. This Committee pro­ findings and delineated an approach or implementa­ vides a broad scientific oversight for the Eos Pro­ tion strategy for developing the Eos data and infor­ gram and Project. The Steering Committee has, in mation system (Chapter V). II. Eos SCIENTIFIC AND OPERATIONAL ENVIRONMENT

The Eos Data Panel developed requirements for groups will require quick-look processing of data an Eos data and information system operating in the from a specific set of instruments, over specific 1990s by considering a number of possible research regions, at specific times. The processed information scenarios. Several of these scenarios are presented should be rapidly transmitted to these user teams so for reference in Appendix I. There are clearly strin­ that it can be used in deciding, for example, the relo­ gent requirements, based on the scientific user cation of mobile, in situ measurement systems (e.g., scenarios, for determining what Eos, other NASA, aircraft, ships, autonomous platforms). and relevant non-NASA data exist, acquiring these A third subgroup can be loosely termed the spec­ data in suitable form, and having the capability to tacular event monitoring group. This group re­ analyze the data. In addition, there are challenging sponds rapidly to major scientific events, such as requirements for mission operations, especially volcanic eruptions. It differs in function from the when operational or rapid response functions are previous group in that the lead time for planning and considered. In the following pages we discuss re­ coordination is greatly reduced. Consequently, it quirements for an Eos data and information system will require rapid decision-making capabilities in an designed to address those needs. Eos operations center and an established real-time processing and display capability that can be used to monitor an event. Eos DATA USERS The potential size of these groups can be esti­ mated using existing organizations and funding as a In determining the characteristics and attributes guide. It has been assumed that there will be no ma­ of an Eos data and information system, the intended jor increase in the number of scientists or funding for user community must be identified, its access pat­ the Earth sciences in general, except for the direct terns anticipated and delineated, and the projected funding necessary to proceed with Eos. The only volume of requests estimated. We have identified real-time operational users of Eos data requiring four major classes of Eos data users: full, instrument data streams will probably be NOAA and possibly 000. At any given time, there 1. Eos Operations Centers Users probably will be several groups (order 20) requiring quick-look data and a few (perhaps one or two) re­ In order to ensure the collection of complete data questing instrument command control. sets, free of instrument errors, timely delivery of data to Eos mission or instrument operations centers 3. Users of Archival Data, Requirements Known will be necessary. A sampling of data from all in­ struments should be processed and delivered to the This group of users has known requirements for centers in a format that is easily displayed and under­ instruments, geographic areas, and times of data ac­ stood. These data will need to be continually moni­ quisition. It will require a catalog of data availabil­ tored to ensure reliable collection. If problems oc­ ity, with quality attributes of the data sets. The size cur, a center will need to have the capability to trace of this group could be as large as 1,000 to 10,000 the data flow back through the processing system to users. Of this group, approximately 50 to 200 will be the instrument, aiding in the isolation and correction active users at any given time. Each group member of problems. would be ordering 5 to 10 computer tapes per month. A few (1 to 10) of the active users will be ordering 2. Instrument Specific, Real-Time Data Users larger volumes of data. With the anticipated im­ provement in computer technology in the 1990s, it is During the Eos spacecraft era, an instrument expected that these users would want to order and specific, real-time data user will require online data process approximately 10 times as much data as they processing and display capabilities. Within this user do currently. group there are several subgroups. Operational users, such as NOAA or 000, who will require en­ 4. Users of Archival Data, Attributes Known tire, real-time data sets from specific instruments, constitute one subgroup. Their processing require­ This user group differs from the others in that it ments will range from raw data to fully processed addresses scientific problems that require data with scientific data. certain attributes (e.g., a particular type of cloud, a A second subgroup is composed of instrument vegetation feature with certain characteristics). This teams or research groups selected from an An­ will require a browse capability to select appropriate nouncement of Opportunity. They may require con­ data. trol over the capabilities of various instruments to There are two different types of browse re­ satisfy scientific requirements. In general, these quirements. One is a visual presentation of the data

2 along with catalog listings of attributes. Well de­ of a master schedule that should be updated every or­ fined, standard data attributes within a discipline bit. The schedule should contain logistical informa­ should be selected automatically and extracted for tion needed to plan an observation. Access to the catalog entry during data processing. The attribute schedule and the subsequent planning could be done file should also be expandable, enabling user­ electronically via a terminal, by formal proposal, or identified attributes to be added. Similarly, as new by agreement between collaborators. The system algorithms are developed for identification of at­ should utilize prompting algorithms for requests tributes, these algorithms should become part of the made electronically. attribute file processing system (but old data should not be reprocessed to include these attributes with­ Uplink Activities - Activity Planning out substantial peer and management review). The user community for these browse catalogues could Eos will utilize many strategies for data acquisi­ be as large as 10,000 individuals. tion. Planning, of necessity, should be conducted in­ The second browse capability required is a re­ teractively, utilizing project-provided software tools duced volume data set suitable for custom process­ for design, iategration and constraint assessment, ing by researchers. This data set will need to be and current resource summaries. These tools should available at a number of processing levels. Most allow command sequence design and integration, users of these browse data sets will want highly pro­ from simple requests to complex operations that re­ cessed versions of the data. The user community for quire the coordination of several subsystems and in­ this type of browse capability could be as large as 500 struments for multidisciplinary investigations. With to 5,000. these tools an experimenter could develop integrated command sequences that may be executed over long time periods, performing routine observations or DATA ACQUISITION providing for complex, critical observations in response to unique opportunities. The development The concept of interactive, transparent system of these plans could be interactively performed in access should guide the creation of an environment concert with other investigators (via teleconferences) in which scientists will operate when using the Eos and with access to central archives of reference data. data and information system. This concept includes the complete data acquisition (including archival Uplink Activities - Command data as well as new observational sequences) and analysis cycle, since it addresses both the need to We envision several options available to the user, transform a request into action (causing a remote depending upon the desired operational scenario. system element to acquire data) and the need to Commands should be separated into categories that deliver processed results to the requester. A resear­ are non interactive, interactive, and critical. Com­ cher's system interface device may be as simple as a mand execution could be in real time, near-real time, smart terminal or as complex as a large, mainframe or preplanned. Interactive commands that require computer, with the majority of scientists depending critical onboard resources should be processed at a upon commercially available, microcomputer-based ground-based control center, while noninteractive workstations. Access to in-flight subsystems should commands should be transparently transmitted be provided through a coordination and control sub­ from the user to the onboard system for final assess­ system for all commands; those not requiring more ment and forwarding to the appropriate instrument than the allotted amount of critical system resources or subsystem. Commands to be used in an emergen­ should pass through this subsystem transparently. cy situation should have prior validation and be cen­ Within a downlink telemetry access period, execu­ trally stored for immediate delivery. Eos should tion times for any flight system operating within the utilize all of these operational capabilities. Eos data and information system should be in sec­ onds, and those for archival data should be within Uplink Activities - Monitor and Control minutes for small volumes, hours for medium vol­ umes, and days for large volumes, depending upon During particular operations, necessary the priority of the request. Thus, transparent access engineering and quick-look data should be provided and rapid and timely responses are not only expected to researchers for monitoring purposes. These data of the system, but also are a prerequisite. should be delivered in near-real time with delays of order seconds when the spacecraft is within a down­ Uplink Activities - Data Requests link telemetry reception period. For emergency com­ mands, loop delays should not be more than 30 sec­ Requesting observations from a flight instru­ onds between command delivery and received re­ ment will involve decisions based on preset priorities sponse. Investigators should have the capability to and the cooperation of many simultaneous resear­ monitor their instrument from their home institu­ chers and operational users. This necessitates the use tion, and under certain conditions they may need

3 command access. Access security must therefore be operation be continued from Eos platforms carrying maintained at appropriate levels at all times. operational instruments. It should be possible to receive directly transmitted data at any modestly Onboard Processing - Command equipped ground station that is within line-of-sight of an Eos platform. This is also an important capa­ Onboard systems should be used for decisions bility for high data rate research instruments (e.g., when rapid response is needed. These situations may the High Resolution Imaging Spectrometer or the include changes of gain settings, filters, or data ac­ Synthetic Aperture Radar), whose duty cycle may be quisition rates during anomalous conditions. These constrained by the satellite's data recording system capabilities can be based on expert systems or soft­ capacity and downlink availability. Most direct ware algorithms and could be used to plan inter­ broadcast data should be produced in a raster format disciplinary investigations. Systems of this genre that can be readily interpreted. could respond to short-term events in a coordinated, priority fashion, allowing scientists to take advan­ tage of measurements that otherwise would not be Data Acquisition - Real-Time Monitoring acquired. The Eos data system should have a real-time telemetry data processing system to monitor satellite Onboard Processing - Function and instrument status, and to confirm responses to Monitoring uplink commands. Data for this purpose should be received in real time, perhaps by direct broadcasts Flight instruments will require a certain level of from the satellite. Eos must maintain a mission oper­ onboard monitoring to ensure their functional capa­ ations center where this real-time data is processed bility and that of various subsystems. The areas of and available to operations personnel. We envision greatest importance are fault identification, isola­ that at least 95 percent of the real-time data broad­ tion, and protection. Systems used for this purpose cast by the satellite could be received, preprocessed, could be a combination of software algorithms and and displayed within 60 seconds from the time of symbolics code. Many of the ground-based monitor­ transmission. ing functions could be automated, tested on the ground, and eventually integrated with the space­ craft for onboard system control. Data Acquisition - Data Records The communications link between the Eos plat­ Onboard Processing - Data form(s) and ground-based data reception facilities Onboard data processing provides an opportu­ should be very stable. As a design goal, for any given nity to red uce the vol ume 0 f in formation transmitted orbit, all of the data transmitted by the space plat­ to the ground. Systems and algorithms that make de­ form should be received in a form suitable for pro­ cisions involving data acquisition strategies and cessing (e.g., as standard formatted data units) by record content should be used, thereby helping to the ground system, within 90 minutes of acquisition. maintain a reasonable downlink telemetry load. To achieve this high percentage of data acquisition, it Techniques for data compression, editing, filtering, may be necessary to develop recall mechanisms to re­ and refining data should be used for operational cover data lost during downlink transmissions. An data sets. Certain levels of data reduction from on­ accurate accounting of data received will be board decisions may become a reality. required. Data sequence and encode peculiarities should be Onboard Processing - Telemetry removed by the data reception system. Data that is transmitted from a recorder, in reverse order, should The onboard processing system should have the capability to format, error code, and buffer telem­ be rectified and presented in chronological sequence. etry data, as well as the capability to merge critical Any encoding or data packet formation applied to ancillary data needed for quick-look and other anal­ data for downlink transmission must be removed yses. This assumes that data collection would be per­ (i.e., after the data have been preprocessed by the formed by a scheme that does not require filler data data reception system it should be in the same format to produce a standard record length, or set interroga­ that it was in when it left the instrument system). tion times. For each unit of data (e.g., one orbit, one hour) preprocessed by the data receiving system, a catalog Onboard Processing - Direct Broadcast inventory record should be generated. This record should contain data record starting and ending Direct broadcast data from NOAA environmen­ times, data accountability information, etc. This in­ tal satellites have proven to be valuable for many formation should be maintained by an Eos data cata­ users. Therefore, it is desirable that this kind of log system.

4 Data Acquisition - Ancillary and can be transmitted effiCiently over medium-rate Correlative Information (9,600 bps) communications links and displayed on generally available graphics and image display ter­ Both for research and quality assurance pur­ minals. To be of greatest utility, this information poses, ancillary and correlative information will be should be available within the time period of one or­ needed. Much of this information will be derived bit (nominally 90 minutes). Many of the near-real­ directly from various spacecraft subsystems (e.g., at­ time data sets will be selected prior to instrument titude control, universal time) or through an ad­ deployment, but it should be possible to implement vanced data collection and location system. In both new data requests in response to evolving research cases, these data should be readily available to the needs and as new field experiment support require­ Eos data: and information system and provisions ments are identified. should be made to merge these data with Eos instru­ ment data, as required. Other types of ancillary information will be resi­ Data Reduction - Processing Level dent in non-Eos archives. Access to these data may Definitions be crucial to a particular research project or to understanding the characteristics and calibration of To facilitate the discussion of data processing, a given instrument. Consequently, provisions should several processing "levels" are defined and used be made to access this information as needed. throughout the remainder of this report. These de­ finitions are:

DATA PROCESSING Level 0 - Reconstructed unprocessed instrument data at full resolutions. The full value of satellite-derived data cannot be realized unless careful thought is given to its process­ Level IA - Reconstructed unprocessed instru­ ing, maintenance, and distribution. The Eos data ment data at fuIi resolution, time referenced, and processing system should: annotated with ancillary information, including radiometric and geometric calibration coeffi­ I. provide near-real-time processing in support of field experiments, event monitoring, and cients and georeferencing parameters (i.e., plat­ quick-look scientific data analyses; form ephemeris) computed and appended but not applied to the Level 0 data. 2. process data to a level that is readily usable, by applying algorithms that have been approved LevellB - LevellA data that has been processed and validated by the research community; and to sensor units (i.e., radar backscatter cross sec­ tion, brightness temperature, etc.). Not all in­ 3. maintain and distribute data sets (i.e., both unprocessed and processed), ancillary infor­ struments will have a Level I B equivalent. mation, and documentation that describes the processing procedures. Level 2 - Derived environmental variables (e.g., ocean wave height, soil moisture, ice concentra­ Eos data can be processed in the traditional sense tion) at the same resolution and location as the by principal investigators or investigator teams at Level I source data. university-managed facilities, or by project teams at project-managed facilities. In either case the need Level 3 - Variables mapped on uniform space­ for timely production of research-quality data sets is time grid scales, usually with some completeness. essential. All data processing requirements noted in and consistency properties (e.g., missing points this report are independent of specific organiza­ interpolated, complete regions mosaicked to­ tional assignments and must be met by any facility gether from multiple orbits). (including commercially operated) that is producing Eos data. Level 4 - Model output or results from analyses of lower-level data (i.e., variables that were not Near Real-Time Processing measured by the instruments but instead are de­ The Eos data processing system should support rived from these measurements). near-real-time data and information processing. These data will be used to support field experiments, Level of Processing monitor environmental events (volcanic eruptions, forest fires, floods, etc.), and provide the research A Levell data record is the most fundamental (i.e., community with an instrument quick-look capabil­ highest reversible level) data record that has significant ity. In general, quick-look information should be in scientific utility, and is the foundation upon which all the form of a graphics metafile or an image file that subsequent data sets are produced. Consequently, all

5 Eos data should be processed to at least Level I. volume of one-tenth to one one-thousandth of the Given the volume of data involved and the likelihood original set should provide these properties. The re­ that instrument teams and individual researchers will quirements of data set subsampling and spectral opt to process or have processed only select subsets band sampling will need further study. However, it is of data, it is recommended that processing beyond suggested that several reduced-volume data sets be Level IA be handled on a request basis (except for produced that are targeted for different scientific ap­ the production of browse data sets). The basic pro­ plications (e.g., from the moderate resolution imag­ cessing should rectify the instrument data at full ing spectrometer), since the degradation of spectral resolution, append calibration coefficients, geogra­ resolution is discipline-dependent and in some cases phic location parameters, etc. It is assumed that the instrument-specific. Spectral bands critical to each accurate determination of calibration, Earth loca­ of the different scientific disciplines should be pro­ tion, etc. will be performed by Eos instrument teams cessed to identify specific data attributes (e.g., per­ and will be the responsibility of instrument principal cent cloud cover, snow, soil, vegetation, water, scene investigators. These tasks are critical to the success­ ful utilization of Eos data and as such should receive average brightness, standard deviation, surface clas­ significant attention to forestall any difficulties in sification, time of day). Pattern recognition producing data to Level IA. algorithms should be applied at this step to identify Level 2 is the first level that is directly usable for attributes that should be recorded in the browse file. most scientific applications; its value is much greater One feature of these data sets is that they do not re­ than the lower levels. Level 2 data sets tend to be less quire external, ancillary data sources for processing. voluminous than Level I data because they have Finally, consideration might be given to producing been reduced temporally, spatially, or spectrally. browse files interactively, from a larger data set, as a Once verified, the value of Level 2 data remains high researcher queries the system. Thus, it would be pos­ for a long period of time, declining only as newer sible to add a variety of browse presentations to the data sets attract the interest of researchers. total Eos holdings. Level 3 data sets are generally smaller than lower­ level data sets and thus can be dealt with without in­ Instrument and Algorithm Performance curring a great deal of data handling overhead. These data tend to be generally more useful for many All Eos instrument parameters should be applications. The regular spatial and temporal monitored through the periodic transmission of organization of Level 3 data sets makes it feasible to standard diagnostic data sets and by examining their combine readily, data from different sources. The performance statistically (e.g., compiling minimum, ability to produce combined or overlay data sets will maximum, and mean values, and standard deviation greatly facilitate many scienti fic investigations. for given measurements) or by generating an alarm if Therefore, it is recommended that any data from the value of agiven measurement does not fall within Eos instruments processed to Level 3 be produced in a predicted range. This might best be done by ex­ standard formats. Since the definition of a Level 3 amining Levell data and compiling instrument per­ data set may be application-dependent it may be nec­ formance data for each unit of data processed. It is essary to produce several "standard" data sets or essential that the performance of each instrument be customized sets to meet particular needs. monitored, and that unexpected behavior be noted In addition to Level I A data, there will be com­ and evaluated. The history of an instrument's per­ ponents of the Eos data and information system that formance should be maintained in archives where it will be required to produce Level 2 and Level 3 data will be readily available to the research community. sets to meet scientific objectives established by Eos­ Eos-derived environmental variables should be sponsored scientific researchers or research teams. monitored in the same manner that instrument pa­ As each data set is processed, attribute and accoun­ rameters are monitored. Statistical information tability information should be compiled (e.g., in­ should be compiled for each unit of Level 2 data pro­ strument name, data starting and ending times, pro­ cessed. Continuous monitoring of algorithm perfor­ cessing level, algorithms used, number of records mance is essential if Eos is to produce uniformly processed, percentage of data processed success­ high-quality Level 2 data. The history of an fully) and appended. This information could also algorithm's performance should be maintained in form the basis of an inventory record that should be the archives. maintained by an Eos data cataloging subsystem. Validation Browse Data Sets All Eos data should be validated to the satisfac­ It is recommended that a reduced volume data set tion of the research community. It is anticipated that be routinely processed to Level 2 for browse pur­ this would be done by first conducting an intense val­ poses. This data set should maintain the statistical idation effort lasting between three and six months, properties of the original. A browse data set with a followed by an ongoing monitoring effort. During

6 the initial validation period, Level I and 2 data 4. preserve, maintain, and distribute all Level would be substantiated by comparing data from Eos IA (produced by Eos platforms) and higher­ instruments with those of similar spaceborne instru­ level data holdings throughout the lifetime ments and with in situ observations. Once the initial of the program; validation has been accomplished it will be necessary to continue monitoring instrument and algorithm 5. accept higher-level data sets produced by in­ performance for departures from the norms estab­ vestigators as part of their research work; lished during the initial validation. It may be nec­ and essary to mount additional validation efforts if the 6. reprocess Eos data when needed. instrument or algorithms do not perform as antici­ pated, for quality assurance purposes, or after The archives should be able to store and maintain known changes (e.g., replacement, refurbishment) these data as well as provide a directory and catalogs of instruments. It is essential that the scientific com­ for their retrieval. The directory should include in­ munity participate in both the initial and ongoing formation about all Earth sciences data deemed rele­ validation efforts and that pertinent feedback from vant by Eos-sponsored researchers. The catalog sup­ any retrospective scientific analysis be utilized when porting the directory should include entries for all appropriate. Eos data sets and extend to the holdings of other rele­ vant archives, when needed. The archives should Algorithm Enhancement and Data also maintain a complete inventory of relevant scien­ Reprocessing tific and technical documents concerning the data and the archives. Finally, it is envisioned that the Eos It is anticipated that during the course of the Eos archival subsystem will be geographically distributed project, new or enhanced processing algorithms or with elements at both data centers and at the active procedures will be found that will substantially im­ data base sites. It is assumed that these elements will prove the utility of Eos data. When this occurs, the be connected via an electronic network and that uni­ Eos data processing system should be sufficiently form access to all system services will be provided. flexible, easily accepting new algorithms and pro­ cedures. Since algorithm changes are inevitable, an User Interface accurate processing history must be maintained for all Eos data. In some cases, algorithm changes will An Eos archival subsystem should be accessible have such a significant impact that it will justify the from remote terminals or workstations via a promp­ reprocessing of previously processed data. There­ ting menu or natural language interface, supple­ fore the Eos data processing system must be able to mented by a free-form command language for exper­ efficiently accommodate new algorithms, access ienced users. All capabilities and functions should be lower-level data that will be utilized in the repro­ fully documented in a user handbook and in online cessing activity, and update the archives with the help files. Consideration should be given to online resultant, improved data. tutorials that could be used to train novice system users. For those users who choose not to interact Maintenance of Data Sets and Documents directly with the Eos archival subsystem, mail and telephone request should be considered. The main function of the Eos archival subsystem is to preserve, maintain, and distribute data from Electronic Directory and Catalogs Eos instruments for use by the research community. However, the term "archives" is broadly defined; it An archival subsystem should provide an online is not merely a repository for data. Rather, it is an in­ catalog of data sets of interest to Eos investigators. formation reservoir and conduit where both the The data set references in the catalog should be de­ quality and quantity of data are increasing and termined by an Eos scientific advisory group or by where access to and use of the data are encouraged. various user groups. Each data set should have a cor­ The Eos archives should be able to: responding higher-level directory entry that contains general information about the data set (e.g., spatial I. accept data from Eos instruments at all pro­ and temporal coverage, parameters measured, data cessing levels; set name, archives location, availability, and contact person). References to pertinent literature should 2. provide access to non-Eos data that are also be included. Those data sets being produced by needed for processing of Eos data or by Eos the Eos data processing system should also have a investigators; lower-level inventory, containing more details (e.g., parameter measured, calibration information, pro­ 3. provide in situ data used in the processing cessing levels, applicable algorithm descriptions, and validation of Eos data sets, or as cor­ measurement precision and accuracy, documenta­ relative data; tion) about availability of data at specific times and

7 places and appropriate access information (e.g., abstracts. Displays should be at the user's terminal, volume identifier for data sets on optical or magnetic or on a central printer, at the user's option. A user storage media). should have the capability to request a copy of any A user should have the ability to search the document found in the search. They should also be catalog by any and all of the following attributes: able to access the complete document text electron­ time, place, instrument or a related group of in­ ically for display or printing at their terminal or struments, project or program, parameter(s), data workstation. set name, etc. The search should be limited to the top-level catalog, until a specific data set is selected Data Set Maintenance and Distribution by an investigator for further examination. At that point the user should be notified whether a detailed The primary function of the Eos archival sub­ catalog exists, and, if so, what further search criteria system is to store, maintain, and distribute data sets are applicable. Detailed search criteria depend on in­ from Eos instruments, from other supporting space­ formation availability for each catalog, but gen­ borne instruments, and from in situ measurement erally should include at least time, place, instrument, systems. The Eos archival subsystem should there­ and data attribute information. fore provide the following services, all of which are An investigator should have the option of dis­ considered to be of equal importance: playing a list of data sets found, or complete top­ level catalog contents, in any of several information 1. Ingest Eos data sets as they become available. categories. If a detailed catalog search is requested, This includes both scalar and raster satellite data, the results should contain all appropriate and avail­ and in situ data sets. The archives should be able to able information for that particular catalog entry. absorb the large data sets ofthe 1990s and beyond on Displays should be at the user's terminal, or on a cen­ a continuing basis without any significant backlog. tral printer, at the user's option. Printed listings The key to performance and utility of the system is should include the user's mailing address, and the way in which data sets are loaded into the ar­ should be shipped by operations personnel during chives. The data must be rapidly available, and of the same shift. known quality. Considerable care must be taken in A data catalog should contain references to perti­ the loading process to remove duplication, to ac­ nent data sets held by archives other than those count for gaps in the data, to sort data into chrono­ within the Eos project. Methods for maintaining the logical order, to detect poor-quality data, and to an­ timeliness and accuracy of these references must be notate data whose veracity is questionable. The au­ established. It should be the responsibility of the Eos thority to purge any data from the archives should project to obtain information to be included in the reside exclusively within the Eos-sponsored scientific catalog, and to verify the accuracy of the catalog research community. contents. 2. With Eos scientific advisory committee ap­ proval, purge erroneous or outdated information or Document Storage and Retrieval information with no significant value. The archives will be dynamic; just as they must efficiently load The research community cannot effectively new data, so they must also efficiently deal with un­ utilize Eos data unless they have access to both open wanted data. and "gray" literature, containing pertinent technical and scientific information (e.g., project 3. Satellite swath data archives. Data coverage plans, instrument and algorithm documentation, will be either regional (e.g., West Coast A VHRR and validation results, data system documentation, data CZCS) or global. Swath data should be selectable by set descriptions, scientific papers). Therefore, the project, platform, instrument, level, version, pa­ Eos archival subsystem should provide an online rameter, attribute, time, and region. bibliography containing annotated references to all relevant published and unpublished literature de­ 4. Grid data archives. Coverage for Level 3 data sets could be regional (e.g., SSM/I polar grids) or rived from the project. Copies of unpublished pro­ ject documentation should be retained as a part of global. Selection of Level 3 data should be by project, the Eos data archives. platform, instrument, level, version, parameter, grid type, time, and region. A researcher should be able to search the biblio­ graphy by specifying any number of attributes, in­ 5. In situ data archives. Pertinent data from both cl uding author, publication data, subject category or fixed and moving platforms should enter the Eos ar­ sub-categories, instrument, parameter, or original chives. Data from moving platforms may not be well document number or citation. The bibliography sub­ organized spatially but should be stored in a sys­ system should respond by displaying the number of tematic manner. The storage method for these data entries found at each stage of specification. The user should depend on data volume and demand. Data se­ should then have the option of displaying a list of lection should be by project, platform, instrument, titles found, or complete citation, including level, version, parameter, time, and region.

8 6. Performance summaries archives. Data ac­ 5. Directory support only - directs user to a counting summaries and other small non-spatial specific non-Eos archival facility for further data sets should be maintained and distributed when in formation. requested. Summary data should be produced perio­ dically. These data sets should have a fixed structure All of these categories should be supported by the with no geographic dependence; thus, these data bibliography, but we expect that the level of support need only be referenced by the appropriate subset of (i.e., the thoroughness with which relevant citations project, platform, instrument, level, version, sum­ are selected, entered, or purged) will vary, according mary type, and time. The storage method for sum­ to the priority implied by this list. mary data will depend on data volume and demand. Performance and Accounting 7. Browse file archives. Browse files are com­ The Eos archival subsystem should meet suitable posed of reduced resolution data sets, should be pro­ performance goals required by the research commu­ cessed at least to Level 2, and have either regional or nity, specified by the project, and agreed to by Eos global coverage. Browse files are designed to provide investigators, at least during prime time (defined as a rapid response to a user wishing to locate specific 0800 Eastern to 1800 Pacific Time on weekdays). data interactively. These files should be optimized to Performance goals should specify system availabil­ deal with communications and remote terminallimi­ ity, data loading time, extraction time, data set gen­ tations and should be referable by project, platform, eration time, browse time (i.e., generation, transmis­ instrument, level, version, parameter, time, and sion, and display), and system response time. region. In order to monitor performance of the system and determine appropriate user charges, various ac­ 8. Maintain data on media that provide long counting factors should be measured and reported. lifetime, rapid and random access, and economical All required information should be recorded auto­ storage. Currently, the primary candidate medium is matically and reported in weekly and monthly sum­ high-performance digital optical disk. Optical disk maries. These data should include catalog usage, storage media eventually could replace existing mag­ bibliography usage, archival loading, data set ex­ netic tape archives. traction, subset generation, and general resource usage. A detailed invoice should be prepared and 9. Distribute data sets on optical disk (or other sent to the user with each data shipment. Summary appropriate media), or transmit data on a communi­ invoices should be prepared for each user on a cations link. As the Eos data base grows, it may monthly basis. In addition, online accounting become necessary to develop a high-speed communi­ algorithms should be available to allow researchers cations link to existing modeling facilities for the to estimate costs before requesting either data or transfer of Level 3 (and possibly Level 4) data sets. data processing services. Data Set Support Categories TELECOMMUNICATIONS The Eos archival subsystem should provide several categories of support for archival data sets. The telecommunications industry is in a state of The support category proposed for each data set rapid change. Deregulation of the industry, the should be selected by the Eos Project, based on growth of satellite and fiber optics high-volume guidance from the research community and cost/ communication facilities, and the transfer of com­ benefit considerations. The categories are: munications backbone switching and mUltiplexing equipment from analog to digital are causing major I. Online support - subset selection by time, changes in communications capabilities. Networks region, platform, instrument, and parameter for transfer of digital information are growing from the archives; full range of output; full rapidly. Any requirements report relies on a catalog support; background knowledge of what is feasible and read­ ily available. Because of the uncertainties in the 2. Offline support - subset selection by media telecommunications industry, and the long lead time volume only, from the archives copy; storage for Eos implementation, the telecommunications re­ media output only; full catalog support; quirements should be periodically reevaluated and updated as new technologies become readily 3. Ordering support - catalog support; ability to available. place an order through the catalog subsystem; orders handled manually; Downlink Communications 4. Catalog support only - limited information in Eos data rates are quite high in comparison to the catalog; no support for obtaining data; other Earth orbiting satellites. The expected average

9 data rate is over 70 megabits per second, with peak Real-Time, Quick-Look Data Access rates of over 400 megabits per second. NASA com­ munications commitments are for an average data The users identified as requiring access to real­ rate of 70 megabits per second, but will not include time, quick-look information need interactive image continuous channel availability. Manned mission processing and display capabilities. These indivi­ (i.e., the core Space Station) communications will duals may need some interactive image browse capa­ have a higher priority. In order to buffer the high bilities as well as the capability to reprocess further peak data rates and to guard against momentary and display the data interactively. Further process­ downlink channel outages, an onboard data storage ing of the data may be mission dependent; conse­ capability should be developed. This data storage quently, applications software modules need to be subsystem should be sized to prevent loss of data. easily added to the system. The user community re­ Data from one half orbit will be over 125 gigabytes. quiring access to real-time interactive quick-look Onboard data storage needs to temporarily store data will probably be limited to Eos instrument these high data volumes, at a minimum. Data from teams and selected active data base sites. Investi­ onboard storage devices should be packed to take gator processing requests should be completed full advantage of broad-band downlink transmis­ within the timeframe of one orbit. Communication sion capabilities. facilities are needed to link quick-look investigators with the repositories. Data rates between 56 kbps and 1.5 mbps will likely be required on this link. The Direct Broadcast Communications link should also be capable of handling multiple NOAA and its international clientele will require users simultaneously. separate data storage and relay systems that will "guarantee" data delivery from any NOAA opera­ Uplink Communications tional instruments that may share the Eos polar plat­ form. This system would likely function in a manner Many mission operations activities will be similar to current NOAA spacecraft, which have on­ planned and scheduled. However, there probably board storage and utilize burst transmissions over a will be occasions when instrument command deci­ receiving site. In addition, NOAA has international sions will need to be made within the time period of commitments to provide direct broadcast weather one orbit. The requisite communications system satellite data for remote regions. There are also a needs to be capable of quickly providing scientists large number of international land-applications data making these decisions with sufficient information. users who are equipped to receive Landsat data. This information includes, in addition to quick-look Consideration should be given to using these direct data sets, trade-off information for various instru­ broadcast systems for transmission of high­ ment configurations. Instrument team requests for resolution imaging spectrometer and synthetic aper­ configuration changes need to be included effec­ ture radar data when they are not being used to relay tively into the command and control decision chain, weather instrument data to NOAA facilities and and a mechanism should exist to arbitrate con flicting clientele. demands. The communications system needs to be functioning continuously, accommodating different Repository Communications for user communities. These configuration decisions Quick-Look Users should be in addition to automated real-time selec­ tion (within a priority set of options) of measure­ After relay by telemetry satellite, Eos data will ment sites for select instruments. first be delivered to NASA mission and instrument repositories. These data repositories are primarily to Eos Information Network Services support NASA mission and instrument team opera­ tions, with the data subsequently being transferred Eos data centers will make Eos-derived data ac­ to Eos data centers, active data bases, etc. The cessible by the greater scientific community. These transfer of data to these facilities could be either centers should have archival data bases and ad­ through electronic communication channels, or vanced data base management capabilities in addi­ through the physical transfer of disk, tape, or other tion to demand processing facilities. The Eos infor­ storage media. However, while the data is still in an mation network should provide the required com­ Eos repository, certain communication functions munications links to and within the scientific com­ are still needed for mission operations, quick-look munity. This network will require catalog and data sets, operational user data sets, and other prin­ browse attribute file, browse image file, and archival cipal investigator functions. The principal investi­ data set access, as well as the quick-look access gators will require quick-look data sets for instru­ previously described. It should also provide for elec­ ment command decisions, and for preliminary scien­ tronic communications between colleagues working tific analysis. on Eos-sponsored research.

IO Catalog and Browse Attribute File Access number of researchers using various communication rates, ranging from dial-up lines to high-speed links Interactive electronic catalog and ordering func­ of 1.5 mbps or more. Network protocols and stan­ tions should be available for Eos researchers with at least 9,600 bps dial-up capabilities. In the catalog, in dards may result from the adoption of standards set addition to location, time, instrument status, and by Space Station communications. Users with dif­ other available information, the attribute file should ferent protocol capabilities will require translator also be accessible for search purposes. An interactive boxes or packages at the gateways into the network. catalog system should provide response to most If extended to include local area network access, the search commands within I to 15 seconds. The system NASA Program Support Communications Network should be available on a continuous basis and be (PSCN) would be one example of the type of capabil­ sized to handle at least 100 simultaneous ity that would be required. The PSCN was originally investigators. envisioned to link NASA researchers together with phone access, packet switched data (9.6 kbps), cir­ Browse Image File Access cuit switched data (56 kbps), and computer network subsystem3 (l.5 mbps to 6.3 mbps). While the PSCN In addition to the attribute browse capability, the will link NASA facilities together with a computer system should allow for reduced-volume data network subsystem, it is not clear if it wiJI include browse. Because of data rate limitations of9,600 bps voice, packet switched, and circuit switched access communications links assumed for most users, elec­ or if it will include extensions into the university and tronic browse of image files may not be possible for international research communities to serve Eos re­ many users. This would necessitate the publication quirements. There are, however, numerous commer­ of an image browse catalog. In this event, it could in­ clude both the catalog and attribute numerical files, cial carriers available that provide the full spectrum which could be computer processed to locate images of communications requirements needed for an ef­ or sequences of images. fective and efficient Eos information network. A select number of users (e.g., those associated with active data base sites) should have available higher data rate communications lines and they DOCUMENTATION should be provided with interactive electronic image browse capabilities. Of particular importance are the Proper experimental documentation encom­ data between the present and the last published im­ passes more than the description of the hardware age browse file. This interactive image browse capa­ and its initial calibration. Documentation should bility could be used by quick-look investigators and also include any information pertinent to scientific research teams monitoring \in instrument as a pre­ interpretation of the data record produced by the ex­ check of ordered data sets. periment. Since Eos is conceived as a global, multi­ year Earth observational program, intercompari­ Access to Archival Data sons between measurements made at different times and locations is central to the scientific return. Con­ Even in the 1990s, many researchers will prob­ sequently, accurate and complete documentation is ably receive high volumes of data through the mails. imperative for the success of this program. Docu­ The majority of researchers can probably accept mentation is considered to be a vital part of the data electronic interactive ordering (i.e., order placement and acknowledgment) with mail delivery. Turn­ record and should be stored and maintained with the around time for orders should be consistent with same care as the data, per se. Some necessary ele­ mail service (i.e., a few days to a week at most). ments of this documentation are briefly ·described The system should also allow for delivery of data below. . electronically, since a few sites may have the high­ speed communications capability needed to receive Trace of Design, Fabrication and Testing of large-volume requests. Other researchers will have lower-speed communications, consistent with lower Hardware volumes of requests. The system should allow for all This information, which is routinely prepared by levels of archival retrievals to be sent over any com­ munication line tied to the system. This would also contractors, is rarely put into accessible form or kept include the use of low-speed links for interactive as part of the data archives. Eos data volumes will be catalog interrogation. so large that digital storage of text materials can easily be supported. All key personnel should be identified and all reports collected in the archives. Carrier Considerations Much of this material may never be used, but some The Eos information network will require com­ of it may prove crucial to understanding a particular munications facilities that tie together a large set of critical measurements.

11 Hardware Description identifying the reasons for and the source of the re­ quested change. Preparation of this document, including the above described information, should be included in the contractual obligations of instrument teams, in­ Bibliography strument principal investigators, and facility­ A list of all papers and reports published in the instrument Centers. It should provide a compact open literature and thought to be pertinent to the description of instrument specifications, including type of instrument or observation being made, cali­ the nature of physical variables measured, noise bration or sensitivity studies, and the precision and characteristics, internal processing, and coding of accuracy of the data should be included as part of the telemetry. Sufficient detail should be included to documentation. Consideration should be given to allow another group or individual to convert raw making the complete text of all unpublished docu­ telemetry signals into basic physical quantities, using mentation (i.e., the gray literature) electronically ancillary calibration information contained within available. the transmitted data record. Processing Algorithms Calibration If an instrument data record is to contain quan­ The procedures and results of all calibration ex­ tities derived from measurements (those quantities periments and tests should be reported in sufficient obtained directly from the telemetry by application detail for other scientists to evaluate their of calibration relations), then the methodology thoroughness and accuracy. All approximations and utilized must be thoroughly explained and all rele­ instrument uncertainties must be identified. Instru­ vant references to literature concerning the method­ ment performance over a complete range of environ­ ology should be listed. All processing software mental conditions should be evaluated. This infor­ should be preserved and documented, along with a mation should allow conversion of telemetry signals report of design and testing rationale used by the to standard physical units of measure. Calibration software creator. This requirement also applies to standards should be traceable to the National processing software used to locate the observations Bureau of Standards; this requires documenting the and calculate observational geometry parameters. history of Eos data records. Correlative Data Record Calibration History Often routine processing procedures utilize addi­ tional data or information to determine derived In addition to documenting calibration tests, a parameters. Although these data may describe sim­ specific time sequence data record should be created ple or well-known quantities (e.g., topography), the from information used to monitor instrument cali­ bration over the course of an experiment. In doing particular version of the information used may dif­ fer slightly from other versions. Therefore, all such so, drifts in instrument characteristics can be data should be considered part of the archives. reconstructed. Both the calibration and calibration history files should have the provision to hold more than one calibration test result or more than one type Data Usage Trace of monitoring information; all sources of calibration Institutions responsible for maintaining Eos data information must be identified and placed in the archives should provide a trace of user access to the archives. data, allowing other investigators to contact col­ leagues familiar with a particular data set's charac­ Command History teristics. The trace should contain specific informa­ tion about the data ordered and the name and ad­ Automated command sequence construction, dress of the individual accessing the data. together with the necessary computer involvement in The above list of documentation is very exacting routine spacecraft communications (i.e., the com­ and will demand significant effort to acquire and mands given every instrument together with its maintain. Notwithstanding, given the requisite status, indicated in housekeeping telemetry) should resources envisioned for Eos and the large invest­ be available in mission operations computers; this ment of time and money it will require, this docu­ information is rarely retained. Provision should be mentation effort appears trivial by comparison. made within Eos data archives to create a data record documenting the history and providing a trace of commands and instrument status throughout the lifetime of the project. Significant reconfigurations NON-Eos DATA AND MODELS of instruments (e.g., gain setting changes) or observ­ Most, if not all, of the research scenarios (vis. ing sequences should be accompanied by log entries Appendix I) have some requirements for access to

12 non-Eos data. In some cases, access to models (e.g., dards, conventions, or guidelines that Eos accepts or global circulation models), or modeling capabilities, develops. Because Eos will be a primary researcher will be needed to place Eos data in context with some interface with the Space Station Program, its con­ physical or chemical process. While these external sistency of operation (i.e., the adherenee to stan­ data, models, and modeling mainframes may not be dards) and its flexibility (i.e., freedom from stan­ a part of Eos as a space-platform mission, they are dards) will weigh heavily in its acceptance by the part of the Eos scientific requirements. Hence, a research community. Unless standards are perceived commitment by Eos to access external, non-project by the user to be beneficial, they will be viewed as data sets and models should be made. Model results overly restrictive and onerous. Thus, early decisions should be accessible and, in addition, researchers should be made, minimizing the set of standards to should be able to request model runs through an Eos be invoked while maximizing their utility. Three fac­ data and information system. tors should be considered before Eos adopts any par­ There are six possible levels of commitment to ex­ ticular standard: ternal archives access. They range from accessing a 1. Historical data from a suite of spaceborne and directory of catalogs, to instant, remote access to the in situ instruments will be used for intercom­ world's data bases. The six levels are: parisons by Eos researchers. These data reside 1. Directory of catalogs in numerous archives, are in a variety of for­ 2. Information on specific catalog access mats, have varying resolutions, are of differ­ routines ing quality, and have a variety of appended ancillary information. Hence, these data will 3. Ability to place an order through Eos require resampling on space-time scales used 4. Direct access to other data bases through Eos for Eos data. 5. Direct orders and near-real-time access to ex­ Common data set organization (e.g., spatial ternal data bases through Eos resolution, map projection) and the use of 6. Direct order and real-time access to data logical catalog and data structures (allowing banks through Eos. various computers to read the files) will minimize the reprocessing task. Common fac­ It would be highly desirable if Eos could provide tors such as these should be studied to provide services through level 5. The user could locate a guidance in selecting standards that will min­ specific non-project data base, order that data imize reprocessing at a future time. through Eos, and have it arrive along with his Eos ar­ chival data in a compatible file format. A single soft­ 2. Eos will be operating in a computer environ­ ware routine based on this common format would ment largely determined by developments in then be sufficient to input all of the data necessary the commercial sector (i.e., by software and for research. hardware manufacturers). We anticipate that Eos data and information system require­ ments will parallel those that focus commer­ cial developments (e.g., nested catalog struc­ STANDARDS tures, advanced data base management capa­ bilities, super micro- and mainframe com­ The Eos project will be operating within the same puters). Since system costs would be mini­ timeframe as the Space Station project, which is mized by adopting commercially available designing a Space Station Information System. It, in products, Eos requirements should be tailored turn, will operate under the auspices of the NASA Office of Space Science and Applications, which is and adapted to these product developments wherever feasible. defining a Science Applications Information Sys­ tem. Each of these will impose some standards on its 3. The computer and communications com­ components. Space Station personnel have discussed munity is moving toward standard practices standards for the operating system, programming that will be both machine- and data­ language, software support environment, data base independent. These practices will allow data management systems, and the use of standard for­ and algorithms to be readily interchanged matted data units. There are potential benefits to even on future systems. Any standards these interconnected systems ifthe various standards adopted under the Eos data and information adopted by such top-level entities as the Space Sta­ system auspices should take these practices in­ tion Information System are acceptable to, and ac­ to account. cepted by, Eos in common or overlapping areas. This will require that Eos project personnel and Areas of Consideration researchers be familiar with, and preferably par­ ticipate in, the definition of these standards. Areas of direct researcher and system interaction Careful consideration must be given to the stan- will include directory and catalog access, query and

13 browse functions, archives access, algorithm de­ 4. Providing coding standards to accommodate velopment, and data interchange. Given the rapid a variety of information and attributes. rate of data base management developments, it is probably inappropriate to attempt to standardize Glossary and Definitions specific data base structures, catalog organizations, query procedures, etc. Rather, a more productive ef­ In the multiple-system, multidisciplinary en­ fort will concentrate on standardizing the tools with vironment of Eos, common understanding of terms which to describe and manipulate these data. Thus, is a necessity. This will require the adoption or appropriate items for standards include data defini­ development of a standard glossary containing tion languages (rather than a specific data structure) operational definitions. This should include tech­ or data system model techniques (rather than defin­ niques as well as definit ions, and include such items ing a specific data model). This approach also recog­ as the data definition language and data modeling nizes the existence of numerous heterogeneous data techniques. It would also be appropriate for the bases and their catalogs, which must be dealt with by Eos. Several NASA Pilot Data Systems are consider­ glossary to list applicable external standards (e.g., ing the creation of uniform catalogs (i.e., uniform Federal Information Processing Standards, information presentation for the researcher) in con­ American National Standards Institute, Con­ trast to standard catalogs. sultative Committee for Space Data Systems, Inter­ Systems are usually described in block diagrams national Standards Organization). at various levels of detail. In such a description, the critical items are the functions to be performed by Algorithm Interchange each block (i.e., its transfer function) and the characteristics of the paths between the block inter­ Much has been said about sharing of algorithms faces. Unless some overriding factor dictates stan­ among scientific researchers. This has been ham­ dards within a block, the appropriate items needed pered in the past by the use of different operating for complete descriptions, and perhaps standards, systems, different programming languages, and dif­ are the functions and interfaces themselves. ferent execution procedures. While it is unreason­ Beyond functions and interfaces, careful con­ able to expect the entire research community to stan­ sideration should be given to standard format struc­ dardize on systems and languages, it is reasonable to tures for data interchange. While they may not ex­ consider a standard interface executive. This would plicitly cover data storage formats, there is a close allow the independent development of algorithms relationship between storage, interchange, and func­ and executives and provide a method for easy inter­ tional formats. Standard formats could serve several change between investigators. purposes; these include: 1. Providing a data organizational structure that Data Interchange Formats will accommodate all types of geographically related spatial and non-spatial data (i.e., sca­ An oft-voiced problem is that of data inter­ lar, polygon, and raster data), features and at­ change, particularly in light of the in finite variety of tributes, and necessary topological relations. formats that are in use today. Several studies are cur­ While it may be desirable to define a single for­ rently underway, aimed at providing a standard for­ mat, the diversity of data types suggests that mat structure: this is inappropriate; I. Potential worldwide data transmission will be 2. Providing a format volume record grouping a feature of future systems. The International Stan­ capable of accommodating all necessary data dards Organization (ISO) has formulated a series of in variously formatted records (i.e., feature standards expected to facilitate international elec­ identification, data quality, spatial data type, tronic message interchange. In addition, the Con­ locational definitions, spatial relationships, sultative Committee for Space Data Systems and ancillary data). These format structures (CCSDS) has been developing a set of recommenda­ should be expandable to allow addition of tions for special purpose techniques that are tailored future types of data; to the unique environment of exchanging data through space data channels. ISO's DIS 821 I stan­ 3. Providing an interchange format that will dard proposes a data definition language, and allow a researcher to read the data set, deter­ CCSDS proposes a number of data formatting stan­ mining the basic logical structure. A family of dards that may be suitable for Eos consideration. format structures necessitates conveying to the receiving computer the logical structure of 2. Two national committees are currently in­ the data being presented; this is the purpose of vestigating the interchange format question, one a data definition language; convened under the U.S. Geological Survey (to unify

14 data representations from Federal agencies) and the structures, system transparency to the researcher re­ other from the National Bureau of Standards via the mains an overriding concern. This must be the guid­ U.S. Geological Survey and the American Congress ing principle in establishing standards and format on Surveying and Mapping (concerned with digital for the Earth Observing System data and informa­ cartographic data standards). The National Com­ tion system. mittee on Digital Cartographic Data Standards is charged with producing a format structure recom­ mendation for the cartographic community before SUMMARY 1986, with eventual consideration for inclusion The foregoing discussion delineates a significant within Federal Information Processing Standards. number of requirements, attributes, characteristics, Careful consideration should be given to working and capabilities that we believe should be accom­ with these national committees and subsequent modated within the Eos data and information sys­ adoption of the standards that they recommend. tem. These factors can be conveniently grouped into 3. NASA is currently sponsoring a study of stan­ seven interrelated and interdependent functional dard formatted data units. A standard formatted categories and are summarized in Appendix II. data unit is a conceptual data object that could be These functional groupings are Eos flight sys­ transmitted between users. It will consist basically of tems, user functions, operational functions, infor­ a formatted and labeled data set; thus, it defines an mation service, advanced data base management, interchange format. It will include a primary label data processing, and non-Eos data base functions. that serves as a global identifier, a set of secondary Eos is unique in that we expect that many of the re­ labels that carry information about the data, and the quirements cannot be accommodated by any single data set itself. It currently focuses on the record functional element or group. Rather, we anticipate level, but may also provide a nesting feature that that many functional elements will need to coor­ would allow a given unit to be composed of multiple dinate their activities to accommodate a given task. standard formatted data unit subsets. The interrelationships and interdependence of the elements within the resulting data and information Clearly, the area of standards and formats needs system is thus quite significant. Consequently, Eos considerable attention by Eos and, in particular, information services functions will be central to en­ consideration in concert with the research commu­ suring straightforward, transparent intra- and inter­ nity who must deal with the resultant standards. Since system interaction, a prime characteristic derived we envision the necessity for an entire suite of format from these researcher-imposed dependencies.

15 III. ARCHITECTURAL CONSIDERATIONS

In this chapter, we propose appropriate design plex system into easily comprehensible modules with principles and consider an Eos data and information clear "strata" in which common data-handling system for the 1990s, the Space Station era. We pre­ functions reside. Strata exchange data according to sent an example of a functional architectural design, well-defined rules and are thus predictable in terms described in terms of elemental composition, top­ of their functional characteristics. Layering is essen­ level functions, and internal and external interfaces. tial for a data system that: This conceptual arc.hitecture illustrates one possible configuration that could satisfy the general data 1. is understandable by a wide range of users and system requirements that are developed in this implementers; report. To ensure that the designs considered were ap­ 2. can expand or contract easily to accommodate propriate to the Eos data and information system, changing user requirements or technological we categorized the requirements, characteristics, developments; and attributes delineated in this report into seven 3. can be tested in a modular fashion by maxi­ functional groupings (cf. Appendix 11). These mizing the functional independence between groupings were then compared to each functional processes residing in different system layers; element of the architectural examples. We believe that the flexibility and modularity of the conceptual 4. displays simple, well-defined system inter­ design presented in this chapter is well suited to these faces that are not constantly changing; requirements and the needs of Eos. Since the functional design of data transport fa­ 5. can adaptively respond to dynamic mission cilities and services is a key and fundamental factor, events; we have selected an architectural example based on space data communication standards developed by 6. facilitates recursive use of replicate hardware the Consultative Committee for Space Data Systems and software elements to lower system de­ and utilizing International Standards Organization velopment and operational costs; and general purpose protocols for transparent inter­ change between dissimilar system elements. We have 7. avoids the need for major advances in tech­ examined this design in sufficient detail to ensure nology by permitting complex tasks to be soundness of approach and methodology. As with broken into pieces that can be handled within any initial concept, this design is open to construc­ existing capabilities (e.g., parallel processing tive criticism, revision, and improvement. A more of very high-rate data streams). detailed description of a system architecture is beyond the scope of this report. Standard Data Structures and Data Autonomy DESIGN PRINCIPLES The second key architectural design principle is standardization of formats and protocols through Designing a data and information system for Eos which data are exchanged between distributed ele­ is clearly a complex problem, particularly with re­ ments of the system. Standard structures promote spect to data handling. To ensure that the resultant "data autonomy," the transport of independent, system can meet scientific needs, the architectural user-defined units of data through the system. Using concept should feature two fundamental principles this principle, the internal format and content of the or design techniques that we believe should be data are defined by the user and transparent to the employed throughout: layering and standardization. communications system. Together, they can be used to create an information For the Eos data and information system to pro­ system that is flexible, transparent, and robust, pro­ vide efficient, high-performance data services that viding the foundation for diverse data processing handle a diverse combination of data types and re­ and archival needs within an Eos data and informa­ quirements, it should be adaptively responsive to the tion system. data per se. The concept of data autonomy (encapsu­ lation of variable data within standard, network­ Layering and Modularity interpretable labels) provides a foundation for these adaptive data services. Data autonomy is achieved in The first architectural principle to be applied this architectural example by employing standards throughout the design of an Eos data and informa­ that require each data source (e.g., engineering sub­ tion system is layering, the technique of dividing and system, instrument) to encapsulate its data messages conquering. Layering breaks the diverse and com- into "source packets" having globally interpretable

16 labels that define the source and destination, class of As the data system further evolves, spacecraft service, priorities and delivery conditions, and pro­ and ground-based data handling and processing vide the information required for verification, vali­ operations may become more independent. The sys­ dation, and accounting of data within the packet. tem's data communications and processing capabil­ Thus, while the packet format is a standard for the ities will need to expand in a controlled, evolutionary network, the format of its contents can be variable; fashion to support growth. The evolving character the data exchange network need only transmit what of the environment in which the system operates re­ it is given. quires that the architecture display a high degree of modularity and structure. Adaptive flexibility must become a cornerstone upon which the system evolves. Adaptive Flexibility We envision an Eos data and information system Distributed System Considerations that is dynamic, evolving to meet the needs of its When the transmitted data stream is received at changing clientele and advantageously employing ground-based facilities, it will be distributed to a new technological developments. Layering, modu­ confederation of geographically dispersed data pro­ larity, structures, and autonomy are hallmarks of cessing and archival facilities. We anticipate that contemporary system design. Together, they provide these facilities will be interconnected by both private ameans through which the resultant data system can (e.g., NASCOM) and shared public (e.g., X.25- . evolve adaptively to meet these changing needs. based) communications networks and private and A data and information system design utilizing public broadcast satellites. standard data structures affords a significant level of System facilities may be distributed worldwide, flexibility; its data services are readily adapted to and will provide specialized data handling, process­ changes in internal data format and data flow or ing, and archival services in support of a large, and routing without changing the system architecture. not yet fully defined, cadre of users. The breadth and These structures should be deliberately designed for diversity of the user community suggests that the Eos flexibility through the provision of features such as data and information system should feature a spec­ secondary headers, and they should have the flexibil­ trum of format and protocol standards for data in­ ity of allowing future versions to be defined as terchange. Application-oriented standards are es­ needed. sential to facilitate the early development of a dynamically adaptive, distributed data system that can sustain planned evolution. DESIGN CONSIDERATIONS The majority of researchers will also have chang­ ing requirements for routine data handling services The Eos data and information system must be such as accounting, merging, grid overlay, map pro­ designed, implemented, and operated affordably, in jection, ancillary data processing, temporary stor­ an environment of almost constant technological age, and retrieval. These are changes that the data and researcher-initiated change. Some of the re­ and information system must efficiently quirements and constraints imposed by this environ­ accommodate. ment are now examined and their ramifications im­ posed upon subsequent designs. Data Types and Rates The data system will be required to transport and Evolution deliver a diversity of digital instrument and engineer­ ing data through bidirectional TDRSS (Telecom­ A key architectural consideration for the data munications and Data Relay Satellite System) com­ system is its projected lifecycle, which will encom­ munications links. As a result, a spectrum of perfor­ pass the development and operation of a very com­ mance requirements will exist, including: (i) raster plex, dynamic system that must accommodate data that can tolerate delivery delays but not data growth and evolution of spacecraft instrumentation, outages (it is moderately tolerant of data errors when technology, and user requirements. At the outset, uncompressed, but intolerant when compressed); (ii) performance requirements imposed on the system other digital data and data transport processes, in­ will push data handling technology to the limit. cluding low- and medium-rate instrument telemetry, Subsequently, data handling capabilities must engineering and housekeeping data, ancillary data, evolve to support a maturing platform that carries an data base transfers, command sequences, memory increasing number of instruments with growing re­ loads, and text and graphics, that have differing and quirements for data storage, processing, and trans­ often conflicting requirements (some, such as mission capacity. System technology will become programs and data base transfers, are intolerant of obsolete quite rapidly unless the system design ac­ any data errors or outages, while others are tolerant commodates changes. of even the poorest communications).

17 The wide diversity of user data requirements sug­ Data Interchange Standards gests that the Eos data and information system could Within the framework of the layered concept of utilize several different classes of data transport ser­ "Open System Interconnection," the International vice, each class displaying a well-defined quality of service that includes clear specifications of data rate, Standards Organization (ISO) is currently develop­ ing a broad spectrum of commercially supported error rate, delay, sequential character, and com­ general purpose data protocols. These protocols are pleteness. The data system must also provide certain designed to provide transparent interchange between value-added utility services for user data streams dissimilar elements within an open, worldwide com­ such as merging, sorting, storage, and remote access. munications system. Emerging ISO standards will likely have direct application to the Eos data and in­ TDRSS Compatibility formation system, although some of the standards All types of Eos digital data, each with its own are relatively unattractive for specialized data ex­ service requirements, are merged into a composite change through either capacity- or bandwidth­ data stream that flows across TDRSS transmission limited space data channels. paths. TDRSS services include uplink and downlink The Consultative Committee for Space Data Sys­ transmission on S-band and K-band data channels. tems (CCSDS) has been developing a set of recom­ A 300 Mbps K-band single access service, using I and mendations for special purpose techniques that are Q channels, will provide the dominant downlink tailored to the unique environment of exchanging data transport service for platform scientific and data through space-based data channels. These operational data. The S-band single access channel recommendations supplement ISO standards in will primarily carry platform engineering and some those specific areas that are unique to space mis­ real-time operational data. sions. Many CCSDS standard data link protocols Initially, data system downlink services will need are directly applicable to the Eos data and informa­ to accommodate a composite data rate that periodi­ tion system, although some, particularly space-to­ cally approaches (and even transiently exceeds) the space links, may require adaptation. present TDRSS limit of 300 Mbps. The uplink data In support of Eos scientific research, the data service will be provided on the single-access S-band and information system will need to exchange heter­ channel at 100 kbps when scheduled. It is important ogeneous data sets not only among its different to note that TDRSS has the capability to com­ elements but also with other systems both internal municate with the Eos platform at any time via the and external to NASA. The use of standard for­ multi-access forward link, but realistic operational matted data units will enable common services support considerations will probably limit the use of within and interchange of these data among various this uplink to emergency situations. These com­ data systems. munications services must transparently support in­ Taken together, the ISO and CCSDS standard stantaneous, dynamic, and adaptive changes in the protocols form the basis for standardization, hence blend and volume of data that flows through the data autonomy, within the architectural example of link. an Eos data and information system considered here. Direct Downlink, Broadcast, and Uplink It is possible that Eos research instrumentation and NOAA operational instruments will reside to­ FUNCTIONAL ARCHITECTURE: gether on the same polar platform(s). NOAA opera­ TOP LEVEL tions will likely require a direct downlink indepen­ The Eos data and information system should dent of TDRSS. The data and information system provide its users with the services of a complete in­ should accommodate this enhanced capability with formation system, including the bidirectional com­ minimal system impact. munications capabilities required for transparently NOAA operations in support of international transferring many divers"e types of data between the weather services require that some onboard pro­ various ground and space-borne elements of the sys­ cessed data be broadcast from the platform to tem. Consequently, it will be necessary not only to ground receiving stations. These transmissions procure and maintain certain physical elements of would only contain real-time data and would not in­ the system, but also to provide an interface with and clude previously recorded information. Relay use of facilities, utilities, and data provided by other broadcast services could also be accommodated for NASA and non-NASA organizations in a manner search and rescue activities. that is transparent to the researcher. Direct uplink capabilities would include only relay broadcast and data collection activities. Instru­ Overview ment and platform command functions would be handled through mission operations via the TDRSS A simplified functional diagram of a distributed uplink. data and information system concept is shown in

18 Figure 1. This architectural example has been ex­ Management and Communications Network), amined in some detail to ensure that it meets the re­ through which commands will be sent and data quirements, characteristics, and attributes de­ received from spacecraft instrumentation. Similarly, veloped in this report. It is important to note that this position and timing information from the Global example describes a distributed data system in which Positioning System can be appended to the data the same functions (e.g., acquisition planning, streams via Interface Units. analysis, storing, and cataloging of data) may be ex­ On the Eos platform, Interface Units can playa ecuted on different data sets at one location or the key role in system resource management. With a same data sets at different locations. These functions large user complement, it is likely that conflicts will or processes and their results must be coordinated, arise when users simultaneously request resources either onboard or on the ground. Exactly where and (Le., power, thermal, pointing, communications, how this coordination takes place will be a function etc.) that exceed those available. Rather than at­ of the level of intelligence that is built into the on­ tempting to check every request centrally, a dis­ board and ground-based elements of the systems. No tributed resource management system can be im­ attempt has been made to detail all of the functional plemented using Interface Units. A set of software­ capabilities included within a given system element. controlled management services (network interfac­ Rather, a few key functions are highlighted for each ing, actuator interlocking, power limiting, com­ and the reader is referred to Chapter II and Appen­ mand verification and validation checking, status dix II for a more detailed accounting. monitoring, fault containment and isolation pro­ Two key elements of the system are the Interface cedures, etc.) could be downloaded to distributed In­ Unit and the Data Management and Communica­ terface Units for execution. tions Network. The Interface Unit includes a pro­ With the exception of direct broadcast access to cessor and memory, and provides the means together polar orbiting platforms by users with receiving sta­ with telementry and telecommand capture and dis­ tions, the user interface to the Eos data and informa­ patch systems (imbedded within the Data tion system will be the same, regardless of

DATA MANAGEMENT AND COMMUNICATIONS NETWORK

DIRECT BROADCAST

DATA MANAGEMENT AND COMMUNICATIONS NETWORK

Figure 1. An example of a data and information system architecture suitable for Eos.

19 geographic location. Operations-oriented data will ment and engineering data to the Mission Operations flow through its own ground station(s) and Eos data Center for quick-look analysis by investigators; (iii) will flow through TDRSS. The types of data flowing high-rate instrument and engineering data and to various Eos facilities and the processing capabili­ subsets of platform engineering data directly to ties resident at these locations are implementation dedicated Instrument Operations Center(s); (iv) questions requiring systems trade-off analysis. The moderate and low-rate instrument and engineering generic architecture considered here does not con­ data and subsets of platform engineering data to ap­ strain the physical location of personnel, data, or propriate Instrument Operations or Mission Opera­ processing capabilities. It is important to note that tions Centers. the use of data standards enables ground data pro­ cessing facilities to upgrade their capabilities and Operational Facilities transfer functions to different elements without ma­ jor perturbations to the entire system. Operational Ground Stations can acquire data A user's low to medium data rate instrument or directly from the spacecraft in a manner similar to engineering subsystem will have an interface with the current NOAA practices. Assuming operational in­ Eos data system Local Area Network via a standard struments are resident on the Eos platform, these Interface Unit, or via the operating system services Stations provide transparent user access to their of an onboard data processor that supports instru­ instruments. ment operations. A source packet will be the stan­ All data flowing through an Operational Ground dard data structure at the Interface Unit. The Inter­ Station will be transmitted to an Operational Pro­ face Unit provides the physical interface to the local cessing Center for reduction and creation of direc­ onboard data network and its protocol, a packet tory and catalog entries. These data will be accessible transfer frame. High data rate instruments (e.g., to other system elements linked to the Network. High Resolution Imaging Spectrometer, Synthetic Aperture Radar) will format their packet data direct­ Instrument Operations Centers ly into transfer frames, which can be switched direct­ ly into dedicated "virtual channels" in the downlink There are two types of Instrument Operations stream (a virtual channel is created when a broad­ Centers envisioned, (i) those that are dedicated to band link is subdivided into two or more parallel specific high data-rate instruments, and (ii) those paths operating quasiautonomously). These instru­ designated for groupings of low to moderate data­ ments can also exchange low-rate source packets via rate instruments. the Network for adaptively optimizing data We anticipate that all unprocessed data would be acquisition. stored at the White Sands receiving center for a two­ At the downlink receiving site, the high-rate day period. During this time period, the data would serial frame stream is immediately split into parallel necessarily have to be transmitted, received, and virtual channels, and selected frames are routed to acknowledged by the appropriate Instrument Opera­ appropriate lower-rate data handlers such as de­ tions Center. coders, packet extractors, data capture devices, etc. Processing quick-look, Level lA, and higher­ Closed-loop retransmission protocols will be im­ level data, and generating catalog, directory, and plemented for certain virtual channels carrying browse file entries, would be performed at the ap­ critical data that cannot tolerate data outages. The propriate receiving Instrument Operations Center. elements of protocol required to execute closed-loop This data and information would then be forwarded command operation procedures may be placed in a to an appropriate Data Center for long-term archival secondary header of the telemetry transfer frames storage, catalog and browse file maintenance, and assigned to those virtual channels that use retrans­ dissemination. mission techniques. Requests for observational sequences, com­ The use of virtual channels in a telemetry frame mands, and command sequences generated by in­ allows a single 300 Mbps data stream to be immedi­ strument teams or associated investigators would be ately split into many lower-rate parallel paths. Thus, transmitted to the Mission Operations Center main­ processing functions such as decoding can be done at taining a master schedule. data rates that are well within the capabilities of ex­ isting technology. Bulk data streams can be readily Mission Operations Center sent to dedicated real-time processors or a storage medium for immediate reception and possibly non­ The Mission Operations Center supports plan­ electronic transfer to a user. ning and execution of instrument operations, result­ Data flowing through TDRSS will have Level 0 ing in the introduction of new Eos data into the sys­ processing performed at a ground data handling tem. Through Mission Operations, a researcher's in­ center (presumably located at White Sands) before strument request will be formulated into a command transmitting: (i) platform engineering data to the sequence and forwarded to the spacecraft. Should Mission Operations Center; (ii) subsets of the instru- the need arise, the Mission Operations Center should

20 provide rapid response for instrument teams or re­ vides for a complete bidirectional interface between searchers requesting command control of an instru­ any number of archives and the Data Management & ment from remote locations. The Mission Oper­ Communications Network. Thus, researchers ac­ ations Center, like the Instrument Operations cessing the system through any other component will Centers, will maintain software-supported planning have transparent access to pertinent non-Eos aids, employing menus or other "user-seductive" holdings. Similarly, Eos researchers working techniques. This software will assist researchers in through the auspices of these non-Eos Data Bases planning and scheduling of observational sequences could have full access to all system services. by Eos instruments. Data Management and Data Centers Communications Network Currently, it is anticipated that there will be a The Data Management and Communications lead Data Center that is responsible for algorithm Network will link together all other system elements development and maintenance; coordination of and subsystems, in a manner that is transparent to a overall data system activities; development, distribu­ user. Its goal is to allow researchers to conduct mis­ tion, and maintenance of various technologies; net­ sion planning, request and receive data, interrogate work management; and maintenance of data ar­ directories and catalogs, and browse data sets re­ chives, the directory, and its own catalog. Data motely and interactively. The success of Eos is directory and catalog entries will be forwarded to therefore directly tied to the effectiveness of this in­ and maintained by this lead Data Center. The direc­ formation Network. tory will be more comprehensive than the Eos data The Network should be managed by Data Center per se; it will include entries from other non-Eos data personnel, thereby ensuring that the Data Centers bases pertinent to Eos scientific objectives. are involved directly in this key activity. The perfor­ Since we recommend that processing beyond mance of the Network should be evaluated periodi­ Level lA be performed on demand, it is anticipated cally by personnel who lead activities of instrument that this processing and eventual scientific analysis teams and Active Data Bases. Thus, since the selec­ could take place in both Instrument Operations and tion of Active Data Bases and instrument teams will Data Centers. The resulting data sets will have direc­ be governed by the scientific community through the tory, catalog, and browse file entries developed at peer-review process, those selected will have a their processing location. The resultant data would significant vested interest and will act to ensure the be entered into an appropriate Data Center for ar­ utility of Data Centers and the Data Management chival storage, catalog, and browse file mainte­ and Communications Network through direct over­ nance, and distribution. sight responsibilities. Data Centers will be characterized by a staffhav­ ing scientific expertise in one or more disciplines, the location of archives of Level lA and higher-level Transparent Data Handling data sets, the capability for scientific data processing The scientific and operational environment en­ and analysis, maintenance of data catalogs and visioned for the Eos era (viz. Chapter II) suggests browse file entries for all data resident at the Center, quite strongly that transparency in data handling will and above all, the ability to retrieve and distribute ar­ be a key factor and fundamental attribute of an Eos chival data and information. data and information system. In this example, the functional configuration is based on standard proto­ Active Data Bases cols that support the integrated transmission of Active Data Bases will be facilities where fo­ many types of user data (e.g., instrument and cused, extended scientific analyses are conducted engineering data, computer memory exchange, text with archival data derived from a Data or Instru­ and graphics) through common, limited-capacity ment Operations Center. Value-added processing data channels. It is also compatible with commer­ (e.g., data reduction) will be performed at these sites cially supported terrestrial standards for open sys­ and the resulting data, together with directory, tem interconnection. These characteristics are catalog, and browse file entries, will be transmitted achieved by using two CCSDS standard application to an appropriate Data Center for long-term storage data structures (the source packet, and the standard and distribution. Additionally, directory and formatted data unit) and two CCSDS data link struc­ catalog entries for these data sets will be generated tures (the telemetry transfer frame and the Reed­ and transmitted to the lead Data Center. Solomon telemetry codeblock) throughout the system. Non-Eos Data Bases Data entering the system utilizes either a source packet, or the standard format data unit (that con­ Many Eos scientific objectives require access to tains either sets of raw packets, or processed results). non-Eos data bases. This architectural example pro- The format for transmitting data bidirectionally

21 through TDRSS links is a telemetry transfer frame, code, all of the Reed-Solomon parity bits are simply that mayor may not be encoded using the Reed­ appended to the end of the frame. Therefore, some Solomon error-correction algorithm. frames can be transmitted with the Reed-Solomon Selection of a telemetry transfer frame as the parity bits attached (permitting virtually perfect­ standard data link structure for bidirectional use on quality frame data to be received after decoding), data channels has significant ramifications. The while others can be tranmitted without Reed­ frame is optimized for efficient channel utilization. Solomon protection (in which case a frame will have It is a fixed-length data structure that has a the TDRSS channel bit error rate of lxlO- 5 ). Thus "natural" size of 10,080 bits, a convenient quantity by selectively encoding some frames, and transmit­ of data to be handling on links operating at multi­ ting others uncoded (saving 15 percent coding over­ megabit rates. Additionally, it is organized around head), different data qualities can be provided the concept of virtual channels, that provides a for dissimilar data types being transmitted through a mechanism for segregating different types of data, common channel. transmitted serially through a common data chan­ nel, into several logically parallel paths. This permits a high-rate serial stream to be immediately split into FUNCTIONAL ARCHITECTURE: many parallel, lower-rate data handling processes at PEER LAYERS a receiving facility (using relatively unsophisticated hardware). This significantly reduces requirements for developing high-rate processing technology. Redrawn (Figure 2), the architectural schematic The telemetry transfer frame is optimized to fit reveals the "peer" layers of data handling. The stan­ within a high-performance Reed-Solomon code­ dard protocol data units, which provide the mech­ block structure. Because this is a block-oriented anism for data exchange across the layers, are shown

Platform Platform Ground user Ground user telemetry telecommand telemetry telecommand processing processing t processing processing SFDU Platform PACKETS Ground system support data TIME CODE support data processing processing

____ I "" __ l ___ --- ~---- ~---- t------~--

Platform DATAt Ground system 0....- local area ~ NETWORK I-- local/wide area t-- networking STANDARDS networking ------~a~fo:------i-----::~-sy~e~----- data capture data capture and dispatch TRANSFER FRAME and dispatch AND CODING STANDARDS

TDRSS TDRSS Radio Radio frequency frequency gateway gateway to ground to platform

SPACE GROUND COMPONENT------~·I~(~------COMPONENTI ~I Figure 2. Peer protocols within the data system example.

22 as well. Although for brevity, the following discus­ by other spacecraft. Within the Eos system, the up­ sion is limited to the space-to-ground data transfer link and downlink data streams are similar; both link, it is important to note that identical, mirror­ contain low- to medium-rate instrument and engi­ image techniques are used on the uplink path and are neering data and computer-to-computer data ex­ applicable to both sides of the ground-based net­ change, and the physical characteristics of the uplink work. This self-similarity in data handling metho­ and downlink are virtually identical. Data retrans­ dology, employed throughout the architectural ex­ mission protocols known as command operation ample, is easily seen. procedures can be incorporated to provide fidelity The concept used here of transmitting bidirec­ and highly reliable delivery for both uplink and tional data streams through space-to-ground links is downlink data traffic that cannot tolerate data slightly different than conventional techniques used outages (e.g. data base transfers, memory loads).

23 IV. MANAGEMENT CONSIDERATIONS

There have been several recent reports published fate of data sets acquired by these projects is gener­ that deal with various aspects of data management ally to fade rapidly from memory. Reasons for this problems in the space sciences. Three of these re­ problem include: ports are particularly apropos and should be care­ 1. Lack of useful documentation. Changes of fully studied and followed by NASA management as personnel or dissolution of groups cause loss the Eos data and information system evolves. These of knowledge of instruments and data charac­ reports are the National Academy of Sciences Space teristics, of calibration and processing proce­ Science Board, Volume 1: Issues and Recommenda­ dures, and of data formats. tions (1982)2 and Volume 2: Space Science Data Management Units in the 1980s and 1990s~ and 2. Lack of planning for hardware or personnel Space Research Data Management in the National changes. Data quality monitoring procedures Aeronautics and Space Administration'. are often nonexistent and necessary software These reports examine many features of past changes are not controlled or documented data management practices and lay a firm founda­ adequately. tion for minimizing problems with future systems. 3. Perception of data archives as passive Volume 1 from the Academy'S Committee on Data storehouses that at best only dispense data. Management and Computation develops a set of Little that is learned about the data is returned seven principles for successful scientific data to the archives, hence data value can only de­ management; this panel fully endorses these prin­ grade with time. ciples and urges NASA to closely follow them in the The success of Eos depends on changing existing future. We therefore consider the recommendations practices in data collection and analysis, overcoming contained within these reports to be appended to these problems. those contained herein. We note that these reports consider a spectrum of Although Eos is essentially a large, long-term past difficulties and delve into the future to the ex­ data collection, processing, and analysis project, its tent that projects in years to come will bear resembl­ data are intended to serve as a dynamic resource for ances to those of the past. We envision an Eos that research on global phenomena. Hence, unlike data markedly differs in many ways from prior NASA processing in many projects, where value resides in projects. Consequently, rather than dwelling on past the final product, value in Eos is distributed over sins, we deal in this chapter with aspects of Eos that many stages of data processing, all of which must be are unique and will require careful thought, con­ retained by an Eos data and information system. sideration, and appropriate attention if the Eos data Furthermore, retrieval of much information is a and information system, hence Eos per se, is to be a matter for experimentation; fixed algorithms are dif­ success. ficult to define. Yet, many steps in the processing of The unique features of an Eos data and informa­ Eos data will be common to all analyses; these steps tion system provide a suite of management issues need to be identified and performed once. These that fall into two categories, those pertaining to pro­ characteristics of Eos suggest several new principles gram and project management, and those involving for the data system and its management that should information systems management. We deal in this be included in its design so that research access to the section with both classes of management issues. The data is facilitated for years to come. These principles distinction between the two categories is, in most are: cases, readily apparent. 1. Archival activities should occur at several stages in the data processing system, represen­ ting "raw" data, processed observables, and LONGEVITY inferred or derived quantities. The definition of these stages depends on the nature of the A crucial difference between Eos and previous observations and processing algorithms. If data collection and analysis projects, which requires calibration and navigation information are ac­ procedural changes, is the intended length of this ef­ curate and readily available, "raw" data re­ fort. Current practices for projects of limited dura­ tention and maintenance may be avoided. tion (up to five years) are determined by and for a Processed observables refer to raw data con­ single group of engineers, programmers, and scien­ verted to the physical quantity directly meas­ tists who participate throughout the lifetime of the ured by the instrument (i.e., spectral project. Many current efforts, having lifetimes of radiance). Inferred quantities are those that nearly 10 years, already show the difficulties of are derived from directly measured quantities changing team members and lack the flexibility to through application of auxiliary data or pro­ embrace new technologies and methodologies. The cessing models.

24 2. Data should be arranged into stable, predict­ when highly automated, should be thorough able units to allow for automated cataloging to allow ready training of new operations and and easy user access to very large data management personnel. Knowledge ofthe sys­ volumes. tem should be independent of the personnel.

3. Processed observable data should include 2. Instrumentation will change (e.g., improve). complete calibration, navigation, and other Data quality and catalog software should be ancillary information to facilitate use and to able to detect and record these changes. Data avoid repetitious processing. This and other analyses that lead to change should also be pertinent information should be appended to part of the Eos archives. the data with convenient frequency (e.g., per image). 3. Computer hardware should change when necessary to keep overhead low and to im­ 4. Catalog documentation should include results prove system capability. Software and pro­ of automated quality assessments and a trace cedural changes triggered by such hardware to cross reference other studies that have changes should be planned; benchmark tests learned something about the data. and standard data sets for reprocessing should be defined to verify the operation of the whole Automated Data Processing system after such changes occur. and Management 4. Software changes will occur to accommodate Much of the documentation chore can be han­ new data, new instruments, and improved dled by the addition of appropriate software for data analysis algorithms. Design of the data man­ processing that generates and accesses summaries of agement system should acknowledge this like­ data attributes (i.e., self-documenting software). lihood. All of these changes should be con­ There are three types of software needed: Eos­ trolled and documented. Management ar­ specific, non-Eos, and user processing software. The rangements should ensure that operational Eos data system should be automated to handle the software is secure, change procedures clearly large data volumes effectively. Proper quality con­ defined, and changes are made using a self­ trol and data management require that the process­ documenting, monitoring system. ing management software obtain and verify many at­ tributes of the data; hence, recordkeeping and gener­ 5. Human-machine interfaces in the data pro­ ation of summary reports should be added func­ cessing and management system should be de­ tions. The management software should determine signed to maintain proper oversight and quali­ the source and contents of data sets, record what ty control. Human tasks should be designed to processing has been performed, where the data are be interesting (not too lengthy, not fatiguing, stored, and form a cross reference list of data not too repetitive). Tasks should require prob­ coverage, time, etc. In other words, this software lem solving and judgment. One good way to should prepare a catalog entry containing this infor­ ensure this type of interaction is to include mation. Any' 'use" of the data should also be moni­ users in the management system such that tored and recorded by the management software. their knowledge is effectively included in data Acquisition of non-Eos data sets for Eos purposes quality assessment. The processing and ar­ requires processing of these data to allow equivalent chival institutions should have associated re­ catalog entries to be made. searchers with vested interests in the data per se.

Planning for Change 6. Use of expert systems in the management soft­ ware should not overlook the utility of incor­ The longevity of Eos and its data and informa­ porating various "scientific" analysis algo­ tion system, together with practical considerations rithms. Even though simple or standard algo­ of equipment, computer hardware and software, rithms may not be valid for all scientific prob­ and personnel, requires that the data system must lems, such algorithms can be usefully applied develop by evolution and that changes to the system for auxiliary quality assurance purposes by be planned rather than ignored. Design of the data assessing data patterns and characteristics that processing and management software and the associ­ can be used to monitor data attributes. For ex­ ated institutional arrangements should incorporate ample, a sea surface temperature retrieval procedures for changing EVERY component, in­ method, which is inaccurate for climate re­ cluding the human element. Efficiency is not as im­ search, may be sufficiently accurate to moni­ portant as flexibility and clarity of design. tor data quality, since expectations about the behavior of a physical quantity like sea surface 1. Changes of personnel will occur. The doc­ temperature (e.g., no rapid changes) can be umentation of the total system, especially utilized for quality control.

25 7. Data catalogs should consist of online, elec­ the data and information system. Unlike many flight tronic data sets that can be amended to accom­ projects of the past, virtually every facet of Eos, modate: new data sets, new attributes for ex­ hence its data and information system, will require isting data sets, new cross references among coordinated interaction to preserve its synergistic data sets, processing and use trace, and characteristics. Synergy within Eos occurs at three bibliography. levels, all pertinent to the data and information system. They are: the scientific, project, and pro­ 8. Data holdings can change in several ways gram levels. other than addition of new data sets: (a) pro­ cessing can provide an alternate form that has Scientific Level been sorted (mapped), edited, or labeled by analysis, (b) processing can produce associ­ On the scientific level, we anticipate system ated, inferred quantity data, and (c) process­ resource conflicts arising that will demand resolu­ ing of several data sets can produce alternate, tion within the confines of the data and information correlated data sets. Planning of self­ system. Multidisciplinary researchers and research documenting data catalogs and a directory teams will have needs for particular observational se­ and holding structures should accommodate quences, while disciplinary researchers may well this inevitable evolution. have requirements for entirely different measure­ ments. Both groups will be affected by spectacular environmental events and the pressures (both scien­ Institutional Commitment tific and political) to respond. These scientific conflicts could easily (and will Considering the above points, in addition to the likely) have an impact on the limited resources of the requirements for incorporating scientific results into data and information system. While they may ram­ the archives we conclude that the role of data pro­ ify and pervade the system, the areas most likely af­ cessing and archival institutions will be more active fected will include command management, instru­ than in past or current practices. These institutions ment and mission operations, network services, on­ need to be part of the research process so that interest board buffers, and data processing services. Since in the data and vigilance over the system will remain these services and facilities will have limited perfor­ high. Provision of the many services outlined here is mance and capacity, a fast, efficient, and effective not a simple or easy task; appropriate institutional mechanism must be established to maximize resear­ rewards must be found. Finding a solution to this cher benefits when dissimilar scientific objectives problem must take account of several characteristics collide. of the current practice of science: 1. Publication of data and its documentation in Project Level refereed scientific literature is not possible, Resource conflicts at the scientific level directly especially for the large satellite data sets. translate into concerns at the project level. A major Hence, scientists do not spend much time on task of project personnel will be resource manage­ data structure or documentation. ment, assuring continued, uninterrupted perfor­ 2. Quality assurance and formal acceptance pro­ mance of the entire data and information system. cedures for large data sets do not exist, and Clearly, this will neither be a trivial task nor will the reliance on individual scientists is inadequate. results of its performance be scientifically inconse­ quential. The usual tools of scientific quality management, There is an additional concern with resource namely peer and publication reviews, are inadequate management that heralds difficulties for the data for creating the Eos data base needed for future and information system. This is synergistic instru­ Earth sciences research. This suggests that the role of ment operation. While the Eos Science Steering the processing, management, and archival institu­ Committee has discussed a number of synergistic in­ tions should be extended to include participation in strument scenarios, we do not presuppose that these research projects and the publication of data docu­ are totally inclusive. As the research community in­ mentation. Consequently, some institutional creases its knowledge base, new requirements for publications might well be elevated to full journal multiple-instrument observational sequences will be status by instituting review procedures similar to identified. Preexisting requirements may remain and those of the refereed literature. circumstances be further compounded by newly de­ ployed instruments that may well place greater demands on available resources. The data system SYNERGISTIC CHARACTERISTICS design, implementation configuration, and opera­ tion must be based on preserving flexibility to the Coordination will be the keystone of a successful greatest extent possible, maximizing scientific pro­ Earth Observing System and the modus vivendi of ductivity to its fullest measure.

26 Program Level might otherwise be a complex scheduling problem, and create yet others, particularly if the Eos plat­ At the program level, resource conflicts take on a form(s) host instruments of an operational genre somewhat different character. To a large degree, the together with a suite of individual, principal in­ data and information system will be the key to ensur­ vestigator, and facilities-class instruments. ing Eos' scientific success. Consequently, fiscal re­ We know that existing systems, such as Landsat, sources must be identified and preserved to ensure its are inadequate, and we know from a management full and complete implementation, regardless of the perspective that an Eos data and information system inevitable pressures to do otherwise. This task will must be vastly superior; we do not know the specific not terminate with the deployment of one or more solutions to the myriad management problems that Eos platforms. Rather, as new instruments are pro­ these high data rates and volumes will create. We posed (and subsequently deployed), and new syner­ therefore recommend that the Eos data and informa­ gistic groupings of instruments are identified, ade­ tion system be built in an evolutionary fashion, quate funding must also be secured for enhance­ enabling solutions to evolve along with the ments to the data and information system, when re­ technology and expertise to cope with this new era in quired. In doing so, the fidelity and performance of space research. the system may be preserved to the ultimate benefit of scientific research. ARCHIVAL DATA SETS DATA VOLUME AND RATE A new objective unique to Eos is to increase our knowledge base, represented by data archives, by The Eos data and information system will be re­ persistently adding the results of scientific data quired to handle daily more data than any system analyses to the basic observational data holdings. ever conceived. In general terms, Eos will produce This continuous evolution of information content several orders of magnitude more data per day and is requires redefinition of the roles of scientists and ar­ envisioned to have a duration exceeding any mission chival institutions as integral elements of this system ever before proposed. Today, the only space mission (which is conceived as a network tying users, dispersed at all close to projected Eos data rates, hence data processing, and archival functions together with a volume, is Landsat. While Landsat processing capa­ central data directory and network management). bilities have been improving, the resultant data re­ Individual investigators or scientific teams have main relatively inaccessible and are used by a rather new obligations to accomplish this knowledge in­ small number of individuals compared to projec­ crease. In return for access to the data system, these tions for Eos. Clearly, the operation of an Eos data investigators must return their results to the system. and information system will create management Whether this obligation is accomplished by pro­ problems of a magnitude that cannot even be fully viding system access to their individual holdings or appreciated at this time by either NASA manage­ by physical transfer of data sets to project archives, ment or the scientific research community who must additional processing of the data resulting from an cope with these data in their research. investigation will be required to provide: There are numerous management issues associ­ ated with high data volumes and rates that can be I. a catalog entry containing descriptions (de­ clearly identified at this time. They involve virtually fined by Eos) of data sources, data properties, all of the recommended functional characteristics analysis methods, and attributes (e.g., loca­ that the Eos data and information system should ex­ tion, time, wavelength); hibit. Since the Eos data and information system will 2. a standard format to allow aCCess from Eos of necessity be a limited resource, it is reasonable to software, and processing by Eos archival expect that the major issue associated with high Eos software; data rates and volumes will be scheduling, including scheduling of data acquisition, data set production, 3. documentation of data set contents, process­ data access, and distribution timing. ing algorithms, instrument characteristics; The architectural example discussed in Chapter and III of this report concentrates on data flow and 4. an evaluation of the results, including error transparency. We have recommended the use of ex­ analyses and validation tests, as well as a rele­ pert systems to the extent possible to reduce delay vant bibliography. times. Yet, we anticipate that even with these ad­ vanced technologies, limitations in telecommunica­ Archival institutions must be active participants tions bandwidth, computational processing power, in the data processing that supports scientific studies etc. will generate a significant number of scheduling rather than passive storehouses. This new role re­ problems. Further, it is likely that the synergistic quires data processing and distribution functions, in nature of the overall mission will both solve what addition to the usual data acquisition and hol~ing

27 functions (i.e., storage, security, purge). Data pro­ The larger user class is composed of consortia of cessing will be necessary to: investigators formed to undertake specific, large research projects external to Eos (e.g., a special field 1. put data sets into any Eos standard formats or study may wish to acquire supporting data being col­ develop software interfaces between data in lected by Eos or a retrospective study may be in­ non-Eos formats and any Eos standards; itiated to produce a climatological data set). This 2. collect simple statistics of the data that can be type of user will access data directories, catalogs, used to assess data quality; and browse files relatively infrequently but will re­ quest rapid or scheduled delivery of large data 3. examine data or apply simple algorithms volumes, possibly for long periods of time. The Eos to develop attribute lists used for search information network should be designed to allow ex­ purposes; pansion (paid for by the user) to support additional 4. compare data attribute lists producing cross large demand uses. A key feature of the Eos infor­ reference entries in the data catalogs; and mation system, in this case, is access to data quality assurance and calibration information, in addition 5. prepare browse data sets by sampling the to the basic observation data, and provision of ap­ original data volume. propriate software interfaces within the network. Data distribution functions will be facilitated by: 1. development and maintenance of data direc­ ECONOMIC FACTORS tories (for all pertinent data sets) and data catalogs (for data sets held by Eos); Most academic research is funded with an assumption that data, once acquired, will be ex­ 2. production of browse data sets; changed through publications and by other means 3. establishment of peer review procedures for where the cost of obtaining the data by other in­ data quality; and vestigators is essentially negligible. In essence, scien­ tists exchange data for personal recognition, which 4. publication of data documentation catalogs. translates into professional growth, increased in­ come through advancement, and peer recognition. Thus, grants for space research are often dominated FLEXIBILITY IN DESIGN: by salaries of scientific personnel compounded with FUTURE USERS institutional overhead. Computer costs may also be substantial, but an approach being adopted more The Eos data and information system is being de­ and more often is for a funding agency to provide a signed as part of the Eos project by evolution of a shared-use computer facility and allocate time to its dispersed data processing network. Since the net­ investigators at no cost to the researcher. This tends work is intended to provide flexible access to large to encourage full use of any large-scale machine, and data processing volumes by many project-related the pattern is generally that new computers are fully users, its design will have potential for other users subscribed within one to two years. entering the system at a later time. Thus the Eos data The traditional procedure for obtaining Earth and information system is meant to continue the pro­ remote sensing data for research purposes is cess of data analysis and research beyond the realm somewhat analogous to the super computer situa­ of the data collection phase of Eos alone. Provision tion. Investigators propose to participate in space should thus be made in the network design for ex­ missions and, if successful, receive access to data at pansion of data access to and for acquiring new data no cost. Copies of archival data are made available sets from two classes of users. at the marginal cost of reproduction. Under this The smaller user class includes individual scien­ system, the aggregate scientific community works to tists or small groups investigating specific justify funding for a major space program or mis­ phenomena, incidents, or specific geographic loca­ sion and then receives the resultant data at marginal tions. This type of user, as a group, will access data cost. In essence, if the promise of future knowledge directories, data catalogs, and browse data very fre­ is sufficient, then present professional stature is ex­ quently but request only modest volumes of data. changed for access to data at little direct cost. Full The network design must provide proper access se­ exploitation of the data is encouraged because the curity and usage accounting, allow remote access to only barriers to obtaining data are successful peer directories, catalogs, and browse data, provide evaluation leading to selection for participation in a usage statistics for accounting, and allow ordering of mission or research program. data. The Eos data and information system design These considerations have direct bearing on the may allow a user to order non-Eos data that is subse­ financial model that should be used in operating quently reformatted by Eos before being transmitted Eos. The current research system will continue to to the user. function if data are provided at costs not exceeding

28 the marginal cost of making the data available. under the purview of other governmental entities. However, an alternate system, which requires those Clearly, to achieve the scientific objectives of Eos, obtaining data for research to pay a proportionate researchers will need easy access to directories and share of the capital cost of acquiring the data (e.g., catalogs, as well as to the data per se resident within Landsat), will necessitate major changes in the way these external archives. research is funded. A full-cost recovery system will The technical challenges of efficiently and trans­ work against the interests of maximizing use of Eos parently linking heterogeneous data bases are not by discouraging access to data, and further is dia­ beyond the capabilities that should be readily metrically opposed to the unique feature of Eos that available during the 1990s. The problems arising requires researche~s to return reduced or derived from the multidisciplinary scientific objective will be value-added data sets to the archives. managerial in nature. Project management's princi­ The scientific community and its sponsors ple task will be to effectively ensure that the direc­ strongly support a system where the value of im­ tories and catalogs that researchers access are up­ plementing a mission is judged in advance, and ac­ dated on a very frequent (perhaps daily) basis. Thus, cess to its data is governed by a peer review selection inter- and intra-governmental communications will process. Under this system, institutional capabilities likely be the deciding factor in determining whether a such as a data system or spacecraft are funded as part researcher's data needs are met. Qf the decision to proceed with the activity, not On a program management level, the problems recovered directly from the researchers utilizing generated by this archival access issue become them. Consequently, it will be more straightforward political in nature. We anticipate the need for multi­ to successfully implement Eos and ensure its max­ ple interagency agreements detailing access pro­ imum utilization if monetary charges for data are cedures, updating methodology, responsibilities, limited to the marginal cost of reproduction. funding, etc. On the international, inter­ Although this economic model will work ade­ governmental level, similar negotiations must take quately for some investigators, it will not work for place and agreements be defined. In both cases, we all. Many researchers will be contractually obligated anticipate that these external governmental entities to provide value-added data to the archives. If a will request reciprocity in archival access. Since researcher is required to pay even the marginal cost many research scientists and colleagues are in the of reproducing the initial data he receives, then it is employ of these external organizations, we expect reasonable to expect that he should receive financial that many will likely be collaborating with project­ remuneration for processing, copying, and pro­ sponsored investigators. Clearly then, reciprocal ac­ viding his derived or reduced data to the archives. cess will be neither a technical nor a scientific issue Unless these costs are considered and included from but, rather, political. the outset of proposal preparation, the task ofretur­ On this basis, we urge NASA to undertake ap­ ning these data to the archives will necessarily re­ propriate negotiations with those governmental quire the use of an individual scientist's research bodies responsible for the operation and main­ funds; hence, the task will be of low priority. tenance of archives holding data pertinent to Eos Thus, some adjustments to the existing economic scientific objectives. The resultant agreements system will need to be made. We anticipate that these should be consistent with the access requirements changes will not require a major restructuring of the delineated within this report. system, but rather note that the problem will likely arise and recommend that NASA management take all necessary steps to ensure that priority is given REWARDS AND PUNISHMENT: to the acquisition of value-added data from Eos A PERSONNEL DILEMMA researchers. If the scientific community is to have easy access to existing, planned, and future data sets, the Eos INTERACTION WITH OTHER data and information system must fully utilize ad­ GOVERNMENTAL ENTITIES vances in data system software and hardware. In order to completely integrate these improvements in­ Unlike many other space research missions, Eos to the Eos data and information system it will be should provide a means for addressing a suite of necessary to greatly expand upon and upgrade ex­ multidisciplinary research problems. To do so re­ isting human resources involved in the fields of data quires access not only to data derived from Eos management, preparation, maintenance, and scien­ spacecraft, but also to other archives, many of which tific validation. The success of the data and informa­ are operated and maintained by other governments tion system requires that these professionals be and governmental agencies. Similarly, we anticipate recognized, encouraged, and appropriately re­ that Eos will enable multiple data source, discipli­ warded for their efforts. nary research to be more effectively conducted. Here Many traditionally trained scientists and too, many of the requisite archival holdings are engineers lack the inclination or ability to adequately

29 perform the tasks necessary to manage, produce, there are two prime issues that NASA management and maintain data sets that are useful to the greater must address at this time. They are (i) the need for scientific community. These tasks include scientific immediate action to plan, design, and implement an quality assurance, validation, algorithm mainte­ Eos data and information system, and (ii) the need nance, and, of particular importance, documentation. to develop the experience, expertise, and technology Employment in the data management sciences is necessary to ensure that the resultant system effec­ currently rendered relatively unattractive because tively meets the needs of the research community. workers receive fewer promotions and other types One means of addressing these issues is to initiate a of recognition than coworkers in the more tradi­ top-down functional design effort that is very close­ tional scientific disciplines. Consequently, the field ly coupled to and iterated with prototype experience has not attracted a large number of individuals of to efficiently converge on an optimal system by the the type and caliber that Eos will most assuredly beginning of the next decade. Regardless of the ap­ need. There are several reasons for these differences proach taken in addressing these issues, there are in recognition and rewards, but primarily they stem two imperatives or guiding principles that should be from the nature of the work; data management sci­ followed throughout the evolutionary process. entists do not produce the same type of publications They are (i) scientific involvement and (ii) scientific as those engaged in traditional scientific endeavors. oversight. Data management publications are produced less The prime objective of a scientific mission is to often and do not generally appear in the refereed obtain new knowledge and understanding. There­ scientific literature. Indeed, the major derivative of fore, it is reasonable that since data are acquired for data management is the data set itself, which is scientific research, scientists should play a signif­ merely described by publications or, more often, icant role in determining what data are required, user's guides and other types of similar documenta­ how they are acquired, how they are processed and tion. Throughout the scientific community, disseminated, and the disposition of the data after recognition of professional accomplishment and ex­ they are collected. Scientists must be actively in­ cellence is primarily related to publications in volved throughout the Eos data and information refereed scientific journals. A major step toward system evolutionary process to ensure production encouraging highly qualified and motivated sci­ of and access to data sets of the highest quality. entists to enter and remain in the field of data man­ This involvement will maximize the return on in­ agement would be to broaden the base of what is vestment and improve the quality of the resultant recognized as a truly significant scientific publica­ data. tion. This then remains the guide for measuring Management of scientific data bases has been professional performance, but should include pub­ the responsibility of project personnel, principal in­ lications of technical memoranda (which describe vestigators, and project archives (such as they are). data sets), user's guides, algorithm and validation Those scientists outside of the project per se who study results, as well as the data sets, per se. actually use the data for research purposes have not been actively involved in this process. These scien­ tists should be involved at the outset via an over­ KEY ISSUES sight and advisory function, since the most success­ Although there are many considerations, both ful examples of data base management involve user technical and managerial, noted within this report, oversight.

30 v. RECOMMENDATIONS AND CONCLUSION

The goal of the Eos data and information system use costs can be recovered. If operational users must be, by the time Eos spacecraft are returning significantly impact Eos operations, then the rele­ data, to meet the challenges of Eos mission opera­ vant operational entity should fund or provide the tions, data transport, processing, and data manage­ enhancements needed to meet their needs. This ment, in addition to the challenges of access to infor­ recommendation necessarily implies that the infor­ mation and data not under direct control of Eos or mation system be designed in a manner that ensures even NASA. An Eos data and information system flexibility, allowing enhancement without affecting must be a system that includes geographically operation of the remainder of the system. We believe distributed sites of varying capabilities and responsi­ that a modular, flexible architecture such as de­ bilities. We expect that by the 1990s local processing scribed in Chapter III of this report has the potential capabilities, combined with network technologies, to meet this as well as many of the other require­ will allow such a geographically distributed system ments presented herein. Consequently, we urge to become a reality. In fact, we envision the key ob­ NASA to explore the adoption of a modular, trans­ jective of an Eos data and information system to be parent architecture for the Eos data and information providing remote and interactive electronic access to system. the variety of capabilities and services that the system offers. We consider the management of this data and information system to be considerably FUNCTIONAL ELEMENTS: THE 1990s more difficult to implement successfully than the technological aspects. Within this report, there are a number of recom­ mendations, requirements, attributes, and char­ acteristics pertaining to detailed functions and OPERATIONAL AND capabilities of the Eos data and information system. COMMERCIAL USERS An annotated listing of requirements is presented in Appendix II and grouped conveniently into seven We recommend that the Eos data and informa­ separate categories. Figure 3 schematically portrays tion system be designed to meet research needs, but these functional categories and their interconnections. that it also be used to meet the needs of operational The seven functional categories or groupings in­ (e.g., NOAA, DoD) users, to the extent that such clude flight systems, user, operational, information needs do not detrimentally impact use of the system services, advanced data base management, Eos data for research. Commercial users, likewise, should be processing, and non-Eos data base functions. Flight accommodated to the extent that their use does not systems include the functions and characteristics of deter Eos research and to the degree that commercial both remote sensing instrumentation as well as the

Eos Flight Operational Systems User Functions Functions Functions

Eos INFORMATION SERVICES FUNCTIONS

Advanced Eos Data NOAA and Data Base Other non-Eos Management Processing Functions Data Bases Functions Functions

Figure 3. Eos data and information system - Generic functional groupings. The schematic interconnection of these elements indicates thedependencies of any one task upon other functional domains.

31 onboard data system. User functions embrace the of an Eos data and information system by attempt­ envisioned modi operandi of researchers, operations ing to carry out these functions on a smalJer scale. personnel, instrument scientists and teams, and The key lies in trying several solutions to the various other individuals requiring access to Eos services. problems so that planning for the resultant system The functions and characteristics inherent in space­ will be based on experience. To secure the enthu­ craft and data and information system operation are siastic participation of researchers and managers, included in operational functions. Eos information the motivating factor for these information manage­ services functions include the suite of network ser­ ment studies should be real scientific investigations vices (both space- and. ground-based) that we envi­ that demand solutions to many of the problems and sion during the Eos era. Many of the new functional functions outlined in this report. requirements for the data and information system are grouped under advanced data base management, Expansion and Focus of Pilot Data Systems while processing requirements for Eos data, per se, are grouped separately. Similarly, the needs for ac­ We recommend that NASA's current Earth cess to and data from non-Eos sources are encom­ science pilot data systems be continued and ex­ passed within the final functional grouping. panded (as appropriate) in their representative Unlike many space research missions, the re­ disciplines. AdditionalJy, we recommend close col­ quirements levied upon Eos as a whole are such that laboration with the UCAR Unidata initiative, which these seven functional elements are very much in­ provides a similar focus for the atmospheric terdependent upon one another. Tasks contained sciences. These efforts, each in a unique fashion, are within one element may not be accommodated solely addressing many of the technical and managerial within the domain of that element but rather may re­ questions that must be answered to design an Eos quire input from one, two, or perhaps an even data and information system. Future efforts should greater number of other functional domains. Thus, address four areas: (i) develop data formats and in­ for Eos to be successfulJy implemented, all of the terface software to alJow access to heterogeneous data and information system functional require­ data from any of the existing pilots; (ii) develop (with ments delineated in this report should be provided. researchers) new analytical tools for multi­ Consequently, we recommend that the Eos data disciplinary research; (iii) improve data sets in pilot and information system of the 1990s include the archives by developing improved documentation functional characteristics depicted here in elemental and formats that cross traditional Earth science form, identified throughout this report, and sum­ boundaries; and (iv) produce an electronic data marized in Appendix II. directory for alJ NASA Earth science holdings, and similarly electronic catalogs for alJ Earth science pilot project holdings. INFORMATION SYSTEM Some or alJ of these objectives have already been EVOLUTION: THE 1980s identified by individual pilot efforts, but they should be focused with an Eos information system thrust, Many existing scientific research projects and leading to progress toward Eos. A key feature must pilot data system studies are considering aspects of be strong interaction and collaboration with ongoing problems that must be solved to implement suc­ research efforts that emphasize the colJection and cessfully a data and information system supporting analysis of multiple data sets. The evolutionary Eos the project and program needs of Eos. Therefore, data and information system should work with re­ progress toward an appropriate information system searchers to facilitate archival access, and the re­ can be obtained by focusing existing efforts, searchers should return reduced and value-added stimulating new scientific uses of data, and coor­ data and analytic results to the archives. This in­ dinating an iterative learning process that is directed teraction will alJow the information system, the pilot toward evolutionary or "test-bed" information systems, and analysis efforts to evolve with a new system concepts. These activities (many already level of sophistication. Developed software should begun or planned) should be focused and used to be considered part of the archives. The Eos data and generate experience with information management information system as welJ as the pilots can be struc­ problems and their optimal solutions, and provide tured to properly manage information while stim­ the nucleus of a functional Eos data and information ulating research. system. We further recommend that the Global Resource The conceptual structure for an Eos information Information System concept be focused on Eos ob­ system that underlies many current activities is that jectives, then utilized for developing the information of a central coordination and information manage­ network aspects of the Eos data and information ment function, together with access to data holdings system. Particular attention should be given to located at a number of data archives. The recom­ developing and utilizing technology to allow mendations noted below are meant to focus study on transparent access to heterogeneous data bases, to the problems of defining the functions and structure advanced data base management software with

32 intelligent (expert systems) search capabilities, and utilized to identify key data sets that need to be to cost-effective network capabilities, including retained, maintained, and organized in a time series direct data broadcast. To the extent applicable, data base format for comparison with Eos data in guidance from the Space Physics Analysis Network the 1990s (e.g., preservation of the Advanced Very activities may be both desirable and worthwhile. High Resolution Radiometer archives).

Science Projects to Focus System Evolution CONCLUSION We recommend that the Earth Science and Ap­ plications Division fund a limited number of The final and perhaps most significant recom­ multidisciplinary research teams to investigate a mendation of this panel is to initiate the planning select number of crucial scientific objectives in and implentation of an evolutionary Eos data and in­ hydrology, biogeochemistry, or climatology, using formation system without delay. A functional currently available data sets. These research ac­ system providing the means through which Eos data tivities would foster multidisciplinary studies of can be most fully utilized will not be built in a matter Earth, studies that require access to multiple, diverse of a few years; it must be allowed to evolve along data sets. These research teams will be responsible with the increasing knowledge bases of the Earth and for using the evolving Eos information system, col­ data management sciences. lecting data from external archives, analyzing the There are two fundamental principles that data, and returning their results and experience to should be followed throughout the Eos data and in­ the embryonic information system. The link between formation system evolutionary process. They are: (i) multidisciplinary research projects and information Involve the scientific research community at the management systems can provide the experiences outset and throughout all subsequent activities, since necessary to move toward an operational Eos data the data will be acquired, transmitted, processed, and information system in the next decade. The key and delivered for scientific research purposes; and is selecting teams to do this type ofresearch now, us­ (ii) provide the researcher with an oversight and ing current data and archival systems to define where review responsibility, since the most successful ex­ many of the problems lie and what the solutions amples of data base management rely on the active might be. Finally, these scientific projects can be involvement of scientists.

33 REFERENCES

1. Science and Mission Requirements Working Group for the Earth Obser­ ving System, "Earth Observing System," Volume 1, Part 1, NASA TM 86129, 1984. 2. National Academy of Sciences Space Science Board, "Data Management and Computation," Volume 1: Issues and Recommendations," National Academy Press, Washington, DC, 1982. 3. National Academy of Sciences Space Science Board, "Data Management and Computation, Volume 2: Space Science Data Management Units in the 1980s and 1990s," National Academy Press, Washington, DC (in press). 4. Ludwig, G.H., "Space Research Data Management in the National Aeronautics and Space Administration," Unpublished consultants letter, 1985.

34 APPENDIX I - RESEARCH SCENARIOS

It is useful to consider how scientists in various studies, (2) routine statistical analyses, and (3) cor­ disciplines would utilize Eos and its data to address relation studies. research problems. In doing so, the Data Panel developed requirements and system attributes by Case Studies considering examples of research activities, most of Case studies are analyses of specific, high-detail which parallel the scientific objectives delineated data sets to diagnose climate processes, to validate within the Eos Science and Mission Requirements remote sensing data analysis techniques by compari­ Working Group report. Necessarily, these scenarios son to other measurements, or to validate the focus on current activities and methodologies. physical interpretation of statistical patterns in­ However, we do not envision major variations in the ferred from coarser data analysis. Case study data ways Eos investigators will approach problems in a sets are generally limited in geographic and temporal functional sense. Therefore, these scenarios provide coverage, but still of large volume because of the a background for developing Eos data and informa­ need for high resolution. Access to raw data is usually tion system requirements. needed. To generalize results from case studies, an ensemble of similar data must be examined. Thus, interaction of the research scientist with the data ar­ CLIMATIC RESEARCH chives involves catalog search to find data sets for This scenario portrays information system re­ specified geographic regions and times, catalog quirements for climate research, with a focus on at­ searches of many different data sets to find those mospheric phenomena. In climate research, not all cases with adequate data for intercomparison and of the quantities to be derived from the data are validation studies, and browsing of reduced-volume known beforehand. Indeed, a main purpose of the versions of the data to find other examples of par­ data analysis is to find and diagnose patterns of ticular phenomena. Data must be cataloged accord­ behavior; the knowledge of climate therefore grows ing to place and time, instrument and satellite, and by accumulations of multiple descriptions of the physical quantities measured to allow for correlation same data with special emphasis on correlations be­ searches. Browsing reduced-volume data to search tween quantities. Summary statistics obtained from for specific kinds of events can be facilitated by an the data are compared with climate model perfor­ interactive expert system (i.e., a computer system mance to improve understanding and simulation that "learns" to sort data from other investigators capability. who have already sorted the data). This system is possible only if the results of previous case studies and statistical surveys are made part of the data ar­ Data Characteristics chives. Updating the catalog and archives and the The primary characteristics of a climate data set menu of sorting criteria allows the use of data to are its global coverage and temporal record length, become more sophisticated with time. More atten­ but to be useful the coverage and length must have tion should probably be paid to implementation of uniform properties. Research objectives, namely this approach than to providing processing software diagnosis of climate processes, also require suffi­ to investigators. cient space and time resolution (i.e., detail) to observe key atmospheric phenomena. Therefore, Statistical Analyses remote sensing climate data can be described as multispectral images with space and time resolution Routine statistical analysis of the data demands of 10 to 50 km and I to 12 hr., respectively. The repeated access to long, continuous time series of dominant length scale of atmospheric motions measurements. These analyses can be considered as argues for image scene sizes greater than 1,000 km progressive sorting of data, producing statistical ("imagery" means data taken rapidly over large characterizations of content. Access must be to both areas and includes such data as temperature soun­ original measured quantities and to derived quan­ ding data). Useful record lengths are from a few tities. Information on calibration, calibration years to decades. The importance of radiation pro­ history, and documentation of techniques used to cesses in climate is such that research interest in derive quantities are necessary parts of the data measured radiances will be equal to the interest in record. Although most users would perform their quantities derived from the radiances. primary processing locally, several routine utilities would be used if available: mapping, statistics, and sorting and classification. The first refers to provi­ Data Access Patterns sion of the data in a standard map grid so that Three interrelated types of data access are often statistics may easily be obtained. A mapping utility used to attain climate research objectives: (1) case should allow not only a selection of the grid and its

35 resolution, but also control of the procedures (ac­ research needs could be met by ready access to com­ cumulation, sampling, replication, interpolation) mand history information (for short time record in­ used to reproject the data. Simple statistical utilities terruptions) and by data processing to produce the would be useful to reduce the data volume sent to a defined climate observations. Examples of this last researcher. For example, an investigator interested strategy include continued production of lowei"­ in interannual variability of synoptic cloud features resolution climate observations by processing of might request data averaged over the diurnal cycle. higher-resolution data collected for some other pur­ Samples of the data may also be desirable. Sorting pose, or continued production of spectral images by and classification utilities make it possible for a processing spectrometer data to simulate broader scientist to select from a long data record those band images. The key requirement is to obtain a· events of interest (determined by simple physical long, uniform data record. criteria). Updates of these utilities based on previous research results allow for more sophisticated Data Management and Control analyses subsequently. The underlying structure of the data base re­ Correlation Studies quired to support the type of research discussed above is that of a centrally coordinated network of Correlation studies can be either case studies or dispersed data archives. The central coordination ac­ statistical analyses; however, the focus is on the cor­ tivity involves maintenance of a master directory of relation between many variables. This kind of study all holdings and a library of software modules is most demanding of resources because it requires needed to interact with archives transparently. The all of the capabilities discussed above as well as ac­ interaction with specific archives includes examina­ curate time and space collocation of mUltiple-source tion of local (more detailed) catalogs, browsing, data. It will also require greater access to other data ordering, and addition of data. Software modules to sets outside of those produced directly by satellite read (and write) data files should be provided, either missions. Sorting of multiple data sets into col­ by the central coordination institution or the local located time sequences requires significant computer archives. resources and detailed catalog and inventory infor­ Key technical difficulties that need to be resolved mation for each data set. Multidisciplinary research (based on current archival practices) are: (e.g., air-sea interaction, land-atmosphere pro­ cesses) will make extensive use of this type of data 1. improve documentation to provide complete analysis. Access to multiple archives containing experiment and calibration information; results of previous data analyses will also greatly 2. improve navigation information and its facilitate this type of work. accuracy; 3. resolve difficulties of multiple data set access Needed Utilities caused by widely varying data formats; Three types of software utilities need to be 4. provide data browse capability in systems that available to researchers either online or as doc­ can "learn" progressively more sophisticated umented software that can be used locally. The first sorting; examines data documentation to determine physical variables (in SI units), to allow specification of 5. provide for more remote data manipulation to measurement geometry and conditions, and to pro­ allow preliminary (simple) studies that lead to vide time and location of the measurement. The sec­ more focused data selections. ond allows mapping of data on a standard grid, time selection, and data volume reduction by a variety of Scale of Activity simple processes: sampling, averaging, sorting, classification, and calculation of simple statistics. Analysis of very large data sets will continue to be The third updates catalogs, inventories, and limited to a small number ('" 10) of groups, usually selection-software menus to reflect previous analysis associated with large models, that have sufficient results. computer resources. The interactive nature of ar­ chives is therefore crucial because the results of an analysis are usually smaller in volume than the Mission Operations original data and of more interest to a larger number The constraints placed on mission operations by of researchers. Consequently, these few, large climate researcher data requirements are primarily groups should probably have component archives of those required by the maintenance of long, consis­ the total network; however, these groups will not be tent measurement records. Changes of observational very interested in providing services to others. The strategy generally would not require rapid system role of a coordinating function will be to provide ser­ response. If space, time, and spectral resolutions of vices (Le., format documentation, catalogs). The the instruments can be altered on command, climate more numerous ('" 100) users of smaller data sets

36 will also be more interactive and use more processing ing algorithms required to produce statistical quan­ power to access data. tities (including a climatology data set for anomaly calculations) and a mapping facility. This type of ac­ cess may take the form of electronic transfer of small CLIMATIC MONITORING data sets (maps) to a remote location for display. More rapid response to data requests (based on a This scenario represents the type of observations browse data set) or a method for remote display of that might be needed in the future to support higher-resolution data sets may be necessary. seasonal forecasting qr to target anomalous events for intensive study. Data collection methodology is focused on uniform, global coverage; however, Mission Operations near-real-time analysis is required to produce a Unlike climate research, this type of study re­ reduced resolution version of the data along with quires not only more timely access to the data stream statistical summaries. In addition, this type of (within one week) and near-real-time processing on a scenario requires an ability to switch rapidly from a routine basis, but also the ability to change observa­ low level to a high level of effort. Thus, routine ex­ tional strategies quickly to get better information amination of summary statistics may lead, in quick about some particular location. Since the research sequence, to a request for more detailed data already supported by this capability is directed toward in the archives and to a change in instrument con­ developing monitoring and prediction techniques, figuration. This capability would support study of some evolution of the observational strategy over the the time evolution of major climate anomalies. course of the mission is expected. .

Data Characteristics Data Management and Control The basic data taken by specific instruments represent a global description of the thermodynamic Not only the instrument configuration, but also state of the atmosphere. These data would be the data processing system will evolve as more is equivalent to current routine weather observations, learned about the controlling factors for climate but a version ofthe data with somewhat lower spatial variations. Consequently, proper quality assurance ("-' 500 km) and temporal ("-' 1 day) resolution would and documentation of observations and processing be prepared. Climate monitoring would entail near­ software is imperative. Results of experimental real-time calculation of a number of statistical quan­ anomaly predictions, whether retrospective, or near­ tities (e.g., anomaly maps) presented in a standard real time, should be stored in some fashion for model map-projection or grid format. Appearance of a intercomparisons. A trace of studies focused upon "significant anomaly" would trigger an examina­ particular anomalous events can benefit subsequent tion of the full resolution data being acquired. analyses of climatic variations.

Data Access Patterns Scale of Activities Routine access to the archives for exammmg Routine access to the archives is likely to be reduced resolution data and statistical summaries limited to a few institutions with climate-scale might be limited to a very few ("-'5) institutions in Global Circulation Models and large computer near-real time (weekly or monthly). Other research resources; however, access to higher-resolution data institutions may access these same records at other sets for certain case studies (e.g., EI Nino, volcanoes) times to test forecast models, which would require a will be more widespread. browse facility to locate "interesting cases." If seasonal forecasts are being attempted, then rapid access to the higher-resolution (unprocessed) data LAND SURFACE CLIMATOLOGY will also be needed. If intensive observations are re­ quested or some change in observational strategy is The objective of a land surface climatology proj­ called for, then data characteristics will be similar to ect is to develop a better understanding of the pro­ those for case studies or field studies discussed cesses occurring within, and the interactions among, above; but access will need to be more rapid. Part of the Earth's biospheric, edaphic, hydrologic, and at­ the decision process in changing instrument con­ mospheric systems, and to determine their role in in­ figuration would be examination of "current" data fluencing climate over land surfaces. To accomplish in mapped or image form. this objective, principal experimental efforts would be organized and conducted as three parallel ac­ Needed Utilities tivities, each of which depends on the development of a supporting data base and data analysis system. The basic utilities needed to provide near-real­ The first activity would be to conduct an analysis time access to statistical summary data are process- of existing remote sensing data in an attempt to select

37 climatically representative study regions. The pur­ procedures to gain rapid, easy access to heter­ pose is to determine the extent to which changes in ogeneous data in geographically dispersed archives. the land surface can be determined and measured, These data would need to be preprocessed and inter­ and to assess the relative sensitivity of climate to compared. They would also be compared to col­ various land processes. The second activity would be lateral data sets in the form of tabular meteo­ to prepare and validate comprehensive global data rological records; digital topographic data; poly­ sets derived from operational satellites. The valida­ gonalland cover and soil designations; and intensive tion would be performed to document the current point, area, and transect data from field meas­ state of the Earth's land surface with respect to a urements. This would necessitate integrating se­ number of select environmental parameters. The lected ground reference data, such as soil and land third activity would involve pilot experiments on use maps, into the data base. A georeference struc­ specific regional or continental land masses to cor­ ture would be used for relating these data sets. relate remote sensing measurements with climate While analysis of retrospective data would pro­ sensitivity parameters and to validate or modify vide some indication of the influence of past land land-atmosphere interchange models for these study surface changes on climate, the present physical and sites. biological state of the land surface should also be These studies could include: described. The best current sources of this informa­ tion on a global scale are data sets derived from 1. Vegetation: Fluctuations in green leaf biomass operational satellites. These satellite and ancillary (monthly, seasonally, and annually) would be data sets would be assembled into a global data base. related to precipitation and surface Specific study sites, ranging in size from 10 to 500 temperature on a continental scale. Biomass km 2 would be selected to represent different climatic would also be related to ecological units (i.e., regimes, and a data base would be assembled. Pilot Holdridge Life Zones) and to processes such experiments that entail collection of field data and as desertification, deforestation, and habitat remote sensing data would be conducted. These ex­ destruction. periments would be designed to determine if specific 2. Soils: Regional measurements of soil moisture processes can be detected through remote sensing of would be related to remote sensing surface earth surface features, and would use data from a variety of instruments. signatures to determine if a broad range of relative differences in surface moisture condi­ Land-atmosphere process models representing tions could be delineated using remote sensing the exchange of mass, energy, and momentum be­ instruments. tween the land and atmosphere systems would be developed. Detailed data sets acquired over the study 3. Hydrology: The application of remote sensing sites would be used to initialize or parameterize techniques to provide better estimates of sur­ models in simulation runs, and to verify and validate face water areal extent and volume, evapo­ the results of the models. The response of the land transpiration, precipitation, snow cover and surface in terms of biomass productivity and water volume, and soil moisture would be explored budget would be modeled given climate forcing on a global or regional basis and used if pro­ functions (precipitation and insolation) and ter­ ven feasible. restrial system properties (vegetation cover and sur­ face roughness). 4. Near-Surface Atmosphere: The capability to remotely detect and quantify climatic varia­ tions in the land surface record and to relate Data Needs these changes to climate process models would be also assessed. Measurements would include Satellite remote sensing data to be examined multi-stage sampling of the land surface for would include visible, near infrared, thermal in­ albedo, vegetation cover, surface roughness, frared, and microwave radiances acquired at spatial insolation, ground temperature, precipita­ resolutions ranging from 30 m to 30 km, and tem­ tion, etc. poral resolutions frorp. less than 1 day to 18 days. In addition to these data, observations from a Approach variety of spectrometers, radiometers, and cameras, flown on aircraft, would be used. Ground-based Historical land remote sensing data would be ex­ measurements of vegetation and soils would also be amined to determine if in regions known to have ex­ collected at selected study sites. These measurements perienced significant variations in their climate, would include physical sampling of vegetation these climatic changes can be detected through an biomass and soil moisture for laboratory analysis. analysis of changes in land cover. The approach Other measurements would include reflectance in would be to assemble land surface data obtained the visible and infrared portions of the spectrum from satellites into a data base, including developing using hand-held and truck-mounted radiometers,

38 and standard meteorological parameters, such as thetic aperture radar, digitized topography, soils temperature and relative humidity, from maps, and geophysical and geochemical data, meteorological stations. together with Eos data. A main objective of these projects would be to extract new information on the Participants types of soils and rocks in given areas. One of the first tasks would be to identify the Institutions participating in this type of project data that are available in a locale of interest. If would fall into two categories, those that would con­ mUltiple-image data sets exist, the researcher must be duct scientific research and those that could provide able to access information that will help identify the data. The number of scientific investigators involved best data sets for a particular project (e.g., if there would probably increase from perhaps 10 initially to are 20 Landsat images over the area, which one as many as 50 in subsequent years. They could be should be used). Information needed to select an im­ geographically located at a total of perhaps 15 age includes data quality, cloud cover, sun elevation, domestic and foreign government agencies, univer­ and weather information (e.g., visibility, wind sities, and research institutes. All of their facilities speed, water vapor content). When using data from would have the hardware and software expertise for multiple sources one of the most critical re­ digitally processing remote sensing data. Perhaps as quirements is that the various data be geometrically many as 10 national and international institutions coregistered in the desired projection. Thus, once the would be involved in providing data, including various data sets have been collected and are in satellite, aircraft, and field measurements digital form the next step would be geometric correc­ tion. Without good registration, the analysis and in­ Information System Considerations formation extraction process can generate un­ satisfactory results. Rectification is currently one of Project needs would include locating and retriev­ the most time-consuming steps in data reduction, ing appropriate data, providing data to co­ and most researchers would prefer not to deal with investigators, and transferring data to computer this complex task. An Eos data and information systems appropriate for each processing step. These system should provide standard services that include steps would typically involve reformatting, projection changes, image coregistration, vector-to­ preprocessing, processing and information extrac­ raster conversion, and generation of data sets with tion, and developing or verifying process models. various grid point settings. This type of project would need an information Another critical task is radiometric calibration. system that provides access to remote data bases and The data and information system should be able to processing services to accomplish the following produce quantitative digital-image data representing tasks: data search, browse, data storage, data pro­ some physical unit (e.g., albedo for Landsat data; cessing (reformatting, registration, etc.), geographic backscatter for radar data). For imaging spec­ overlay of dissimilar data, and data base manage­ trometer data, radiometric calibration should in­ ment. An additional requirement would be to pro­ clude both instrument corrections (gains, offsets, vide these services quickly and in a straightforward noise removal) and, in some cases, correction for at­ and easy-to-use manner. This could involve the use mospheric contributions and variations induced by of techniques being developed in the field of ar­ changing incidence, emission, and phase angles. tificial intelligence. The focus of the information Corrections such as these will allow data collected at systems needs discussed here allows scientists to different times or by different satellites to be used focus on the research, and not waste their efforts on and compared, because the sensor values will repre­ the details of accessing and preparing data for sent the same physical units relative to ground analysis. materials or cover. The processing stage comprises geometric and radiometric calibration, and once complete, the GEOLOGIC MAPPING researcher can begin the data analysis and informa­ tion extraction phase. One of the first problems will Digitally merging and interpreting data collected be determining which set of "group" of data should by different remote sensing instruments with map be used to extract the desired information. Because and field data is rapidly becoming a major task of of the large volume of data that is present when scientists in many different disciplines. For example, various data sets are merged (e.g., imaging spec­ Landsat Multispectral Scanner (MSS) data have trometer has dozens of image planes; thematic map­ been merged with both Seasat and SIR-A radar data per has three visible, three near-infrared, and one for geological analyses. These data have also been thermal band; synthetic aperture radar with one combined with geophysical, geochemical, and soils microwave band; plus topography, digitized soil data that were digitized from existing map sources data, and geophysical data), the researcher must and then converted into images. One can envision consider ays of grouping the data into subsets. The projects that utilize Landsat thematic mapper, syn- choice can be a subset of three bands, which has the

39 most information (with the least amount of duplica­ available, followed by eventual operational utiliza­ tion) for color composite representation or digital tion. The theoretical studies generally precede the classification; or bands where data with high correla­ data systems tests by 1 to 10 years. The data systems tion can be grouped together. Statistical methods tests are usually performed within the first year of such as principal components analysis, "selective" data availability. The processing system is set up to principal components analysis, and optimum index provide data in near real-time for a one to three factors can be used to accomplish data plane reduc­ month test period. It is set up as a "bare bones" 'tion and help extract information from the data. system without the redundancy, complete documen­ These techniques must be used because of the large tation, etc., required of an operational system. New number of combinations that can be generated (e.g., data are fed into a model running in parallel to an even Landsat thematic mapper data has 15 indepen­ operational model, and the forecasts are compared dent ratios, which can be grouped into 455 three­ to assess the impact of the new data. These data ratio combinations). Alternatively, deterministic ap­ systems tests are generally run in real-time with a proaches can be used, such as expert systems for post facto analysis and evaluation period. If the data identification of particular rock types from imaging show a positive impact, they are then brought into spectrometer data. operational use with the necessary processing equip­ ment and software being procured to provide re­ liable service. This generally requires one to three METEOROLOGICAL USES years after the data systems tests. If the satellite system providing new data will not be available for a Meteorology has a very active component of number of years, the sequence stops at the data operational activities involved in daily weather systems test until an operational satellite system is forecasting. The operational activities are strongly being procured. This generally requires five to eight tied to numerical weather prediction models. These years after the initial data availability on a prototype models use global coverage of various atmospheric satellite system. state parameters (including pressure, temperature, moisture, and winds throughout the depth of the at­ Field Studies mosphere) and runs are made every 12 hours. Horizontal resolution on the order of 100 km and There have been a number of large me­ vertical resolution on the order of 1 km are required teorological field studies (BOMEX 1968, GATE in the troposphere. Boundary conditions such as sea 1975, SESAME 1979, FGGE 1979, etc.) that last for surface temperature, soil moisture, snow cover, several months and focus on a particular phenome­ cloud cover, and topography are required by some of non, such as tropical convection. These studies use a the more advanced models. Current data assimila­ combination of in situ instruments and remote sens­ tion models use conventional radiosonde and sur­ ing gadgetry. An operations center directs the place­ face observations, satellite sounding radiometer ment of aircraft and dictates instrument schedules to data, cloud drift wind data, aircraft reports, and maximize information on the mechanisms under satellite imagery. study. In order to do this, the operations center re­ Most of the recent advances in weather fore­ quires real-time, quick-look data from all instru­ casting skill have resulted from either improved ment systems, including satellite instrumentation. In observing systems or better numeric models. previous field experiments, satellite data have either Because Eos instruments will provide global been received and processed directly at a field site, or coverage of meteorological parameters that could arrangements have been made to have the data improve operational weather forecasting, there will received and processed at a central facility and then be a very strong desire by the operational user com­ transmitted to the field she. munity to gain real-time access to selected Eos data. Following the field exercise, there generally is a In particular, the moderate resolution imaging spec­ two to five year period where research groups do in­ trometer, the high resolution multi-frequency tensive studies with the data. Quick-look field pro­ microwave radiometer, the laser atmospheric cessing of the data has already established a general sounder, the scatterometer, and the Doppler lidar framework of locations, times, etc., that are most would be useful for forecast models. promising for further study. Hence, the catalog re­ Data access patterns for operational users of Eos quirements are fairly straightforward, including lists data will require the full data stream processed to in­ oftimes, dates, etc., of data availability, data quality clude calibration, earth location, and scientific units indicators, etc. Prior experience with archives of in near real-time (within three hours of satellite data used within field experiments has observations). Previous experience with new data shown that approximately 20 to 100 separate groups sources has shown that the general sequence of will request data. Data requests are generally for 1 to events includes theoretical studies that show the 10 computer tapes, with an occasional user re­ potential impact of the new data, data systems tests questing 100 or more tapes. For a two-month experi­ of one to three months when the data first become ment, frequently two to five days will be selected by

40 most groups for intensive study with a few groups re­ characteristics from remote sensing data is central to questing data throughout the entire time period. The the economy of large-scale research. Therefore, in data requests generally reach a peak about one to the first approach, spectral signatures of vegetation two years after the field experiment. The studies would be collected and correlated with laboratory generally have a requirement for merged data sets measurements such as leaf reflectance. These data from a large number of different data sources. would be used to interpret measurements from air­ Hence, accurate location information is extremely craft and spacecraft where atmospheric conditions important in these studies. Calibration and conver­ attenuate and distort the characteristics of the sion to scientific units is also required for these signatures. studies. The second approach would employ both The actual instruments of interest for a given manual interpretation and machine classification of field of study will vary according to the purposes of satellite data to stratify vegetation and other surface the experiment. The Eos Surface Imaging and Soun­ features into broad, physiognomic categories (based ding Package (SISP), Sensing with Active Micro­ on vegetation structure) suitable for global com­ wave (SAM) package, and the Atmospheric Physical parison. Aerial photographs, field reconnaissance and Chemical Monitors (APACAM) would have in­ and other data sources would be used in this analysis. struments of interest, including very limited data sets Maps would be derived on scales consistent with ex­ of the very high-resolution imaging spectrometers isting small-scale vegetation maps. This approach and synthetic aperture radar instruments. would provide both a comparison for current infor­ mation sources, and an assessment of the methodol­ Case Studies ogy of very large-area vegetation mapping. How­ Meteorological case studies generally focus on a ever, because of the resources that would be required specific time and place where a particular meteor­ to process data for the entire land surface of the ological phenomena is occurring. The investigators Earth, an appropriate strategy would include the use are generally of two types, those having specific of coarser resolution data for primary stratification times and places already established from other data in a multistage sampling approach. Even using sources (these investigators are generally like field coarse-resolution data and statistical sampling, study investigators in that all they require are lists of assembling the required data would be a significant times and locations of data availability, quality in­ challenge. dicators, etc.) and those with only a specific phe­ nomena in mind (but no time or location). These re­ Information System Considerations searchers require extensive browse capabilities to lo­ cate likely data sets. The most successful browse ca­ In support of this research, a data and informa­ pability includes sampled data sets in an image or tion system should provide computation and infor­ graphical presentation. Books, movies, video discs, mation management support for a highly dynamic etc., of these-sample data sets can be rapidly exam­ and diverse set of processing and data requirements ined to locate a place and time that shows the desired that result from biospheric research needs. The ser­ phenomena. Then, the listings of data times, avail­ vices of the system should include data acquisition, ability, etc., are used to order data. search, browse, access, preprocessing, image pro­ cessing and display, data management (physical and electronic), and operational management functions. VEGETATION BIOMASS 1. Data Access: Online catalogs of and browse The purpose of this type of research is to gain a access to archival data would be beneficial. In better understanding of the spatial distribution of addition, the ability for direct ordering would vegetation characteristics and processes, including also be of value. The capability to browse biophysical factors (leaf area index, biomass, net large-image data bases before ordering images primary productivity, canopy temperature, and would be an important function. albedo), and plant physiological processes (evap­ otranspiration, photosynthesis, and respiration). 2. Data Input: Direct transmission of in situ data Objectives include: developing methods to meas­ (both vegetation and radiometric) between ure (by remote sensing) biomass and net primary field sites and processing centers would reduce production of terrestrial vegetation, and to employ delays by an order of magnitude (from months satellite images for assessing and improving the cur­ to a few days). Entry or conversion of an­ rent representational accuracy of continental-scale cillary data (topographic, soils, climatic) to an land cover information. acceptable format would add significantly. 3. Preprocessing: Registration (band-to-band Approach and sensor-to-sensor) and common data for­ Two approaches would be employed in this type matting would be of tremendous value and of research. The ability to infer key vegetation high priority. Also of value would be the

41 capacity to digitize photographs with interac­ Vegetation Mapping tive input from investigators at remote locations. Along certain areas of the Colorado River, three to four major crops are grown with irrigation water 4. Analysis: Efficiency of analysis could be in­ from the river. The amount of water that can be creased if real-time interaction between cen­ used is strictly controlled, and information on its ters and remote investigators were possible. use is gathered by the Water Resources Division of S. Archival Storage and Catalog: An active the United States Geological Survey. Information directory with up-to-date documentation of gathered with stream gauges along the river is used parallel and ancillary data sets held within to compute water usage. A study was conducted to and external to Eos would be of great value see if Landsat multispectral scanner data could be and high priority. used to predict water usage along the river. The method used a ratio of band 4 (chlorophyll) to 7 6. Distribution and Network: Access to (vegetation cell structure) to map the crops in an geographically dispersed data bases and the area. Then, known water usage for each crop was ability to overlay them in common format is used to predict the amount of water removed from of high priority. Data, besides being in com­ the river and the data sets correlated. This is clearly patible file format, must carry documenta­ a time-dependent problem. tion of quality and type. Time scales for such There are a number of problems where temporal access should be on the order of a few days. changes (man-made or naturally occurring) are of Networks of computer processors would be interest. There are several simple statistical valuable. parameters that can be computed for images and 7. All of the above require that an advanced used to monitor temporal changes of interest. data management and control system be a Statistical information that can be used for this pur­ part of the overall Eos concept. This system pose includes the average, standard deviation, and should facilitate researcher access, process­ skew coefficient, plus the correlation coefficient ing, and analysis in a manner essentially between images and a mean image for the area. transparent to the scientist. It should in effect Principal components of the images would also be permit scientists to function as scientists, not useful to identify changes that have occurred. librarians, communications experts, image With Eos data bases, differences between the processing specialists, or computer scientists. averages and standard deviations of images can be used to check for low frequency and total contrast changes. The correlation coefficient between the mean and any given image can be used to select im­ TEMPORAL INFORMATION ages that warrant examination. Temporal monitor­ EXTRACTION ing and comparisons should be made against the Following are several short scenarios that current scene average. Monitoring and change demonstrate the use of temporal information ex­ detection capabilities will allow the system to do tracted from data collected by remote sensing automatic-browse and search of data. These de­ instruments. rived data would be of value in a number of other research areas. Mapping Lava Flows Updating Digital Line Graph Data To understand volcanic processes and to gauge the extent of potential hazards, radar images col­ The Mapping Division of the United States lected every four to six hours over active volcanic Geologic Survey generates Digital Line Graph data fields are used to map the spatial distribution, flow by digitizing most of the information present on direction, and volume of each new flow. Data must either 1:2S0K or 1:24K topographic maps (e.g., be collected during the eruption regardless of the roads, railroads, drainage features). They are in­ weather conditions. Because of high cloud cover terested in using data collected by remote sensing probabilities, an imaging radar system would have instruments to update their Line Graph files. High to be used. If radar stereo images were collected, spatial-resolution image data that have been geo­ topographic information before and after the metrically and radiometrically calibrated is re­ various eruptions could be produced and used to quired. Digital processing techniques for pattern calculate the volume of magma erupted. Topo­ recognition and classification should be used (e.g., graphic data could also be used to compute slope, mapping of roads) and the registered Line Graphs which is used to predict the direction and possible should be periodically updated. Image data with at speed of any new flow. Eos and aircraft data least lO-meter spatial resolution recorded in one to could be used to meet the temporal coverage three spectral bands will be needed. Useful bands requirements. will be similar to thematic mapper bands 2, 4, and

42 5. A single band with five-meter spatial resolution application). Although a high spatial-resolution im­ may be preferred because of the very fine detail that ager is not included on the current Eos payloads, will be needed for identifying roads and other high­ one can imagine that an Eos Information System frequency patterns (spatial information may be could provide directory and catalog information for more important than spectral information for this this type of data.

43 APPENDIX II - REQUIREMENTS SYNOPSIS

This report presents more than 100 requirements gain settings or filter changes, or data acqui­ and recommendations concerning the functional sition rates during anomalous conditions). characteristics and attributes of a requisite data and information system for Eos. We include this Appen­ 5. Flight instruments will require a certain level dix as a means for assessing the interdependence of of onboard monitoring to ensure their func­ functional system elements and as a convenient tional capability and to format, error code, reference for analysis of the architectural example and buffer telemetry data, as well as the presented in Chapter III. This listing is not complete, merging of critical ancillary data needed for nor is it intended to be the sole reference for in­ quick-look and other analyses. dividuals seeking a definitive accounting of re­ quirements and recommendations set forth by the 6. The flight data system should be capable of Eos Data Panel. Rather, it is meant to show the rela­ acquiring and merging ancillary (e.g., time tionship between functional elements of an Eos data and position) and correlative (e.g., in situ) and information system and to allow the reader to data into instrument data streams. quickly obtain an overview of the requirements con­ 7. It should be possible to receive directly tained within this report. Eos Data Panel recommen­ transmitted data at any modestly equipped dations should not be taken out of context, and ground station that is within line of sight of therefore the reader is referred to-the main body of an Eos platform. this report for a complete accounting. The requirements listed below are grouped into 8. At least 95 percent of the real-time data seven generic classifications: Eos flight systems, broadcast by the satellite should be received, user, operational, information services, advanced preprocessed, and displayed within 60 data base management, data processing, and non­ seconds from the time of transmission. Eos data base functions. An eighth category, management considerations, has been included and 9. In order to buffer the high-peak data rates contains recommendations dealing with both im­ and to guard against momentary TDRSS plementation and the unique characteristics of the channel outages, an onboard data storage data and information system. The seven classes of capability should be developed. system requirements are subdivided as appropriate and references to text indicated. No priority is in­ dicated by the relative order in which these recom­ mendations or requirements are listed. II. USER FUNCTIONS I. Capabilities of data centers should be periodically evaluated by key personnel who I. FLIGHT SYSTEMS FUNCTIONS lead activities of instrument teams and active data base sites. 1. Execution times for any system operation within an Eos data and information system 2. All Eos data should be validated to the should be in seconds, and those for archival satisfaction of the research community. data should be within minutes for small volumes, and days for large volumes, 3. Investigators will require quick-look data depending on the priority of the request. sets for instrument command decisions, and for preliminary scientific analysis. 2. The Eos spacecraft should include a flexible flight information subsystem for controlling 4. Users will submit instrument requests to an onboard processing of both uplink and instrument operations center for execution. downlink information. This interaction will most likely occur over an electronic information network. 3. Interactive commands that require critical onboard resources should be processed cen­ 5. One of the unique features of Eos is the re­ trallyat a ground-based control center, while quirement that a researcher return to the ar­ non-interactive commands should be directly chives the results of his research, in the form transmitted from the user to the onboard of reduced or derived data (Le., value-added system for final assessment and forwarding data). to the appropriate instrument or subsystem. 6. Instrument teams will guide instrument and 4. Onboard systems should be used for deci­ algorithm development and will be involved sions when rapid response is needed (e.g., in mission operations.

44 III. OPERATIONAL FUNCTIONS B. Instrument A. Mission 1. A mission operations center is the focus for real-time command and control of the space 1. A master schedule should be updated every platform. Instrument operations centers are orbit and it should contain logistical infor­ the focus for Eos instrument operations. mation needed to plan an observation. 2. Access to spacecraft subsystems should be channeled transparently through any con­ 2. A mission operations center is the focus for trol system for non-interactive commands. real-time command and control of the space platform. Instrument operations center(s) 3. Commands should be categorized as non­ are the focus for Eos instrument operations. interactive, interactive, and critical. 4. Research groups should have the capability 3. Subsystems of the information system will support mission operations, including rele­ to request acquisition of special obser­ vational sequences from one or more vant network functions, acquisition and instruments. delivery of data to mission and instrument repositories, quick-look data production, 5. Instrument teams may require command and the large-scale processing needs of in­ control over various instruments to satisfy strument teams. scientific objectives. 4. Instrument teams will guide instrument and 6. Researchers will require near-real-time pro­ algorithm development and will be involved cessed data to decide, for example, appro­ in mission operations. priate locations for mobile, in situ instru­ ment systems (e.g., aircraft, ships). 5. A mission operations center will need to continually monitor a sampling of data in 7. Instrument specialists and operations per­ near-real time for quality control, error sonnel will require rapid decision-making capabilities. This requirement includes detection, and instrument assessment. establishing real-time processing and display 6. The capability to reconfigure observational capabilities that can be utilized to monitor sequences when malfunctions or special events. events occur is needed. 8. All Eos instrument parameters should be monitored by examining their performance 7. If problems occur, control centers need to statistically. have the capability to trace the data flow back through the processing system to the 9. If problems occur, control centers need to instrument, aiding in the isolation and cor­ have the capability to trace the data flow rection of these problems. back through the processing system to the instrument, aiding in the isolation and cor­ 8. The capability should exist to integrate com­ rection of these problems. mands and create command sequences from simple requests to complex operations that 10. Users will submit instrument requests to the require the coordination of multiple sub­ instrument operations centers for execution. systems and instruments. This interaction will most likely occur over an electronic information network. 9. Access security must be maintained at ap­ propriate levels at all times. IV. INFORMATION SERVICES 10. All Eos instrument parameters should be FUNCTIONS monitored by examining their performance statistically. 1. Required functions and capabilities should not constrain physical location of personnel, 11. Automated command sequence construc­ data, or processing capabilities. tion, together with the necessary computer 2. A major goal of the system should be to pro­ involvement in routine spacecraft com­ vide remote electronic access for the scien­ munications, should be available in mission tific community to the variety of capabilities operations computers. and services that Eos will provide. 12. In some cases, instrument command deci­ 3. Particular emphasis should be placed on sions will have to be made within the time technology associated with transparent ac­ period of one orbit. cess to dispersed, heterogeneous data bases.

45 4. The system will require cost-effective net­ chers will require an electronically accessible work capabilities, including direct broadcast directory of pertinent Earth sciences data. access. 2. Self-documenting software will be required 5. The network should provide access to mis­ to create directory entries for newly acquired sion operations, archives, selected active data. data bases, and to large mainframe computers. 3. The Earth science data directory requires maintenance and periodic updating as 6. Data center personnel should manage the project-sponsored researchers identify new network, ensuring that the data centers are data sets and as new Eos data are acquired. involved directly in many key aspects of the project. 4. The master directory should include infor­ 7. Because of rapid advancements in the mation on non-Eos data deemed pertinent by telecommunications industry, and the long Eos-sponsored researchers. lead time for Eos implementation, the telecommunications requirements should be 5. A researcher should be able to search the periodically reevaluated and updated as new directory by project, platform, instrument, technologies become more readily available. data processing level, version, parameter, time, location, or any combination of these 8. Communications linking quick-look users attributes. with data repositories will be required. Data rates between 56K baud and 1.5M baud will 6. The directory should include information likely be required on this link. about supporting catalog access. 9. The system should provide for a spectrum of electronic data delivery rates, ranging from B. Electronic Catalogs 9,600 baud to 6.3M baud. 1. Eos-sponsored multidisciplinary and multi­ 10. Access security should be maintained at ap­ ple data source, disciplinary-oriented resear­ propriate levels at all times. chers will require electronically accessible 11. Provisions should be made within the net­ catalogs of Earth sciences data. work design for future expansion to accom­ 2. Self-documenting software will be required modate new user's access to the services and capabilities afforded. . to maintain and update the catalogs as new data are acquired. 12. NOAA and its international clientele will re­ quire separate data storage and relay systems 3. The Eos data catalog should be accessible by for data delivery if they deploy operational project, platform, instrument, data process­ instruments on an Eos spacecraft. ing level, version, parameter, time, geo­ graphic location, or any combination of the 13. Subsystems of the information system will above. support mission operations, including rele­ vant network functions, acquisition and 4. The Eos catalog should contain a use trace in­ delivery of data to mission and instrument cluding user name and address for any par­ repositories, quick-look data production, ticular data set. and the large-scale processing needs of in­ strument teams. 5. To the extent possible, the Eos project should 14. Users will submit instrument requests to the endeavor to create, maintain, and update instrument operations control centers for ex­ similar catalog services for pertinent data ecution. This interaction will most likely oc­ held external to the project. cur over an electronic information network. C. Browse Files 15. Access security must be maintained at ap­ propriate levels at all times. 1. The Eos project will be required to create, maintain, and update browse files within the V. ADVANCED DATA BASE archival system. MANAGEMENT FUNCTIONS 2. Browse files should be updated in real-time A. Electronic Directory for non-image data. 1. Eos-sponsored multidisciplinary and multi­ 3. Near-real-time updates will be required for ple data source, disciplinary-oriented resear- some critical image data. It is anticipated

46 that these data will be of reduced resolu­ internal and external processing history, and tion (either spectrally or spatially). coding of telemetry. 4. Browse files should be accessible on an in­ 3. All sources of calibration information, pro­ strument, time, geographic location, or cedures, and results must be identified and any combination of these factors. placed in the archival documentation. 5. Browse files should be searchable visually 4. An annotated bibliography covering all per­ via attributes and by expert systems. tinent papers and reports published in the refereed literature should be included as a 6. Data attributes (e.g., cloud type and cover, part of the project's documentation (e.g., vegetation type and cover, snow cover, instrument descriptions, observations and data quality) should be appended to an in­ sequences, calibration and sensitivity ventory record within the browse files. studies, precision and accuracy of 7. Attribute files should be expandable, ena­ measurements). bling researcher-identified attributes to be 5. Documentation should also include any in­ added subsequently. formation pertinent to scientific interpre­ 8. Researchers will require processing of tation of the data produced by various reduced-volume data sets to Level 2 for experiments. browse purposes. 6. Consideration should be given to providing 9. Pattern recognition algorithms should be online, electronic access to technical reports developed and applied to browse data for and memoranda (i.e., the gray literature). attribute identification. F. Archives 10. Online browse capabilities may not be possible for many users; this will 1. The archival system should be able to necessitate publication of an image browse assimilate data at all processing levels. catalog. 2. Access to non-Eos data that are needed for 11. Consideration should be given to creating processing of Eos data or by Eos in­ special purpose browse files interactively. vestigators will be required. 3. In situ data used in the processing or valida­ D. Data Ordering tion of Eos data should be easily accessible and maintained within the archives. 1. The system should support online, elec­ tronic ordering of all Eos data. 4. The archives must be able to assimilate, maintain, and distribute higher level or 2. To the extent possible, the system should reduced data sets produced by either active support the online, electronic ordering of data base sites or by individual investigators pertinent non-Eos data sets. in their research work. 3. Interactive ordering capabilities should be 5. Directory and catalog entries for all data available with at least 9,600 bps dial-up stored and maintained within the archives links. will be required. 4. The system should be available on a con­ 6. The archives will require a fast, efficient, tinuous basis and be sized to handle at least and cost-effective retrieval system for all of 100 simultaneous users. its data holdings. 7. A history of both instrument and algorithm E. Documentation performance should be maintained and ac­ cessible via the archives. 1. Documentation is considered to be a vital part of the data record and should be 8. The Eos archival subsystem should be ac­ stored and maintained with the same care cessible from remote terminals and worksta­ as the data per se. tions via a prompting menu or natural language interface. 2. The documentation should include a con­ cise description of instrument specifica­ 9. All capabilities and functions of the archival tions, including the nature of physical subsystem should be fully documented in a variables measured, noise characteristics, user handbook and in online help files.

47 10. The archives should maintain a complete in­ poses during fixed periods of time (e.g., ventory of relevant scientific and technical scientific processing for particular instru­ documents concerning the data and the ment team activities). archives. 7. Access security must be maintained at ap­ 11. Processed data should be retained in access­ propriate levels at all times. ible form, along with associated documenta­ tion, to avoid redundant processing. 8. Users identified as requiring access to real­ time, quick-look information need interac­ G. Standards and Formats tive image processing and display capabilities. 1. Considerable and careful attention must be given to the adoption of standards and for­ 9. Processing facilities will be needed to sup­ mats; the research community should be in­ port requested higher-level data processing timately involved in this process. for scientific research purposes. 2. We envision the necessity for an entire suite 10. Data reduction, grid overlay, standard pro­ of format structures and, consequently, jection, data set merger, etc., will be re­ system transparency remains an overriding quired by Eos researchers. concern. II. Distributed data processing facilities will be 3. Consideration should be given to various required to perform similar higher-level data standards adopted by the Space Station Pro­ processing functions. gram in areas of overlapping concern if they 12. All data processing requirements are in­ are acceptable to Eos. dependent of specific organizational 4. Careful consideration should be given to assignments and must be met by any facility working with both established national and (including commercially operated) that is international committees addressing stan­ producing Eos data. dards and possible adoption of the stan­ 13. Access to large mainframe computer facili­ dards that they recommend. ties will be required for Eos researchers 5. A common, standard glossary containing engaged in model development, evaluation, operational definitions used within Eos simulation, and experimentation. should be created and maintained by the 14. Data sequence and encoding peculiarities project. must be removed by the data reception system. VI. DATA PROCESSING FUNCTIONS VII. NON-Eos DATA BASES 1. Processing requirements within the system will range from raw data to fully processed FUNCTIONS scientific data. 1. Many Eos-sponsored research tasks will re­ 2. Instrument-specific, near-real time, or even quire access to operational data bases such real-time data processing, delivery, and as NOAA's. The project should make every display capabilities will be required. effort to provide this access in a user transparent fashion. 3. Algorithms to be used for processing data should be approved and fully validated by 2. International data bases such as the holdings the research community. of the World Data Centers will be required by Eos researchers. Access to these data sets 4. Requirements for processing include: Level should be through the Eos data and infor­ 0, all data (for temporary repositories); mation system. Level I, all data (for long-term archives); Levels 2, 3, and 4, only upon request. 3. National archives such as those of the Na­ tional Climate Center, National Ocean­ 5. The requisite data processing capability ographic Data Center, etc. should be online must support some near-real-time data and to Eos researchers via the data and informa­ information processing. These results tion system. should be available to the requester within the time period of one orbit. 4. Reduced data sets produced from data derived from other archives, as well as the 6. Data processing subsystems should be original data, should be maintained within capable of enhancement for special pur- the information system.

48 5. Directory entries covering pertinent, non­ 7. We recommend that the Global Resource In­ Eos data sets should be obtained and main­ formation System concept be focused on tained within the system. Eos objectives, then utilized for developing the network services aspects of the Eos data 6. The data and information system should ac­ and information system. como date catalog entries for non-Eos data sets. 8. We recommend that the Earth Science and Applications Division fund a limited num­ VIII. MANAGEMENT ber of multidisciplinary research teams to CONSIDERATIONS investigate a select number of objectives in hydrology, biogeochemistry, and climatol­ 1. The scientific community must be involved ogy using existing data sets. during the development and evolution of the Eos data and information system. 9. We recommend that the Eos data and infor­ mation system be built on the experience 2. Active researchers should have an oversight gained with existing Earth science pilot pro­ and review responsibility for the data and jects, that these projects be focused toward information system. Eos objectives, and that close collabora­ 3. NASA management should pay particular tion be maintained with the ueAR Unidata attention to existing National Academy of initiative. Sciences reports dealing with space research 10. We recommend that planning and evolu­ data systems. tionary implementation of an Eos data and 4. A crucial difference between Eos and pre­ information system begin without delay, vious flight projects that requires procedural enabling the resultant system to reflect the changes is the length of the effort. experience and expertise of those people who must manage and utilize its capabilities. 5. An Eos data and information system must evolve and be a system that includes geo­ 11. Eos should accommodate operational users graphically distributed sites of varying capa­ if their activities do not detrimentally impact bility and responsibility. the conduct of Eos-sponsored research. 6. A critical factor in developing the Eos data 12. If operational uses are made of Eos data, and information system will be strong in­ and if these uses significantly affect the in­ teraction and close collaboration with ongo­ formation system, then those operational ing research efforts that emphasize the col­ agencies should provide enhancements to lection and analysis of multiple data sets. the system to meet their needs.

49 BIBLIOGRAPHIC DATA SHEET

1. Report No. 12. Government Accession No. 3. Recipient's Catalog No. NASA n.1-87777 4. Title and Subtitle 5. Report Date EARTH OBSERVING SYSTEM June 1986 DATA AND INFORMATION SYSTEM 6. Performing Organization Code REPORT OF THE EOS DATA PANEL - VOLUME IIa 610 7. Author(s) 8. Performing Organization Report No. See Supplementary Notes 86FSOOI 9. Performing Organization Name and Address 10. Work Unit No. Laboratory for Atmospheres Goddard Space Flight Center 11. Contract or Grant No. Greenbelt, Maryland 13. Type of Report and Period Covered 12. Sponsoring Agency Name and Address Technical Memorandum National Aeronautics and Space Administration Washington, D.C. 14. Sponsoring Agency Code

15. Supplementary Notes R. Arvidson, Washington Univ.; F. Billingsley, Jet Propulsion Lab; R. Chase, Woods Hole Oceanographic Inst.; P. Chavez, USGS Flagstaff; M. Devirian, NASA Headquarters; J. Estes, Univ. of California at Santa Barbara; G. Hunolt, NOAA Satellite Data Services Div.; J. Klose, Jet Propulsion Lab; G. Ludwig, Consultant; F. Mosher, NOAA National Severe Storms Forecast Center; W. Rossow, Goddard Institute for Space Studies. 16. Abstract

The purpose of this report is to provide NASA with a rationale and recommendations for planning, implementing, and operating an Earth Observing System data and information sys- tem that can evolve to meet the Earth Observing System's needs in the 1990s. The Earth Observing System (Eos), defined by the Eos Science and Mission Requirements Working Group, consists of a suite of instruments in low Earth orbit acquiring measurements of the Earth's atmosphere, surface, and interior; an information system to support scientific re- search; and a vigorous program of scientific research, stressing study of global-scale processes that shape and influence the Earth as a system. The Eos data and information system is conceived as a complete research information system that would transcend the traditional mission data system, and include additional capabilities such as maintaining long-term, time- series data bases and providing access by Eos researchers to relevant non-Eos data. The Working Group recommends that the Eos data and information system be initiated now, with existing data, and that the system evolve into one that can meet the intensive research and data needs that will exist when Eos spacecraft are returning data in the 1990s.

17. Key Words (Selected by Author(s)) 18. Distribution Statement Eos Data Users, Data Acquisition, Unlimited - Unclassified Data Processing, Telecommunications, System Architecture, Data Management STAR Category 42

19. Security Classif. (of this report) 20. Security Classif. (of this page) 21. No. of Pages 22. Price* Unclassified Unclassified 62 A04 ... *For sale by the National Technical Information Service, Springfield, Virginia 22161 GSFC 25-44 (10177) End of Document

r·