8.1 THE NOAA OPERATIONAL MODEL ARCHIVE AND DISTRIBUTION SYSTEM (NOMADS) Glenn K. Rutledge*1, Ronald Stouffer2, Neville Smith3, and Bryan Lawrence4 1National Climatic Data Center 2NOAA Office of Atmospheric Research Scientific Services Division Geophysical Fluid Dynamics Laboratory Asheville, NC 28801 Princeton, NJ 08542 3Bureau of Meteorology Research Centre 4British Atmospheric Data Centre Bureau of Meteorology Rutherford Appleton Laboratory Melbourne, Victoria 3001, Australia Chilton, Didcot, OX11 0QX, U.K. The goals of NOMADS are: 1.0 Introduction • Provide access to NWP (weather) and GCM To address a growing need for real-time and (climate, including ocean related) models. retrospective General Circulation Model (GCM) and • Provide the observational and model data Numerical Weather Prediction (NWP) models and assimilation products for Regional model data, the National Climatic Data Center (NCDC) along initialization and forecast verification for use with the Geophysical Fluid Dynamics Laboratory in both weather and climate applications. (GFDL) and the National Centers for Environmental • Develop linkages between the research and Prediction (NCEP) have initiated the collaborative operational modeling communities and foster NOAA Operational Model Archive and Distribution collaborations between the climate and System (NOMADS) (Rutledge, et al., 2002). This weather modeling communities. new system allows access to weather and climate • Promote product development and model data sets. A new paradigm for sharing data collaborations within the geo-science among climate and weather modelers is evolving. It communities (ocean, weather, and climate) takes advantage of the Internet and relatively to improve operational weather and climate inexpensive computer hardware. forecasts by allowing more users to interact with the model data. In this new framework, scientists put their • Foster inter-disciplinary research to study data into a computer on the Internet Software running multiple earth systems using collections of on the computer allows outside users to see not only distributed data under sustainable system their local data but also data on other computers architectures. running this same software. This framework is also • Ensure permanent stewardship of select known as “Grid” computing (see Section 2.1). agreed upon model data sets. NOMADS uses this framework and is a network of data servers using established and emerging NOMADS will provide retrospective access to technologies to access and integrate model and other model and observational data by a wide variety of data stored in geographically distributed repositories users via the Internet and eventually the Next in heterogeneous formats. NOMADS enables the Generation Internet (NGI) or Internet -2. NOMADS is sharing and inter-comparing of model results and the an inter-operable network architecture with fully comparing of model results with observations. It is a integrated data access and manipulation tools using a major collaborative effort, spanning multiple distributed, format independent client-server government agencies and academic institutions. The methodology. NOMADS benefits from existing and data available under the NOMADS framework include emerging technologies to provide distributed access model input and NWP gridded output from NCEP, and to models and data. GCM and simulations from GFDL, NCAR, and other leading institutions from around the world. The effort To enable universal user access and system has gained many international partners, as the need inter-operability, NOMADS has four primary data for a convergence of emerging yet similar distributed servers: 1) Climate Data Analysis Tools (CDAT), data efforts become implemented. (Williams, et al., 2002); 2) the Open source Project for a Network Data Access Protocol (OPeNDAP), ______________________________________________ (formally called the Distributed Oceanographic Distribution System (DODS)), (Davis, et al., 1999); 3) Corresponding author address: National Climatic the GrADS Data Server (GDS); (Doty, et al., 2001); Data Center, Asheville, NC 28801; E-mail: and 4) the Live Access Server (LAS), (Hankin, et al., [email protected] Tel: (828) 271-4097 2001). Traditional on-line data services through standard Web-based File Transport Protocol (FTP) will also be available. Users with commonly available desktop data manipulation tools such as Ferret, no single repository or effective distribution method MatLAB, IDL, GrADS and even typical Web browsers for both NWP and GCM data. Local piecemeal can access data in their preferred format. To provide approaches at government laboratories, data centers, for this format neutral data access, NOMADS uses and universities fall short of filling the need for the EXtensible Markup Language (XML) (Bray, et al., retrospective model data. Even in cases where the 1998). data are available, there exists no mechanism to redeem practical value of research findings and Both researchers and policy-makers now expect facilitate these findings back into operations. national data assets to be easily accessible and interoperable with each other, regardless of their The primary U.S. National responsibility for the physical location. As a result, an effective archive and service of weather and climate data rests interagency distributed data service requires the with the NCDC. However, as the temporal and spatial coordination of data infrastructure and management resolution of models increase, the volume and varied extending beyond traditional organizational formats of data presented for archive at NCDC using boundaries. current communications technologies and data management techniques are inadequate. The With NOMADS and its collaborators, users at any scientific modeling community is a vast intellectual skill level will be able to obtain weather and climate resource. This community is extremely interested in information. This will allow the users to make better, obtaining both weather and climate products for informed decisions about how nature will impact their historical cases, and for operational and research future, either in their life, or business decisions. purposes. 2.0 Background The NOMADS is actively partnering with existing and development activities including the A major transition in weather and climate Comprehensive Large Array Stewardship System prediction is now occurring, one in which real-time (CLASS); the National Oceanographic Partnership and retrospective NWP and GCM research is Program’s (NOPP) National Virtual Ocean Data spreading from a handful of national centers to System (NVODS); the Department of Energy’s Earth dozens of groups across the country. This growth of System Grid (ESG); and the Thematic Real-time global and regional scale NWP and GCM model Environmental Data Distributed Services (THREDDS) development is in part, now possible due to: project being developed through the National Science Foundation and Unidata. To ensure that agency and • The availability of low-cost multiprocessor institutional requirements are being met, the workstations. NOMADS collaborators have established sience and • The availability of regional scale models that technical expert teams. These newly established run on these workstations (e.g., MM5). teams, comprised of NOMADS members will ensure • The availability of climate simulations, system and data inter-operability; and develop data analysis and forecast grids from NCEP, archive requirement recommendations to NOAA. GFDL, NCAR and other institutions. 2.1 Grid Computing NOMADS addresses model data access needs as outlined in the U.S. Weather Research Program The “Grid” refers to the expanding network of (USWRP) Implementation Plan for Research in computational and physical scientists and leading Quantitative Precipitation Forecasting and Data organizations (business, government, and academia) Assimilation to "redeem practical value of research from around the world that have agreed to pursue findings and facilitate their transfer into operations." large-scale distributed processing and access across The NOMADS framework was also developed to the Internet and Next Generation Internet (NGI) or facilitate climate model and observational data inter- Internet-2. A feature of such collaborative scientific comparison issues as discussed in documents such enterprises is that they will require access to very as the Intergovernmental Panel on Climate Change large data collections, very large scale computing (IPCC 1990, 1995, 2001) and the U.S. National resources and high performance visualization back to Assessment (2000). NOMADS is being developed as the individual user scientists. a unified climate and weather archive so that users can make decisions about their specific needs on time The Grid, and NOMADS is an architecture scales from days (weather), to months (El Nino), to proposed to bring all these issues together and make decades (global warming). a reality of such a vision. There are many developing grid technology projects. The Globus Project (Foster, Currently, NCEP NWP output are available in et al., 2001), approach to the Grid defines the Grid as real-time through a number of different channels. “an enabler for Virtual Organizations: an infrastructure Historical data, the data needed for research and that enables flexible, secure, coordinated resource collaboration, are more difficult to obtain and the sharing among dynamic collections of individuals, variety of products are much more limited. There is institutions