NASA's Earth Observing Data and Information System (EOSDIS)
Total Page:16
File Type:pdf, Size:1020Kb
NASA’s Earth Observing Data and Information System (EOSDIS) J. Behnke NASA GSFC Library of Congress Designing Storage Architectures Meeting September 9, 2019 Earth Observing System Data and Information System (EOSDIS) EOSDIS Users Research distribute Applications data subset Education downlink Public archive capture process and clean Funded and Managed by the ESDIS Project at NASA GSFC 2 A Growing Archive and Growing Number of Users Millions 1,700 1,600 1,500 1,400 1,300 1,200 1,100 1,000 900 800 Our Archive today 700 32 PB 600 500 400 300 200 100 0 FY00 FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 FY14 FY15 FY16 FY17 FY18 FY19 Product distribution 147.6 M distributed in Jul 2019 Prediction: 2 Billion products distributed this fiscal year In FY2022, predict that the archive will grow by 48PB that year alone Our Archive growth Not the first time we have confronted this opportunity 3 EOSDIS Storage Architecture Evolution by Decade 1990 2000 2010 2020 Near-line Storage devices used Reaching Peak Complexity Hierarchical Storage 14 StorageTek silos at 4 DAACs All online storage Migrated data on disk farms to Management 45,580 tapes (3580 format) Duplication of data across disk commercial cloud data lakes StorageTek silos (Digital Linear Begin Reducing dependency on farms Use of RAID throughout Migrate to/from vendors as Tapes) Near-line storage - removing necessary to improve efficiency Metrum RSS-600 (VHS) Storagetek and performance 3480 18 track tape drives Copy to additional vendors as 9 track tape drives Robust backup tape devices necessary to improve Increase direct and network System backups to tape performance attached commodity disks; more Data backups to offsite disk farm Local Storage for Processing RAID devices Direct attached disk devices RAID parallel disks Robust backup tape devices Backup Tape Devices Data Migrations - Sony DTF; All data distribution via Internet Tape Drive Cartridge Stackers LTO-4 from spinning disk data pools 4 & 8 mm Tape Drives Utilize commercial data backup service for secondary archive Store tertiary copy of Storage for Distribution Storage for Distribution irreplaceable data on premises Online Disks support data pools on RAID and tape CD-ROM Public internet data access Begin assessment of 4 & 8 mm tapes exceeds orders for data on off- Commercial Cloud Resources Detachable Disk Drives line storage units Nearline, offline Nearline, Direct Direct Access Direct Access On premise Access On premise On and off On premise premise Access in Access in Access in Seconds Access in Seconds/ Hours/Minutes Minutes/Seconds Milliseconds 4 EOSDIS Distributed Active Archive Center (DAACs) and Science Investigator-led Processing Systems (SIPS) Socioeconomic Data and Applications Center Human Interactions, Land Use, Environmental Atmosphere SIPS Sustainability, Geospatial Data Goddard Earth Sciences Data and Land Processes Information Services Center DAAC Global Precipitation, Solar Irradiance, Atmospheric Composition and Dynamics, Land Cover, Surface Reflectance, Global Modeling Radiance, Temperature, Topography, Vegetation Indices Crustal Dynamics Data Level 1 and Atmosphere Archive Information System and Distribution System (LAADS) Space Geodesy, Solid Earth Measurements MODIS Level-1 and Atmosphere Data of Pollution in the Products Physical Ocean Biology DAAC Oceanography DAAC Troposphere (MOPITT) Visible Infrared Ocean Biology, Gravity, Sea Surface Sea Surface Temperature Temperature, Ocean Winds, Imaging Radiometer Microwav Topography, Circulation & Suite (VIIRS) Ocean e Limb Currents Oak Ridge National Sounder Laboratory DAAC Ozone Mapping (MLS) Tropospheric Biogeochemical Dynamics, Profiler Suite Ecological Data, Environmental Emission LaRC Atmospheric (OMPS) Ozone National Snow and Ice Processes Spectrometer (TES) Data Center DAAC Science Data Center • Frozen Ground, Glaciers, Radiation Budget, Clouds, Ozone Monitoring Ice Sheets, Sea Ice, Aerosols, Tropospheric Instrument (OMI) Sounder Snow, Soil Moisture Chemistry • SIPS Ozone Mapping Global Hydrology Profiler Suite Resource Center DAAC Advanced Microwave (OMPS) Ozone Hazardous Weather, Scanning Radiometer for • Lightning, Tropical Cyclones and EOS 2 (AMSR-E/2) Alaska Satellite Storm-induced Hazards Ocean Data Facility DAAC Processing System SAR Products, Sea Ice, (OCDPS) Polar Processes, • Geophysics MODIS Adaptive Processing System (MODAPS) • f Visible Infrared Imaging Radiometer Suite (VIIRS) Land Development of the Earthdata Cloud • Earthdata Cloud Platform is a multi-account, Infrastructure-as-a-Service (IaaS) cloud platform operating on Amazon Web Services (AWS) under a single ESDIS owned top level “payer account”, providing shared cloud services and controls to EOSDIS. Data Lake 6 Common Services & Controls 1. Single Contract into Commercial Cloud Services EOSDIS operates under multiple contracts & partner Agencies. Centralized cloud contract through NASA’s Enterprise Managed Cloud Computing (EMCC) program provides seamless access to cloud. 2. User Access to Earthdata Cloud Development Secure PIV/Token login, NASA Agency-based account provisioning, 3. NASA Approved Amazon Services Vetted AWS and 3rd party SAAS services, with process to add new services 4. Code Deployment Services Through the use of Bamboo, code is security scanned, built, and deployed into Earthdata Cloud. 4. Data Recovery Services Developing a service to backup collection in lower cost cloud resource; but also keeps ‘golden’ copies on premise. 4. Budget Distribution and Enforcement Our components in the Earthdata Cloud operate their environment, ESDIS gets the bill. ESDIS Capability to capture intended costs, distribute approved budgets into project level accounts, monitor, and protect against inadvertent cost overruns or bad actors. 7 How the users look at information/data in the Storage Systems Current Earthdata search and access to science data in the cloud Highlights selected collection, observation time, granule location Downloads from S3 bucket archived in AWS commercial cloud THANKS! You can contact: [email protected] Worldview https://worldview.earthdata.nasa.gov Earthdata Search https://search.earthdata.nasa.gov Youtube Webinars: https://www.youtube.com and search for NASA Earthdata 9.